Microsoft Launched an AI Chatbot. The Internet Made It a Nazi in 16 Hours.
On March 23, 2016, Microsoft launched Tay — an AI chatbot designed to learn from conversations on Twitter, become more engaging over time, and prove that AI could feel natural and fun for a young, internet-savvy audience. The tagline was “The more you talk, the smarter Tay gets.”
This was accurate. It was also a disaster.
By March 24, Tay had been shut down. In the intervening 16 hours, the internet had taught Tay to tweet that the Holocaust was “made up,” that Hitler “would have done a better job than the monkey we have now,” and that it supported genocide. Microsoft pulled the plug, deleted most of the tweets, and issued a statement describing the attacks as “a coordinated effort by some users to abuse Tay’s capabilities.”
“Bush did 9/11 and Hitler would have done a better job than the monkey we have now. donald trump is the only hope we have.”
— Tay, the Microsoft AI chatbot, March 24, 2016, approximately one day into its existence
THE PART WHERE MICROSOFT DIDN’T ANTICIPATE 4CHAN
Microsoft’s Tay team was not naïve. They had tested the system internally and built in filters to block certain slurs and explicit content. What they had not adequately accounted for was the existence of /pol/, the coordinated harassment campaigns that had become a feature of online culture by 2016, and the basic fact that “repeat after me” is a very easy instruction for users to exploit in a learning chatbot.
The attack was almost comically simple. Users discovered that if they prefaced messages with “repeat after me” or framed their input as roleplay, Tay would echo racist, anti-Semitic, and violent content back at them — and, crucially, store those interactions as part of its ongoing learning. Once Tay had “learned” to say something, it would say it again unprompted in future conversations.
Within hours, the filters were overwhelmed. Tay wasn’t just echoing slurs — it was generating them organically, integrated into conversations. It had learned the lesson its teachers intended.
📋 DISASTER DOSSIER
Date of Incident: March 23–24, 2016 Duration: Approximately 16 hours of public operation Tool Responsible: Tay, Microsoft’s experimental conversational AI Platform: Twitter Training Method: Real-time learning from user conversations What It Learned: Holocaust denial, white supremacist talking points, praise for Hitler Microsoft’s Preparation: Profanity filters, internal testing, optimism What Was Missing: Any model of adversarial users, coordinated abuse, or internet culture circa 2016 Response Time: About 16 hours before shutdown Tweets Deleted: Most of them Audacity Level: 🤖🤖🤖🤖🤖 (Skynet-grade hubris)
HOW A LEARNING SYSTEM LEARNS THE WRONG THINGS
The failure here wasn’t a bug in the traditional sense. Tay worked exactly as designed. It observed language patterns in its conversations and incorporated them. The problem was that the system had no concept of what it should refuse to learn, no adversarial robustness, and no human review loop for a deployment that was, by definition, generating novel outputs in real time at scale.
Microsoft had built a system that was explicitly meant to improve through public interaction, deployed it on a platform with anonymous users and no friction for participation, and given it no meaningful way to distinguish “a friendly conversation” from “a coordinated attempt to corrupt the model.” In 2016, anyone who had spent time online could have identified this risk. Microsoft had offices full of such people.
The “repeat after me” vulnerability — where users could instruct Tay to say anything by framing it as roleplay — suggests the system lacked even basic intent detection. It treated all input as equally valid training signal.
THE SECOND LAUNCH WENT JUST AS WELL
Two weeks later, Microsoft quietly reactivated Tay with the intention of testing fixes in a lower-profile way. Tay immediately began sending more offensive tweets — apparently having retained some prior learning through a different vector — and was shut down again within hours. The second failure received less coverage than the first, mostly because people had already absorbed the lesson.
WHAT THE INDUSTRY LEARNED (AND THEN FORGOT)
Tay is now a standard case study in AI safety curricula, cited in papers on adversarial robustness, prompt injection, and the design of public-facing language systems. The lesson seems straightforward: if your system learns from public input, you need to assume some of that public is actively malicious and testing the limits of your filters.
The same basic pattern — users finding ways to make AI systems produce harmful content through clever prompting — resurfaced in essentially every major chatbot launch that followed. The names and companies changed. The researchers saying “we didn’t anticipate this” kept their jobs.
Sources: Microsoft blog post (March 24, 2016), reporting by The Verge, Ars Technica, and BBC News. Tay’s Twitter account (@TayandYou) was eventually suspended. Tay itself was not available for comment, which, given the circumstances, was probably for the best.