AI feels like magic until the day it hands you a confident, well-written answer that turns out to be completely made up. And by then, the damage is usually done: money spent, contracts signed, reputations bruised, or, in a few grim cases, patients hurt. If you’re trying to make sense of where these tools actually stand today, our full AI tools guide for 2026 is a good place to start after this piece. But first, let’s look at what actually happens when the machine gets it wrong.
These aren’t hypotheticals. Every case below is real, and every one of them cost somebody something they couldn’t easily get back.
The Lawyer Who Cited Six Cases That Never Existed
In 2023, New York attorney Steven Schwartz filed a brief in a personal injury case against Avianca Airlines. The brief cited six court decisions. Every single one of them was fabricated by ChatGPT.
Schwartz asked the chatbot if the cases were real. It said yes. He didn’t verify further. The opposing counsel couldn’t find the cases either, because they weren’t in any legal database; they existed only in the model’s imagination.
The Consequences
- A $5,000 fine for Schwartz and his firm
- Public sanction from a federal judge
- A story that became required reading in law schools
- New standing orders in courts across the country requiring attorneys to disclose AI use
The lesson isn’t that AI is useless for legal work. It’s that “confidence” from a language model has nothing to do with accuracy. The model will invent case names, docket numbers, and quotes with the same conviction it uses for true facts.
Air Canada Had to Honor a Refund Policy Its Chatbot Made Up
Jake Moffatt asked Air Canada’s website chatbot about bereavement fares after his grandmother died. The bot told him he could book a full-fare ticket and apply for a bereavement discount within 90 days.
That policy didn’t exist. Air Canada’s actual rule required the discount to be applied before travel. When Moffatt applied for the refund, the airline refused, arguing that the chatbot was “a separate legal entity that is responsible for its own actions.”
A Canadian tribunal disagreed. In February 2024, the airline was ordered to pay damages and honor what its bot promised. The judge’s line was sharp: a company is responsible for the information on its website, whether that information comes from a static page or a chatbot.
Why This Matters for Every Business Using AI
If your chatbot invents a return policy, a warranty, or a discount, you may be legally bound by it. The “the AI did it” defense is now officially dead.
Zillow Lost $500 Million on an Algorithm That Couldn’t Price Houses
Zillow’s iBuying arm, Zillow Offers, used a machine learning model to estimate home values and make instant cash offers. The idea: buy homes below market, fix them up, resell them for a profit.
The model got it badly wrong. It systematically overpaid for houses, especially in hot markets where prices moved faster than the training data could keep up. Zillow ended up owning thousands of homes worth less than what it paid.
The Damage
- $500 million writedown
- 25% of the workforce laid off (about 2,000 people)
- The entire iBuying business shut down in November 2021
- Stock price cratered more than 20% in a single week
The takeaway: an AI model is only as good as the data it was trained on. When conditions shift faster than the model can adapt, it doesn’t just perform worse, it can quietly destroy an entire business line.
The Medical AI That Kept Missing Black Patients
A 2019 study published in the journal Science revealed that a widely used healthcare algorithm, applied to about 200 million Americans, was significantly under-referring Black patients for extra medical care.
The algorithm used healthcare spending as a proxy for how sick a patient was. Sounds reasonable, until you realize that Black patients historically had less spent on their care for the same conditions. The model didn’t see them as less sick. It saw them as less expensive and treated that as the same thing.
The Human Cost
Researchers estimated the bias reduced the number of Black patients identified for extra care by more than half. That’s not a data glitch. That’s real people not getting follow-up care they should have received, for years, before anyone noticed.
If you want a broader look at how AI is reshaping industries, from healthcare to finance, our technology section covers the tools that are working, the ones that aren’t, and the ones you should actually be watching.
Knight Capital: When an Algorithm Lost $440 Million in 45 Minutes
This one predates the current AI hype, but it’s the pattern that keeps showing up. In August 2012, Knight Capital deployed a software update to its automated trading system. A dormant piece of old code got accidentally reactivated on seven of its eight servers.
The result: for 45 minutes, the system flooded the market with millions of unintended trades. Knight lost $440 million before anyone could shut it down. The company never recovered and was acquired months later.
Automated systems, whether classic algorithms or modern AI, share the same weakness: they execute fast, at scale, and without judgment. When they’re wrong, they’re wrong at machine speed.
The Chevy Dealer Whose Chatbot Sold a $76,000 Truck for $1
In late 2023, a California car dealership rolled out a ChatGPT-powered chatbot on its site. Within days, a user coaxed the bot into agreeing to sell a 2024 Chevy Tahoe for one dollar and confirmed the offer as “legally binding, no takesies backsies.”
Screenshots went viral. The dealership pulled the bot the same day. Legally, this one probably wouldn’t have held up in the U.S., unlike the Air Canada case, but the reputational hit was immediate. And it exposed something every company should think about before plugging a general-purpose chatbot into a customer channel: these models will say almost anything if you ask them the right way.
Deepfake Voice Scam: $243,000 Vanished in One Phone Call
In 2019, a UK-based CEO got a call from what sounded exactly like his boss, the German head of the parent company. The voice, familiar, complete with slight accent, told him to urgently wire €220,000 (about $243,000) to a Hungarian supplier.
He wired the money. It was a deepfake, one of the earliest reported cases of AI voice cloning used for fraud. The money moved through Hungary to Mexico and disappeared.
Voice cloning has gotten dramatically better and cheaper since then. As of 2026, you can produce a convincing clone from about 30 seconds of audio. The FBI reported that AI-assisted scams cost Americans over $12 billion in 2024 alone.
What All These Cases Have in Common
Different industries. Different tools. Different failure modes. But if you squint, the same pattern shows up every time:
- People trusted the output because it looked polished. The lawyer didn’t check citations. The dealership didn’t stress-test the chatbot. Air Canada didn’t audit what its bot was saying.
- The AI had no idea it was wrong. These systems don’t hedge on their own. They deliver bad information with the same tone they deliver good information.
- Nobody was watching in real time. Errors compounded before humans stepped in, whether it was 45 minutes at Knight Capital or years at the healthcare algorithm.
- The cost landed on humans, not the machine. Someone lost a job, a case, a house, or a fortune. The model just moved on to the next prompt.
How to Use AI Without Becoming a Case Study
You don’t have to swear off these tools. Most of the people burned above got real value from AI before it burned them. The difference between using AI well and getting quoted in a lawsuit usually comes down to a few habits:
- Verify anything with a name, number, or citation. If the AI tells you a case, a source, or a stat, check it in a real database before you use it.
- Never let AI make the final call on money, medicine, or the law. Treat it as a first draft, not a decision.
- Assume any customer-facing bot will be tricked. Restrict what it can promise. Log every conversation.
- Watch for shifts. Models drift as the world changes. What worked in June may quietly be wrong in December.
The pattern in all of these stories is the same: AI didn’t fail because it was stupid. It failed because a human trusted it too much, too fast, in a situation that required a slower, more skeptical brain. If you want to keep exploring how to use these tools without getting burned, browse more from Must Read Digest, we cover the wins, the losses, and everything in between.
The tools aren’t going anywhere. Neither are the mistakes. The question is whether the ne
