It’s been quite a pair weeks for stories about AI within the courtroom. You would possibly have heard concerning the deceased victim of a road rage incident whose family created an AI avatar of him to point out as an impact statement (possibly the primary time this has been done within the US). But there’s a much bigger, much more consequential controversy brewing, legal experts say. AI hallucinations are cropping up increasingly more in legal filings. And it’s beginning to infuriate judges. Just consider these three cases, each of which provides a glimpse into what we are able to expect to see more of as lawyers embrace AI.
Just a few weeks ago, a California judge, Michael Wilner, became intrigued by a set of arguments some lawyers made in a filing. He went to learn more about those arguments by following the articles they cited. However the articles didn’t exist. He asked the lawyers’ firm for more details, and so they responded with a brand new transient that contained much more mistakes than the primary. Wilner ordered the attorneys to present sworn testimonies explaining the mistakes, by which he learned that one among them, from the elite firm Ellis George, used Google Gemini in addition to law-specific AI models to assist write the document, which generated false information. As detailed in a filing on May 6, the judge fined the firm $31,000.
Last week, one other California-based judge caught one other hallucination in a court filing, this time submitted by the AI company Anthropic within the lawsuit that record labels have brought against it over copyright issues. One in all Anthropic’s lawyers had asked the corporate’s AI model Claude to create a citation for a legal article, but Claude included the fallacious title and creator. Anthropic’s attorney admitted that the error was not caught by anyone reviewing the document.
Lastly, and maybe most concerning, is a case unfolding in Israel. After police arrested a person on charges of cash laundering, Israeli prosecutors submitted a request asking a judge for permission to maintain the person’s phone as evidence. But they cited laws that don’t exist, prompting the defendant’s attorney to accuse them of including AI hallucinations of their request. The prosecutors, based on Israeli news outlets, admitted that this was the case, receiving a scolding from the judge.
Taken together, these cases point to a major problem. Courts depend on documents which might be accurate and backed up with citations—two traits that AI models, despite being adopted by lawyers desirous to save time, often fail miserably to deliver.
Those mistakes are getting caught (for now), however it’s not a stretch to assume that in some unspecified time in the future soon, a judge’s decision will probably be influenced by something that’s totally made up by AI, and nobody will catch it.
I spoke with Maura Grossman, who teaches on the School of Computer Science on the University of Waterloo in addition to Osgoode Hall Law School, and has been a vocal early critic of the issues that generative AI poses for courts. She wrote concerning the problem back in 2023, when the primary cases of hallucinations began appearing. She said she thought courts’ existing rules requiring lawyers to vet what they undergo the courts, combined with the bad publicity those cases attracted, would put a stop to the issue. That hasn’t panned out.
Hallucinations “don’t appear to have slowed down,” she says. “If anything, they’ve sped up.” And these aren’t one-off cases with obscure local firms, she says. These are big-time lawyers making significant, embarrassing mistakes with AI. She worries that such mistakes are also cropping up more in documents not written by lawyers themselves, like expert reports (in December, a Stanford professor and expert on AI admitted to including AI-generated mistakes in his testimony).
I told Grossman that I find all this slightly surprising. Attorneys, greater than most, are obsessive about diction. They select their words with precision. Why are so many getting caught making these mistakes?
“Lawyers fall in two camps,” she says. “The primary are scared to death and don’t wish to use it in any respect.” But then there are the early adopters. These are lawyers tight on time or with no cadre of other lawyers to assist with a transient. They’re longing for technology that might help them write documents under tight deadlines. And their checks on the AI’s work aren’t at all times thorough.
The indisputable fact that high-powered lawyers, whose very career it’s to scrutinize language, keep getting caught making mistakes introduced by AI says something about how most of us treat the technology without delay. We’re told repeatedly that AI makes mistakes, but language models also feel a bit like magic. We put in a sophisticated query and receive what appears like a thoughtful, intelligent reply. Over time, AI models develop a veneer of authority. We trust them.
“We assume that because these large language models are so fluent, it also implies that they’re accurate,” Grossman says. “All of us kind of slip into that trusting mode since it sounds authoritative.” Attorneys are used to checking the work of junior attorneys and interns but for some reason, Grossman says, don’t apply this skepticism to AI.
We’ve known about this problem ever since ChatGPT launched nearly three years ago, however the beneficial solution has not evolved much since then: Don’t trust all the things you read, and vet what an AI model tells you. As AI models get thrust into so many alternative tools we use, I increasingly find this to be an unsatisfying counter to one among AI’s most foundational flaws.
Hallucinations are inherent to the way in which that giant language models work. Despite that, firms are selling generative AI tools made for lawyers that claim to be reliably accurate. “Feel confident your research is accurate and complete,” reads the web site for Westlaw Precision, and the web site for CoCounsel guarantees its AI is “backed by authoritative content.” That didn’t stop their client, Ellis George, from being fined $31,000.
Increasingly, I even have sympathy for individuals who trust AI greater than they need to. We’re, in spite of everything, living in a time when the people constructing this technology are telling us that AI is so powerful it needs to be treated like nuclear weapons. Models have learned from nearly every word humanity has ever written down and are infiltrating our online life. If people shouldn’t trust all the things AI models say, they probably need to be reminded of that slightly more often by the businesses constructing them.
The Algorithm
