Judges warn about AI hallucinations; lawyers still cite them

AI hallucinations – In Alabama, a judge sanctioned an attorney for briefs packed with AI-made legal citations. In the months since, courts have kept punishing similar behavior—even as researchers and educators show why people keep deferring to generative systems. The fallout is n

In April, an Alabama Supreme Court action landed like a jolt to the legal system’s idea of due diligence: a lawyer was sanctioned for filing briefs crowded with inaccurate citations generated by AI—complete with numerous references to cases that do not exist.

The story could have ended with correction. After the attorney was informed that one filing contained a made-up precedent, he promised it wouldn’t happen again. But a concurring opinion from a justice captured the problem in plain terms: the lawyer then cited “nonexistent cases at the end of the very next sentence.”.

That same week, at least one other lawyer was sanctioned for continuing to file AI-hallucinated material after being warned not to do so.

The details matter because they show how persistent the failure is. Courthouses keep records, and courtroom proceedings are public—false claims are easier to trace than in many other settings. That clarity has not stopped the same pattern from repeating.

One way to see the scale is through a database maintained by Damien Charlotin. a senior research fellow at the Paris School of Advanced Business Studies (HEC Paris). His list compiles more than 1. 400 cases over the past three years in which courts have addressed AI errors. including filings by attorneys and self-represented litigants. Charlotin says that as recently as last fall. the list appeared to be growing exponentially. but it has since leveled off into a steady flow of exasperated judicial rulings. “For the past two or three months, we have reached a plateau of around 350, 400 decisions a quarter,” he says. Charlotin has also created an AI-powered reference checker called Pelaikan.

The legal system’s willingness to sanction is a big part of why this work is so visible. Yet the same failure—trusting confident text that is wrong—has spilled beyond court.

AI-hallucinated errors have ensnared journalists, software developers, academic researchers, and government consultants, including some who already knew AI can fail. On May 19. the New York Times reported that the author of The Future of Truth. a book about how AI is shaping discourse. acknowledged his text contained more than a half-dozen fabricated or misattributed quotes produced by the technology.

A thread runs through the courtroom cases and beyond them: people keep trusting AI’s answers even when they know these systems can be wrong. So far, that misplaced trust has led to dismissed legal appeals, attorney fines, fired journalists, and software outages.

Researchers now worry the stakes are rising as AI moves deeper into day-to-day professional work. Alan Wagner. an associate professor of aerospace engineering at Pennsylvania State University. describes a basic human tendency behind the problem: “Humans essentially have a tendency to believe that machines have more knowledge than they do. don’t break and are infallible.”.

There’s also a specific kind of persuasion at play—AI can produce answers that sound realistic while being false in a way that humans rarely generate. A study published this past February put participants through an image classification task with guidance they were told came either from humans or AI. In both cases, the guidance was right only half the time. But when participants were told the advice came from AI. those with positive attitudes toward the technology performed worse than those who held less favorable views. No comparable effect appeared when participants were told the advice came from humans. “The results suggested that AI guidance has a quite specific ability to engender biases. ” says Sophie Nightingale. a senior lecturer in psychology at Lancaster University in England. who co-authored the study.

Wagner’s work points toward how dangerous that bias could become outside the office. His research, co-authored by others, was inspired by drone warfare. In experiments. participants were asked to categorize images as civilians or enemy combatants and decide whether to fire a missile at each potential target. A robot then provided feedback on each classification, but that feedback was random.

Even though participants’ initial assessments were mostly accurate, they reversed their views in most cases where the bot disagreed. The scenario was a simulation. but the paper says participants were “shown imagery of innocent civilians (including children). a UAV [uncrewed aerial vehicle] firing a missile. and devastation wreaked by a drone strike.” Colin Holbrook. an associate professor of cognitive and information sciences at the University of California. Merced. says the findings should be read through the seriousness of the setting. “I think that’s the context in which those findings have to be interpreted,” Holbrook says. “These people were really trying. These people thought that it mattered,” he adds. And if the scenario had been real, Holbrook warns, “they would have killed a lot of innocent people.”.

That framing lands uncomfortably close to the reason people keep trusting AI in the first place. Wagner’s colleagues and other researchers argue that the technology doesn’t just generate content—it can change how people decide. Compared with earlier automation tools. today’s AI handles a wider variety of tasks. including generating computer programs and drafting legal briefs. More output means more to check, but it also means users can defer the thinking entirely to the system.

Researchers at the University of Pennsylvania’s Wharton School recently called this “cognitive surrender.” In an experiment run by postdoctoral researcher Steven D. Shaw, participants received item-by-item feedback on a series of tasks and cash rewards for correct answers. Both practices reduced deference to faulty AI, but neither eliminated it.

Educating AI users is another obvious route, yet the results have been mixed. Multiple judges have suggested attorneys should already know not to file AI-generated legal material without checking it, yet hallucinations still surface in court filings.

Lab studies show a similarly limited effect from warning messages. In one recent study. researchers at Boston University “inoculated” students by alerting them that the AI chatbot ChatGPT tends to produce inaccurate summaries of academic sources and struggles with complex math. Participants then completed related tasks using the tool. When participants were warned about the source summaries, verification rates rose significantly for that part of the test. But the warning had no significant effect on the math problems; verification rates remained low. Some participants told the researchers they came in trusting AI’s mathematical abilities. Others said the experiment’s built-in time constraints—designed to mimic real-world deadlines—made them verify less often.

Chi B. Vu. a graduate student in human-AI interaction at BU’s Division of Emerging Media Studies. co-authored that work and wrote to Scientific American that awareness alone isn’t enough. “Our findings suggest that awareness alone isn’t enough,” Vu wrote. “The message wasn’t ignored exactly; it was overridden by competing pressures and trust in certain tasks conducted by [generative] AI.”.

Warnings compete with other signals too: advertising that highlights AI’s potential, and workplace pressures to use it to save time. As AI gets better across more tasks, users may also become less inclined to double-check—meaning they never even encounter the errors that remain.

Nightingale puts it bluntly: “They don’t ever get to the ground truth,” she says. “They don’t have any reason to question it because they carry on in their lives thinking that AI tool is correct—because ‘Why wouldn’t it be?’”

The court sanctions. the database of more than 1. 400 decisions. and the growing research into guidance and verification all point to the same uncomfortable reality: the moment AI gets trusted too easily. errors can move faster than corrections. And for professionals working under deadlines—whether in law. journalism. engineering. or high-stakes operational decisions—the cost of that speed can be immediate.

AI hallucinations legal sanctions Alabama Supreme Court Damien Charlotin Pelaikan cognitive surrender AI guidance bias drone warfare simulation ChatGPT warnings verification