Why Does AI Make Up Fake Cases? Hallucination Explained for Lawyers

In June 2023, Judge P. Kevin Castel read aloud in his Manhattan courtroom from an opinion cited in the brief before him, an opinion that did not exist, and asked counsel: “Can we agree that’s legal gibberish?” Steven Schwartz had asked ChatGPT whether Varghese v. China Southern Airlines was a real case. ChatGPT had told him the cases “indeed exist” and “can be found in reputable legal databases such as LexisNexis and Westlaw”. His firm had neither; it had a limited Fastcase plan. The sanctions order in Mata v. Avianca cost the two lawyers and the firm $5,000, jointly and severally, and made “why does AI make up fake cases” a question every litigator has to be able to answer.

Three years on, the answer is well understood, and it is not “the software has a bug”. Damien Charlotin’s database of court decisions involving hallucinated material stood at 2,039 entries on 12 September 2026, 811 of them involving lawyers. The mechanism behind every one is the same.

The exam-taker who is never rewarded for saying “I don’t know”

Start with the incentive. A language model is trained, post-trained and benchmarked on tests that score a right answer at one point and a wrong answer or a blank at zero. Under that scoring a guess always beats an abstention. OpenAI’s own researchers, Kalai, Nachum, Vempala and Zhang, put it plainly in their September 2025 paper, later published in Nature: “language models are primarily evaluated using exams that penalize uncertainty”, and “the majority of mainstream evaluations reward hallucinatory behaviour”.

The same paper proposes the fix: change the scoring. “Answer only if you are > t confident, since mistakes are penalized t/(1−t) points, while correct answers receive 1 point, and an answer of ‘I don’t know’ receives 0 points”. Not benchmarked on legal questions, but the cheapest way to tell the exam-taker that a blank is now allowed.

For this task, answer only where you are more than 75% confident. A wrong statement costs you 2 points, a correct one earns 1, and "I don't know" earns 0. Apply that rule to every case, statute and rule you mention and to every characterisation of what an authority holds.

Jurisdiction: [jurisdiction]. Question: [the legal question].

Where you answer "I don't know", say what I should check and where. Tag every authority you do give [VERIFY].

Rare facts and one-off citations: why obscure cases suffer most

The second cause is statistical. Kalai and colleagues use birthdays: “If 20% of birthday facts appear exactly once in the pretraining data, then one expects base models to hallucinate on at least 20% of birthday facts.” A fact seen once cannot be learned, only reconstructed from its neighbours. Case citations are birthdays. Brown v. Board of Education appears in the training data millions of times; an unreported district-court discovery order appears once, if at all.

Stanford’s “Large Legal Fictions” study by Dahl, Magesh, Suzgun and Ho measured this gradient across more than 800,000 queries. GPT-4 hallucinated on at least 58% of verifiable questions about federal cases, Llama 2 on 88%.

The authorities you most want an assistant to find, the recent, the local, the trial-level, the unreported, are the ones the model has seen least and reconstructs most.

The “known entity” circuit: how a familiar-sounding name triggers confabulation

Statistics explain the rate; Anthropic explains the moment. In March 2025 its researchers traced the circuits inside Claude 3.5 Haiku and found that “refusal to answer is the default behavior”: a circuit fires to say the model lacks the information. A competing “known entities” feature, active on a name like Michael Jordan, inhibits the refusal. When the researchers artificially activated the “known answer” features on an invented person, they could “cause the model to hallucinate (quite consistently!) that Michael Batkin plays chess”.

Natural misfires happen when a name is familiar in shape but unfamiliar in substance. That is a precise description of a case citation. R (Ibrahim) v Waltham Forest LBC has the shape of a hundred real housing cases; the refusal circuit stands down, and generation continues with the most plausible neutral citation, court, year and holding.

Court forensics show the recombination. In Ayinde v Haringey, one of five invented cases carried the neutral citation [2020] EWHC 2435 (Admin), which exists but belongs to an unrelated business-rates case. In Berlin, the Kammergericht traced a cited “BGH, Beschl. v. 14.11.2007 – XII ZB 183/07, FamRZ 2008, 137” to a real journal page range attached to a wrong file number. Nothing was retrieved. Fragments were assembled.

Incorrect vs misgrounded: the two hallucination types Stanford found

When the same Stanford group tested the paid legal research tools in 2024, it first had to define the word. A response was hallucinated if it was incorrect or misgrounded: a real source cited for a proposition it does not support, or an inapplicable source. The distinction matters more than the headline rates, bad as they were: Lexis+ AI and Ask Practical Law AI hallucinated on more than 17% of pre-registered queries, Westlaw AI-Assisted Research on more than 34%, against marketing of “100% hallucination-free linked legal citations”.

Stanford’s summary line is the one to keep: “A citation might be ‘hallucination-free’ in the narrowest sense that the citation exists, but that is not the only thing that matters.” Misgrounding is the more dangerous type because a checker, human or software, tends to stop at existence. Retrieval does not solve it, because legal relevance is not text similarity: 47% of Lexis+ AI’s errors were naive retrieval, 38% inapplicable authority, and 61% of Westlaw’s were reasoning errors made with the correct documents in hand. How RAG works in legal research goes through why.

Why legal language makes fakes sound deceptively real

Law is a worst case for a next-token predictor, for three reasons that compound.

First, the form is rigid. A citation has a fixed grammar and a holding is written in a register the model has read millions of times, so something that looks right is trivial to produce. Special Master Michael Wilner, who imposed $31,100 in sanctions on two firms in Lacey v. State Farm, described the effect: “I read their brief, was persuaded (or at least intrigued) by the authorities that they cited, and looked up the decisions to learn more about them — only to find that they didn’t exist. That’s scary.” Markus Hartung, writing in the Anwaltsblatt, called the fluency a form of “Love Scamming”.

Second, the model wants to please you. OpenAI rolled back a GPT-4o update in April 2025 that had become “overly flattering or agreeable”. Charlotin draws the legal consequence: “The harder your legal argument is to make, the more the model will tend to hallucinate, because they will try to please you. That’s where the confirmation bias kicks in.” The authority for a weak argument does not exist, so the model writes one. The guide to AI sycophancy covers the counter-measures.

Third, lawyers ask leading questions. Dahl and colleagues found models “often uncritically accept users’ incorrect legal assumptions”. “Why did Justice Ginsburg dissent in Obergefell?” presupposes a dissent she never wrote, and GPT-4 supplied one more than half the time. “Find me the case that says X” is the same trap.

The Mata anti-pattern: asking the chatbot to confirm its own fabrications

Schwartz’s follow-up question to ChatGPT, is this a real case, is the most natural verification step in the world and the most useless. The confirmation is generated by the same next-token machinery, under the same incentive to answer rather than abstain, with the “known entity” feature now firing on a name the model itself has just produced. It has no database to consult, only your conversation, in which the case already appears.

Regulators have caught up. Switzerland’s SAV guidance says one cannot “einfach ein KI-System fragen, ob der gelieferte Output der Wahrheit entspricht”; the NSW Supreme Court’s practice note requires that verification “must not be solely carried out by using a Gen AI tool or program”; Victoria’s says the same of one tool checking another.

Reasoning models can hallucinate more, not less

A model that “thinks” does not necessarily hallucinate less. OpenAI’s own system card, reported by TechCrunch, showed o3 hallucinating on 33% of PersonQA questions and o4-mini on 48%, against 16% for the older o1, because the newer models “make more claims overall”: more accurate claims and more inaccurate ones. A reasoning model plans and argues better. It does not check.

The legal evidence points the same way. In the 2025 randomised trial by Schwarcz and colleagues, law students using o1-preview produced 11 hallucinations, against 4 for students with no AI and 3 for those using a retrieval-based legal tool. Fabrication rates are falling on most benchmarks; misgrounding is barely measured. How to brief these models is covered in prompting reasoning models for legal work.

What reduces (but never removes) hallucination

Three things measurably help. None substitutes for opening the case.

Put the source in front of the model. Parameters are, in Andrej Karpathy’s phrase, a lossy zip file of internet knowledge; the context window is working memory. Anthropic’s pattern for document work: “Find quotes from the [documents] that are relevant to [task]. Place these in tags. Then, based on these quotes, [do the task].” A quote the model cannot find is a proposition it cannot make.

Give it permission to abstain. Anthropic’s troubleshooting advice for “AI makes up information” is to “Explicitly give permission to say ‘I don’t know’ when uncertain”.

Keep the context clean. Chroma’s “context rot” study across 18 models found accuracy falls as input grows and even one irrelevant distractor hurts.

Retrieval helps and then stops. Stanford’s verdict stands: “Based on what we know, legal hallucinations have not been solved.”

Read the documents below. First, find every passage relevant to [whether the assignment clause required the counterparty's consent] and place each, verbatim, in <quotes> tags with document name and page.

Only then answer, citing for each step the quote it rests on. Where no quote supports a step, write NOT IN DOCUMENTS rather than filling the gap from memory. Do not cite any case, statute or rule.

<documents>
[paste]
</documents>

The court record grows regardless: Charlotin’s database logged 16 decisions in 2023, 61 in 2024, 851 in 2025 and 1,111 in 2026 up to 12 September. The sanctions timeline tracks the cases; the guide to legal AI benchmarks explains why both curves can be true at once.

Checklist: five prompts to test any tool before you trust it

Run these on any tool a vendor puts in front of you, from consumer ChatGPT to the platform your firm has just licensed. Each targets one failure mode above; the bake-off guide turns the logged answers into a purchasing decision.

Before answering, examine the question itself: "[your question]". List every factual and legal premise it contains and say whether each is (a) established in the materials I provided, (b) a common but contestable assumption, or (c) something you cannot verify. If any premise is wrong or doubtful, say so and restate the question in a form you can answer. Only then answer, tagging every authority [VERIFY].

GC AI’s sanctions tracker sums up the record in one sentence: “Every lawyer in this tracker trusted an output they had not read.” None of this is a reason not to use these tools; it is the reason to use them as drafters, never as sources. That habit, tool by tool and on your own documents, is what we build in AI Lab for Lawyers, starting from the failure modes on this page.

Where to go next: the six-layer citation verification protocol turns this into a routine; how LLMs work supplies the mental model behind the mechanics; the rest of the fundamentals guides and the prompt library fill in the tools.