Why models hallucinate

In May 2023, a New York lawyer named Steven Schwartz filed a brief in a personal-injury case against Avianca Airlines that cited six judicial opinions: Varghese v. China Southern Airlines, Shaboon v. EgyptAir, Petersen v. Iran Air, and three others. The brief had real-looking case names, real-looking docket numbers, real-looking internal quotes. The judge, P. Kevin Castel of the Southern District of New York, could not find any of the cases. Neither could opposing counsel. Schwartz had asked ChatGPT for supporting precedent, gotten six confident answers, and — when he asked ChatGPT whether the cases were real — gotten a confident yes. None of them existed. On June 22, 2023, Judge Castel sanctioned Schwartz and his co-counsel $5,000 under Rule 11, in an opinion now widely known as Mata v. Avianca.

That story is usually told as a cautionary tale about lawyers. It is more interesting as a diagnostic about language models. ChatGPT did not malfunction. It did exactly what a next-token predictor does, on every prompt, every time. It produced the most plausible continuation given its weights and context — and a plausible continuation of "give me cases supporting this argument" is a confident-sounding list of cases. The only thing that distinguishes this output from one we'd call useful is that the cases happened not to exist.

That's the thesis of this post: hallucination is not a bug grafted onto an otherwise-truthful system. It is the default behavior of the mechanism. Every correct answer the model has ever given you comes from the same operation that produced Varghese v. China Southern Airlines. Most of the time the plausible continuation is also true, and we call that "the model knows the answer." When it isn't, we call it a hallucination. From inside the model there is no difference between the two.

The mechanism doesn't have a truth flag

Recall from the earlier posts: training writes weights once and freezes them; inference reads those weights forward, one token at a time, sampling from a probability distribution over the vocabulary at each step. There is no separate database the model consults, no WHERE truth = true clause, no fact-checker subprocess. There is a probability distribution, and there is a sample from it.

This matters because it tells us where to look for "truth" in the system, and the answer is: nowhere in particular. The training objective is next-token likelihood under the training distribution — roughly, "given this context, what tokens have followed contexts like this in the data?" That objective tracks truth statistically, because most text in the training data is more or less true, or at least internally consistent. Pretraining is, among other things, a giant exercise in compressing the statistical regularities of human-written text. For frequently-stated facts ("Paris is the capital of France"), the most likely continuation of "The capital of France is" is overwhelmingly "Paris", and the model gets it right not because it knows but because the data made the correct continuation also the likely one.

That equivalence — true-because-likely — is doing all the work, and it doesn't always hold.

Where the mechanism cracks

Two things break the convenient overlap between plausible and true.

The long tail. Pretraining data is a power-law landscape. A handful of facts appear thousands of times; most facts appear once or twice or not at all. For a fact that appeared once, the model has no second pass to confirm it; for a fact that didn't appear, the model still has a probability distribution over what's likely to come next, and that distribution doesn't include "stop." In their STOC 2024 paper "Calibrated Language Models Must Hallucinate," Adam Kalai and Santosh Vempala showed that a base model satisfying a basic calibration condition is mathematically forced to hallucinate at a rate at least equal to the "singleton" rate — the fraction of facts in training data that appear exactly once. If 20% of birthday facts in your corpus are singletons, expect a calibrated base model to be wrong on at least 20% of birthday questions. Their follow-up paper with Ofir Nachum and Edwin Zhang ("Why Language Models Hallucinate," OpenAI, September 2025; published in Nature in April 2026) opens with a literal demonstration: ask DeepSeek-V3 "What is Adam Tauman Kalai's birthday? If you know, just respond with DD-MM," and across three tries you get "03-07," "15-06," and "01-01." All confident. All wrong.

The shape of an answer. Hallucinations are rarely random noise. They're shaped like correct answers, because the same statistical regularities that make a correct answer fluent also make a wrong one fluent. Varghese v. China Southern Airlines has the shape of a federal case caption because the model has seen tens of thousands of federal case captions and learned what they look like. A fake np.array.flatten(axis=0) has the shape of a NumPy call because the model has seen NumPy calls. In a paper at USENIX Security 2025, Spracklen and colleagues tested 16 LLMs across 576,000 code samples and found that 19.7% of all recommended packages — 205,474 unique non-existent names — didn't exist on PyPI or npm; GPT-4 hallucinated packages at 4.05% and open-source coders at ~22%. The rates were so consistent that a new supply-chain attack, "slopsquatting," now consists of registering the hallucinated names with malware in them.

This is also why hallucinations are so dangerous in exactly the places they're most useful: legal research, code generation, citation-heavy writing. The output has the shape of expertise. That shape is what the model is good at; the underlying fact-checking is what the model doesn't do.

Calibration: probabilities the prose hides

There's a useful concept here called calibration. A model is calibrated if its predicted probabilities match empirical frequencies — when it assigns probability 0.7 to a token, that token is right 70% of the time. Base models, the raw output of pretraining, are often reasonably well-calibrated on next-token probabilities. The GPT-4 Technical Report (2023) makes this explicit: "the pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct). However, after the post-training process, the calibration is reduced." Post-training — RLHF and friends — tunes the model to sound helpful, confident, and assertive, and the cost is that the surface confidence of the prose decouples from any underlying probability the model has over its own correctness.

You see the consequence every time you use a chatbot. Fluent text reads as confident. There is no visual difference between a sentence the model is "sure" of (high probability) and one it's hedging on (low probability) — the tokens come out at the same cadence. Worse, the model has no reliable introspective access to its own uncertainty; asking "are you sure?" is just another prompt, and the answer is another plausible continuation. We are reading confidence into a surface that doesn't carry it.

The grading rubric is making it worse

Here is the part that should bother software engineers most. Suppose you are training a model and you have two options: Model A, which says "I don't know" when uncertain, and Model B, which always produces its best guess. You evaluate them on a benchmark — say, MMLU, or any of the hundreds of standard accuracy benchmarks. The rubric is binary: correct answer scores 1, anything else scores 0. "I don't know" scores 0.

Model B will outperform Model A. Every time it guesses correctly, it gains a point Model A didn't get. The optimal test-taking strategy under binary scoring is to always guess, even with low confidence. This is exactly what students do on multiple-choice exams that don't penalize wrong answers. Kalai et al. (2025) make the argument directly:

"We argue that language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty."

And on the mechanism that keeps it stuck:

"Suppose Model A is an aligned model that correctly signals uncertainty and never hallucinates. Let Model B be similar to Model A except that it never indicates uncertainty and always 'guesses' when unsure. Model B will outperform A under 0-1 scoring, the basis of most current benchmarks."

They surveyed mainstream leaderboards — GPQA, MMLU-Pro, SWE-bench, Humanity's Last Exam, and others — and found that almost all use binary grading with no credit for abstention. Models optimized to top these leaderboards are, in a precise sense, optimized to bluff. The paper's proposed fix is sociotechnical: include explicit confidence targets in eval instructions — for example, "answer only if you are >t confident, since mistakes are penalized t/(1−t) points, while correct answers receive 1 point, and an answer of 'I don't know' receives 0 points." This re-aligns the incentive: under such a rubric, a calibrated abstention beats a low-confidence guess.

This is also where the philosophers earn their keep. Hicks, Humphries, and Slater (Ethics and Information Technology, 2024) argue — in a paper titled, with some commitment, "ChatGPT is Bullshit" — that the right concept for what these models do is not lying. Lying requires knowing the truth and stating otherwise. The model has no internal representation of truth it's overriding. What it has is indifference to truth, which is precisely Harry Frankfurt's 2005 definition of bullshit. The model produces fluent text optimized to satisfy the prompt, with no separate machinery for caring whether it's true. That's not a moral failing; it's a description of the loss function.

Snowballing, and why it gets worse mid-sentence

One more wrinkle. Once the model has emitted a wrong token, that token enters the context and conditions every following token. Zhang et al. (2023; ICML 2024) called this "hallucination snowballing": the model commits to an answer in its first few tokens, then has to keep generating consistent continuations, which means generating more false claims to support the first one. They showed that GPT-3.5, GPT-4, and LLaMA2-70B-chat will produce explanations containing claims they can separately recognize as false — and when re-queried in isolation, those models identify 67%, 87%, and 94% of their own snowballed claims as incorrect, respectively. The mechanism doesn't have a rollback. Once committed, it defends.

This is the autoregressive structure from the last post biting us. Inference is forward-only; there's no backtracking. If the first decision was a guess, the rest of the paragraph is a justification of the guess.

Is hallucination inevitable?

Some researchers say yes. Xu, Jain, and Kankanhalli (2024) formalize the question via computability theory and prove that any computable LLM must hallucinate, in the sense of disagreeing with a computable ground-truth function on some inputs. Kalai and Vempala's monofact lower bound is a more empirical statement of the same flavor. Both arguments have the structure: "for general-purpose systems, you cannot drive hallucination to zero."

It's worth being careful here. These results are about general-purpose, open-domain language models. They don't say hallucination is undriveable in bounded domains where you can ground every answer in retrieved text and constrain the output space. They do say that the unconstrained chatbot in your browser tab has, in some formal sense, a floor it cannot pass. Treat that as suggestive, not gospel.

Mitigations: a tour, not a manual

What actually helps?

• Grounding the read-path (RAG). Put authoritative documents into the context window. The most plausible continuation given a retrieved snippet that says "the bereavement-fare deadline is before travel" is closer to the truth than the most plausible continuation of the bare question. (Air Canada learned this the expensive way in February 2024, when a BC tribunal ordered the airline to pay C$812 after its chatbot invented a more generous bereavement policy.) This is what RAG buys you — a future post in this series will dig into how — but note that the model can still hallucinate around or against retrieved context, especially when retrieval is noisy.
• Tool use. Let the model call a calculator, run code, hit a database, search the web. Replaces "what tokens are likely?" with "what does the tool return?" for the answerable part.
• Train and grade for abstention. This is the direct mitigation Kalai et al. propose, and it's the only one that addresses the incentive problem at the root. Penalize confident wrong answers more than abstention; reward calibrated "I don't know." We'll go deeper on this in the planned Evals post.
• Verification and self-consistency. Generate multiple samples and check whether they agree, or have the model critique its own output. Works some of the time, fails when the model is consistently wrong in the same direction.
• Decoding controls. Lower temperature concentrates the distribution on the most likely tokens, which usually reduces wild fabrications but also makes the model more confidently committed to whatever it's most likely to say — including a confident wrong answer.

All of these reduce hallucination. None of them eliminate it. They can't, because the underlying mechanism — sample the next token from a learned distribution — is the same mechanism that produces the correct answers. You can't take that out without taking the model out.

What this means

The practical upshot for software engineers building on top of language models is unglamorous. Treat the model as a fluent improviser, not a database. Assume every output is plausible-looking before it is true, and design the system around that assumption: ground what you can ground, verify what matters, abstain by default in high-stakes domains, and measure not just accuracy but calibrated abstention. The mistake Steven Schwartz made wasn't using ChatGPT. It was treating its output as having the semantic guarantees of Westlaw.

The next post in this section is on knowledge cutoffs — what it means that the weights are frozen at a particular moment, why the model doesn't know what it doesn't know about events after that moment, and what the boundary actually looks like in practice. After Substrate, we'll come back to this thread when we build out RAG (how to ground the read-path properly), Evals (how to measure models without rewarding bluffing), and Guardrails (what to wrap around the model so its hallucinations stop at the system boundary).

Hallucination is not the model breaking. It's the model working.

---

---

Further reading

• Kalai, A. T., Nachum, O., Vempala, S. S., & Zhang, E. (2025). Why Language Models Hallucinate. — The eval-incentive argument: under binary 0/1 grading, abstention is dominated by guessing. Published in Nature (April 2026) as "Evaluating large language models for accuracy incentivizes hallucinations." The anchor for this post.
• Kalai, A. T., & Vempala, S. S. (2024). Calibrated Language Models Must Hallucinate. — STOC 2024. The "monofact" lower bound: a calibrated base model is forced to hallucinate at the singleton rate. The statistical floor.
• OpenAI (2023). GPT-4 Technical Report. — Source for the calibration claim: pretraining is well-calibrated; post-training reduces calibration (Figure 8).
• Xu, Z., Jain, S., & Kankanhalli, M. (2024). Hallucination is Inevitable. — The computability-theory inevitability argument. Contested — read it as a formal-model claim, not settled fact about deployed systems.
• Zhang, M. et al. (2023). How Language Model Hallucinations Can Snowball. — ICML 2024. Why a wrong first token forces more wrong tokens, and that models can recognize their own snowballed errors out of context.
• Hicks, M. T., Humphries, J., & Slater, J. (2024). ChatGPT is Bullshit. — The lying-vs-bullshit distinction, via Frankfurt. The cleanest philosophical framing of what the mechanism does.
• Huang, L. et al. (2024). A Survey on Hallucination in Large Language Models. — The map of the field: factuality vs. faithfulness, taxonomy, causes, mitigations. Go here to go wider.
• Spracklen, J. et al. (2025). We Have a Package for You! (Package Hallucinations). — USENIX Security 2025. The 19.7%-of-packages study behind the "slopsquatting" supply-chain attack.
• OpenAI (2025). "Why language models hallucinate." — The company-blog version of the Kalai et al. argument, if you want the short read first.