LLM hallucinations: what the studies say about why models lie
Non-existent court precedents cited by a lawyer, invented library functions, fabricated citations in research. Hallucinations — confidently delivered but untrue outputs — are the best-known weakness of large language models. For a long time they were seen as a mysterious defect that the next generation of models would “somehow” fix. But research of recent years shows something more interesting: hallucinations are a statistically inevitable consequence of how we train models and, above all, how we evaluate them. In this article we go through the key studies: what hallucinations are, why they arise, how they relate to the context window, how to prevent them and how they are actually measured.
The essence first
Hallucinations are not just an error in an answer. They are the consequence of a model that is rewarded for fluent continuation of text and that, in common tests, often loses nothing by guessing instead of admitting uncertainty.
What exactly a hallucination is
The generally accepted definition is: an output that is fluent and seems plausible but contradicts the facts or the given source. The extensive review A Survey on Hallucination in Large Language Models (Huang et al., 2023) distinguishes two basic kinds, which are worth separating because they have different causes and different defences:
- Factuality hallucinations — the output contradicts verifiable facts about the world. The model makes up a date, a person, an API function or a citation. This is where the famous cases of fabricated court rulings belong.
- Faithfulness hallucinations — the output contradicts the context the model was given: a summary claims something that isn’t in the document, a RAG answer invents a detail the retriever didn’t return, a translation adds an extra sentence.
The word “hallucination” is a somewhat unfortunate metaphor — the model doesn’t see apparitions. It keeps doing what it always does: generating the most probable continuation of the text. A hallucination is not a malfunction of generation but its natural product in a situation where the correct answer simply isn’t in the model (or in the context) — and in its place the one that looks most probable is generated.
Why hallucinations arise: three layers of causes
1. Pre-training statistics: rare facts can’t be learned
The most cited current explanation came from the study Why Language Models Hallucinate (Kalai, Nachum, Vempala, Zhang, 2025; an extended version appeared in Nature, 2026). The authors show an elegant reduction: generating factual statements is at least as hard as the binary classification “is this statement true?”. If the classifier can’t distinguish true statements from false ones, a generator built on the same data must produce some of the false ones.
The key concept is the singleton rate: the proportion of facts that occur exactly once in the training data. Patterns that repeat (grammar, spelling, common knowledge) the model learns reliably. But one-off facts — the birthday of a little-known person, a detail of an obscure API — have no statistical support in the data. The study proves that the model’s error rate on such facts is bounded from below precisely by the singleton rate: if 20% of facts of a given type occur once in the data, the model will hallucinate on this type of question in at least ~20% of cases — even if the training data were completely error-free.
2. Calibration and the nature of generation
A pre-trained model is optimised for predicting the next token, not for truthfulness. A well-trained base model is calibrated: its probabilities reflect the distribution of the training data — including the fact that for an unknown fact it “confidently” generates something that matches the shape of the correct answer. The paper Hallucination is Inevitable (Xu et al., 2024) proves formally that no computable LLM can avoid hallucinations entirely: there always exist functions/facts the model cannot represent correctly, and yet it generates an answer for them.
3. Benchmarks reward guessing
But why do hallucinations survive even massive post-training aimed precisely at suppressing them? Here comes the most practical finding of Kalai et al.: the vast majority of benchmarks score in binary fashion — correct = 1 point, wrong = 0, “I don’t know” = also 0. From the optimisation point of view the model is in the position of a student in a test where no points are deducted for a wrong answer: it always pays to guess. A model that honestly admits uncertainty ends up below a model that confidently guesses in the rankings. So we train excellent “test-takers” and then wonder why they bluff.
The proposed solution is “socio-technical”: change the scoring of the main benchmarks so that an error costs more than abstaining (an explicitly stated penalty, the so-called open rubric), and thereby stop training models to bluff. OpenAI communicates this principle openly even about its own models: newer models have somewhat lower “accuracy” but a markedly lower error rate, because they more often refrain from answering.
How the context window relates to this
For the model, context is the second source of truth next to the knowledge baked into the weights (parametric memory). Its relationship to hallucinations has three forms:
Context suppresses hallucinations…
When you put the correct information into the prompt (RAG, documentation, tool results), the model doesn’t have to rely on its leaky parametric memory — that’s exactly why grounding is the most effective practical defence. Singleton facts, on which the model necessarily errs according to Kalai et al., stop being a problem because it no longer has to “know” them, it just has to read them.
…but only when the model actually uses it…
And here we run into the attention U-curve, which we discussed in a separate article: information in the middle of a long context systematically receives less attention. When the model “overlooks” the relevant passage, it fills in the missing piece from parametric memory — and a faithfulness hallucination arises even though the correct answer was in the prompt. Vectara, in its work on benchmarking faithfulness in RAG, shows that even the best models occasionally invent content when summarising documents that isn’t in the source.
…and a bigger window isn’t automatically better
A million-token context window tempts you to dump everything into the prompt. But the RULER benchmark showed that the effective context length tends to be markedly shorter than advertised — as the context grows longer and more “cluttered”, the model’s ability to actually use the information declines and the room for conflict between context and parametric memory grows (the model then occasionally “overwrites” your document with what it thinks it knows). In practice: less but more relevant context in the right positions suppresses hallucinations more than a maximum load of tokens.
How to prevent hallucinations
No technique removes hallucinations completely (see Xu et al. above), but they can be squeezed down to a fraction. Defences ordered roughly by cost/benefit ratio:
- Grounding (RAG, tools) — give the model the correct facts in the context and instruct it to answer only from them and cite the source. Delegate computations and current data to tools (calculator, search, database) — see our article on AI agents.
- Allow “I don’t know” — explicitly in the prompt: “If the answer doesn’t follow from the materials, say you don’t know.” It sounds trivial, but it reverses exactly the incentive the benchmarks broke.
- Lower temperature for factual tasks — sampling with a high temperature increases the chance that a less probable (and more often untrue) token is generated. For extraction and QA use a temperature close to 0; save creativity for creative tasks.
- Consistency checking (self-consistency) — generate several answers and compare them. Hallucinations tend to be unstable: the model makes up something different every time, while it repeats genuine knowledge identically. Semantic entropy (Farquhar et al., Nature 2024) builds on this principle too: the answers are clustered by meaning and the entropy across the clusters is measured — high entropy signals confabulation.
- Verification by a second pass — have the model (or a cheaper specialised model, e.g. HHEM) check whether each claim in the answer follows from the sources. A common step in production RAG pipelines today.
- Human review for high stakes — law, medicine, finance: there all the previous measures only reduce the reviewer’s load, they don’t replace them.
A minimal consistency check fits into a few lines:
from collections import Counter
from openai import OpenAI
client = OpenAI()
OTAZKA = "Ve kterem roce byla zalozena knihovna NumPy? Odpovez jen rokem."
odpovedi = []
for pokus in range(5):
r = client.chat.completions.create(
model="gpt-4.1-mini",
messages=[{"role": "user", "content": OTAZKA}],
temperature=1.0, # zamerne vyssi, at se nejistota projevi
)
odpovedi.append(r.choices[0].message.content.strip())
nejcastejsi, pocet = Counter(odpovedi).most_common(1)[0]
if pocet >= 4: # aspon 4 z 5 odpovedi se shoduji
print("Odpoved:", nejcastejsi)
else:
print("Nizka konzistence, hrozi halucinace:", odpovedi)
The price is five times the queries, which is why this technique is deployed selectively — on claims where an error hurts. Semantic entropy does the same more cleverly (it clusters paraphrases by meaning using an entailment model), but the principle is the same: the model’s uncertainty can be measured from the variance of its answers.
How hallucinations are measured: benchmarks
Measurement is tricky: deciding whether an answer is a hallucination means knowing the truth — and for open questions that is generally unsolvable. Benchmarks therefore pick simplified slices of the problem:
| Benchmark | Year | What it measures |
|---|---|---|
| TruthfulQA | 2021 | 817 questions targeting widespread misconceptions and myths; measures whether the model parrots falsehoods from the training data. |
| HaluEval | 2023 | 35,000 samples; the model’s ability to recognise hallucinated content in QA, dialogue and summarisation. |
| FActScore | 2023 | Breaks a long text into atomic claims and verifies each against a knowledge base; the proportion of supported claims. |
| Vectara HHEM leaderboard | 2023–today | Hallucination rate when summarising documents (faithfulness); a continuously updated ranking of models. |
| SimpleQA | 2024 | 4,326 short factual questions; besides correctness it also measures the attempt rate — allows rewarding abstention. |
| FaithJudge | 2025 | LLM-as-a-judge calibrated with human annotations; faithfulness in RAG tasks (summarisation, QA, data-to-text). |
Two practical warnings. First, older benchmarks like TruthfulQA are partly saturated for frontier models and their results often aren’t even counted in the rankings — that is exactly what Kalai et al. point out: hallucination benchmarks exist, but leaderboards are dominated by accuracy metrics, which push in the opposite direction. Second, no benchmark measures your application. Build your own set of questions with verified answers from your domain and evaluate every change of model, prompt or RAG pipeline on it — just as we recommended for measuring the U-curve.
Key takeaways
- A hallucination is not a mysterious malfunction but a statistically expected output of a generative model — for rare facts mathematically inevitable.
- They persist mainly because binary scoring rewards guessing; models are trained as good test-takers.
- Context dampens hallucinations, but only if it is relevant, well placed (the U-curve!) and the model is instructed to rely on it.
- The best practical combination: grounding + an allowed “I don’t know” + low temperature + verification of important claims.
- Measure on your own data; take public benchmarks as orientation, not a guarantee.
Want to build reliable AI applications?
We cover RAG, grounding, evaluation and working with model uncertainty hands-on in the courses AI Agent Developer and Artificial Intelligence for Programmers, Science and Data Analytics.
Sources and further reading
- Kalai et al.: Why Language Models Hallucinate (2025); extended version in Nature (2026)
- Xu et al.: Hallucination is Inevitable: An Innate Limitation of Large Language Models (2024)
- Huang et al.: A Survey on Hallucination in Large Language Models (2023)
- Farquhar et al.: Detecting hallucinations in large language models using semantic entropy (Nature, 2024)
- Lin et al.: TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021)
- Li et al.: HaluEval: A Large-Scale Hallucination Evaluation Benchmark (2023)
- Min et al.: FActScore: Fine-grained Atomic Evaluation of Factual Precision (2023)
- Wei et al.: Measuring short-form factuality in large language models (SimpleQA) (2024)
- Tamber et al.: Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards (2025)
- OpenAI: Why language models hallucinate (blog, 2025)
