
Hallucination: when an AI states something false as fact
In short: A hallucination is a confident, plausible AI output that is false or unsupported by its source, and structural checks catch what prompts cannot.
A hallucination is when an AI language model states something false as fact: content that sounds confident and plausible but is made up or unsupported by the information it was given, such as an invented citation, a wrong figure or a non-existent API. A September 2025 paper by Kalai and co-authors argues that models guess when uncertain because training and evaluation reward guessing.
Hallucination in plain words
Think of a very fluent person who would rather give you a confident wrong answer than say "I don't know".
How it works
Models predict the next token from patterns. When the answer is not well represented in training data or context, they still produce the most plausible-sounding continuation.
Kalai and co-authors trace the habit to scoring. In their words, training and evaluation procedures "reward guessing over acknowledging uncertainty", which turns models into good test-takers. Their proposed fix is to change how existing benchmarks are scored.
The problem is older than chat assistants. A survey submitted in February 2022 already covered hallucination across summarisation, dialogue, question answering and machine translation.
Example
As an illustration, asked for the case law supporting an argument, a model cites a judgment with a realistic name, court and year that does not exist. In code, it calls a library function with a sensible name that the library never had.
Where you'll meet it
- Legal, medical and financial answers where errors are costly
- Code using non-existent packages or functions
- Summaries adding details absent from the source
- RAG systems answering beyond the retrieved passages
What it changes for you
For developers: ground answers in retrieved sources, require citations, allow and reward "I don't know", and check facts in code where possible (does the package exist, does the number add up). For product teams, design for verification: show sources and make checking easy. For leaders, hallucination risk decides where human review is mandatory.
Hallucination vs a model being outdated
An outdated model gives an answer that was true at training time. A hallucination is not grounded in any real source.
Common mistake
My view: trying to prompt hallucinations away with "do not make things up". The instruction helps a little, while structural checks do far more: retrieval with citations, groundedness evals and code that validates names, numbers and links.
Related terms
Groundedness · Retrieval-augmented generation · LLM-as-a-judge · Evals · Structured outputs
From the archive: A hallucination rate is not a launch criterion
Catch hallucinations with citations, evals and validating code, because an instruction alone will not.


