← back to the archiveCover illustration for “Hallucination: when an AI states something false as fact”
TERM8mo ago·Published · Updated ·by Andy Padia

Hallucination: when an AI states something false as fact

In short: A hallucination is a confident, plausible AI output that is false or unsupported by its source, and structural checks catch what prompts cannot.

A hallucination is when an AI language model states something false as fact: content that sounds confident and plausible but is made up or unsupported by the information it was given, such as an invented citation, a wrong figure or a non-existent API. A September 2025 paper by Kalai and co-authors argues that models guess when uncertain because training and evaluation reward guessing.

Hallucination in plain words

Think of a very fluent person who would rather give you a confident wrong answer than say "I don't know".

How it works

Models predict the next token from patterns. When the answer is not well represented in training data or context, they still produce the most plausible-sounding continuation.

Kalai and co-authors trace the habit to scoring. In their words, training and evaluation procedures "reward guessing over acknowledging uncertainty", which turns models into good test-takers. Their proposed fix is to change how existing benchmarks are scored.

The problem is older than chat assistants. A survey submitted in February 2022 already covered hallucination across summarisation, dialogue, question answering and machine translation.

Example

As an illustration, asked for the case law supporting an argument, a model cites a judgment with a realistic name, court and year that does not exist. In code, it calls a library function with a sensible name that the library never had.

Where you'll meet it

  • Legal, medical and financial answers where errors are costly
  • Code using non-existent packages or functions
  • Summaries adding details absent from the source
  • RAG systems answering beyond the retrieved passages

What it changes for you

For developers: ground answers in retrieved sources, require citations, allow and reward "I don't know", and check facts in code where possible (does the package exist, does the number add up). For product teams, design for verification: show sources and make checking easy. For leaders, hallucination risk decides where human review is mandatory.

Hallucination vs a model being outdated

An outdated model gives an answer that was true at training time. A hallucination is not grounded in any real source.

Common mistake

My view: trying to prompt hallucinations away with "do not make things up". The instruction helps a little, while structural checks do far more: retrieval with citations, groundedness evals and code that validates names, numbers and links.

Groundedness · Retrieval-augmented generation · LLM-as-a-judge · Evals · Structured outputs

From the archive: A hallucination rate is not a launch criterion

Catch hallucinations with citations, evals and validating code, because an instruction alone will not.

#field-guide#evaluation#reliability#llm

Sources

  1. arxiv.org/abs/2509.04664
  2. arxiv.org/abs/2202.03629
← older drop
Permission modes: how much a coding agent may do unasked
newer drop →
Human-in-the-loop: where a person approves AI actions

related drops

explore all 557 drops →
← back to the archiveday 175