A hallucination rate is not a launch criterion
A single hallucination percentage hides the product decision. Consequence, detectability, recovery and propagation determine whether an AI failure is tolerable.
“The feature hallucinates 3% of the time. Should we ship it?” sounds like a model-quality question. It is really four product questions hiding behind one percentage.
Marily Nika's useful AI PM prompt separates consequence, detectability and recovery. I would add propagation: can the wrong output remain local, or can it trigger another person, system or decision?
The resulting launch rule is less tidy than a universal accuracy threshold, but much more honest.
Start with the failure, not the average
A nonexistent restaurant in a brainstorming list is not the same failure as a nonexistent legal precedent. An incorrect meeting title is not the same failure as an invented dosage. Even within one product, “3% hallucination” can combine several categories with radically different costs.
I would ask for the failure table before the aggregate score:
| Failure class | Consequence | Can the user detect it? | Can it be undone? | Can it propagate? |
|---|---|---|---|---|
| Weak suggestion | Low | Usually | Yes | No |
| Invented factual claim | Medium to high | Sometimes | Sometimes | Yes |
| Wrong autonomous action | High | Often too late | Costly | Yes |
That table changes the work. The team may discover that most errors are harmless while one narrow slice is unacceptable. The right response is then to constrain that slice, not wait for a model-wide score to move from 97% to 98%.
Detectability changes the quality bar
The most dangerous answer is not always the least accurate one. It may be the answer that looks plausible to the person least able to check it.
Sources, provenance and visible uncertainty can improve detectability, but they do not make every failure safe. A citation must resolve to the claim it appears to support. A confidence label must be calibrated on relevant cases. Telling users to verify everything shifts the product's quality burden onto the people who came because they could not verify it themselves.
This is why expert-facing systems can require a higher bar, not a lower one. The product may be operating exactly where the user lacks independent knowledge.
Recovery is a designed feature
Retry, edit, undo, approval, escalation and deterministic fallback are not polish around an AI feature. They determine how much imperfection the product can tolerate.
If an assistant drafts text that remains visibly editable, recovery may be cheap. If an agent sends a message, changes a record or moves money, the identical model error has crossed into a different product. Human confirmation can help, but only if the human sees the consequential fields and has enough context to challenge them.
I would also test the failure after it leaves the model. Can a fabricated number be copied into a board deck? Can a wrong category trigger an automated workflow? Can a correction reach downstream systems, or does the original error persist in an audit trail?
Launch can be conditional
Ship/no-ship is often the wrong binary. A team can ship a narrower action set, require confirmation for high-consequence cases, route uncertain outputs to a person, expose sources, add an undo window or fall back to a deterministic flow.
Write those controls beside the failure table. Then name the residual risk owner. If nobody owns the remaining 3%, the launch decision is merely an accuracy number looking for someone else to absorb its consequences.
Do not ask whether 97% is good enough; ask what the other 3% can do, who can see it and how the product gets back to safety.
