Temperature zero is not a determinism guarantee

There is a genre of AI-build tutorial that closes with a reassuring line: set temperature to 0, and the same input always produces the same output. A guest tutorial doing the rounds this weekend — a pitch red-flag scorer wiring Gemini 2.5 Flash, Tavily, and Supabase together — sells exactly that: temperature 0 means "same verdict, every time." It is a lovely promise. It is also false, and the numbers on how false have been public for ten months.

Horace He and colleagues at Thinking Machines Lab ran the experiment last September: 1,000 completions at temperature 0 from one model (Qwen3-235B), one prompt. They got 80 distinct outputs. The runs stayed identical through token 102 — "Feynman was born on May 11, 1918, in" — then forked: 992 continued "Queens, New York," eight broke to "New York City." Temperature 0 did its job perfectly. The answer still split eighty ways.

The consensus reading of temperature 0 is "determinism." That is a category error. Temperature 0 is a sampling setting — it tells the model to take its highest-probability token instead of rolling dice. It says nothing about whether the probabilities themselves come out identical run to run. On a hosted API, they don't. He's team traced the cause exactly: server load changes the batch size your request lands in, the inference kernels are not batch-invariant, so the same request walks a different numeric path and can round to a different winning token. You didn't touch your prompt. Someone else's traffic changed your answer.

Here is the claim I'll put my name on. In these stacks the model is the one component that never promises consistency — the only deterministic layer is the cache in front of it. The tutorial contains the honest version of its own promise and doesn't notice it: the Supabase cache. A cached verdict genuinely is the same every time, because it isn't calling the model — it's returning a row. The rulebook and the fixed output schema narrow variance too. But "same verdict every time" is a property of the cache, the rulebook, and the schema working around the model — not something you bought by typing temperature=0.

I have watched this bug get misdiagnosed. At Trigent we had a client eval harness that pinned temperature to 0 and still showed run-to-run drift on a fixed prompt set — a handful of cases flipping pass/fail between nightly runs. The reflex was to blame the prompts: the eval set is flaky, tighten the wording. We spent real hours hardening prompts that were fine. The drift was the serving batch, exactly as He describes — nothing in our harness had changed, only the load on the endpoint at 2am versus 2pm. Pinning temperature felt like control. It was theatre.

One thing the tutorials skip, and it's the other half of the honesty: if your consistency comes from the cache, your correctness now rides on cache freshness. That pitch-scorer does live "reality checks" — pay ranges, market facts — then caches the result. With no TTL, it will re-serve last week's number as this week's research, deterministically wrong. Determinism and staleness are the same coin: the property that makes the cache consistent is the property that lets it lie.

So the implication if you ship an LLM feature that must be reproducible: stop treating temperature=0 as a guarantee and start architecting the consistency. If you truly need identical outputs — a scoring rubric, a compliance check — put a cache with an explicit key in front of the model and a TTL matched to how fast the underlying truth moves. If you need to prove end-to-end determinism, batch-invariant kernels exist but cost roughly 1.6x–2.1x on inference. Determinism is a bill, not a flag.

What's in it for you

I ship these while I'm still in the mess of them, so the finding reaches you cheap. From this one:

• A bug you'll stop misdiagnosing. Run-to-run eval drift at temperature 0 is the serving batch, not your prompts — so you can quit rewording prompt sets that were already fine.
• A test you can run today. Fire one request a few hundred times and diff the outputs; where they come back identical is where your real determinism actually lives.
• A design rule. If you truly need reproducible outputs, architect it — a cache with an explicit key and a TTL — instead of trusting a sampling flag to hand you a guarantee it never made.

Temperature 0 buys the model's most confident answer, not the same answer twice — put the determinism in a cache with a TTL, because the model was never going to hand it to you.