LLM findings need an expiry coordinate
In short: Behavioural claims about LLMs should carry the tested model, configuration, protocol and date, because published findings can decay.
On 2 November 2026, Stanford researchers Owen Queen and James Zou reported that 44% of 50 widely cited LLM findings had expired under their fitted September 2026 results. Another 34% had weakened across a panel of 20 models.
That does not make the old papers useless. It makes their operating context part of the claim. LLM findings need an expiry coordinate: the tested model, configuration, protocol and observation date must travel with any behavioural result used in an engineering decision.
Stanford found that LLM findings decay unevenly
Queen and Zou retested 50 findings across 20 models released from July 2024 to September 2026 by eight developers. They reproduced a central effect, ran the same minimal design across the panel, mapped each result to a relative-effect scale and fitted its change over model release dates.
| September 2026 verdict | Findings | Share |
|---|---|---|
| Expired | 22 | 44% |
| Weakened | 17 | 34% |
| Persisted | 8 | 16% |
| Strengthened | 3 | 6% |
The verdicts describe fitted panel-wide effects, not a pass or failure for every individual model.
Capability failures faded fastest: letter counting, arithmetic in other bases, retrieving a fact from the middle of a long context and fabricating citations. Preferences and judgment errors lasted longer. Judges favouring their own answers, praise following a user's preference and non-uniform random-number generation were the three strengthened effects.
Among the 22 claims with estimated expiration dates, the median was 14.7 months from publication. That is not a universal shelf life. It excludes findings that had not expired, and the authors warn that early dates rest on fewer model observations.
A panel verdict can hide the model you actually run
The citation example is the useful warning. Walters and Wilder's 2023 study reported that ChatGPT fabricated 55% of cited works in generated literature reviews. Queen and Zou reproduced 59% with GPT-3.5 Turbo, then measured about 1% for Claude Sonnet 5.5 and 0% for GPT-6.1 Sol.
Yet Qwen3.7-Flash fabricated 69% of references under the same test. The panel-wide finding can expire while one candidate model still fails worse than the historical baseline. “Recent models fixed this” is therefore the wrong procurement statement. The named model, configuration and task still need a local result.
The expiry coordinate I would attach to every eval claim
I did not rerun the 1,000 model-claim combinations or independently reproduce the Stanford analysis. This is labelled editorial judgement, not a Trigent or client deployment.
For any behavioural claim entering an eval plan, I would record six coordinates: model and version; provider and reasoning configuration; prompt or task protocol; scorer and threshold; observation date; and known model-level exceptions. A paper DOI remains useful evidence, but it cannot restore an unavailable model or reveal a changed default.
This extends the archive's eval-history rule. Versioning the prompt and fixture is still insufficient when the research premise itself came from another model generation. It also sharpens the habit of checking each sentence against its citation: verify that the cited finding still describes the system you are about to govern.
The Stanford project has real limits. Each claim uses one minimal design. Some retired models required substitutes. Default reasoning configurations differ, and the web report is not a peer-reviewed paper. Those constraints should stay beside the headline percentages.
What's in it for you
- Add model, configuration and observation date to the next behavioural claim in an eval document.
- Re-run a load-bearing research result when the production model or provider default changes.
- Keep per-model exceptions visible instead of inheriting a panel-wide verdict.
Treat every LLM finding as versioned evidence, not a permanent property of the technology.
