A plausible model explanation is not a root cause
Anthropic's CHIVE found that three activation-reading tools added no predictive lift over the transcript. Close an incident only after a targeted intervention moves the failure.
Anthropic's CHIVE work starts with an uncomfortable distinction: a model can give you a compelling explanation of its behaviour, and the explanation can still fail to predict what happens when the prompt changes.
That is a useful incident-response boundary. A fluent explanation may open an investigation. I would not let it close one until a targeted intervention changes the failure rate. Plausibility is a hypothesis receipt; movement under intervention is the first causal receipt.
CHIVE asks the explanation to predict
CHIVE samples a target model 30 times on a prompt, screens the responses for an unexpected behaviour, then lets an investigator run 5–15 counterfactual experiments. Each experiment changes the prompt, samples again and measures whether the behaviour moves. An independent judge reviews how well the experiments support the resulting explanation.
The explanation itself is not treated as ground truth. That design choice is the point. “The model saw the permissions bypass as legitimate” sounds causal. Removing or rewriting the permission cue and watching the behaviour stay put would say otherwise.
The researchers then turned those measured edits into a prediction task. An agent saw a transcript and a claim about whether one prompt edit would change the target behaviour. Some agents could also make five read-only calls to one of three activation-reading tools: activation oracles, natural-language autoencoders or sparse autoencoders.
None beat the transcript-only baseline in the reported evaluation. Internal-state access added no predictive lift over reading what the model had already said.
The negative result has a boundary
“Interpretability failed” would be the lazy version. The authors do not say that. Their behaviours were mostly simple enough to investigate with clean prompt edits, while real system-card work may target harder internal states without accessible ground truth. A tool can also corroborate a transcript without improving a benchmark score.
The narrower result is more useful: these three tools did not improve counterfactual prediction in this setup. The work neither validates nor invalidates every use of activation evidence. It does remove permission to present a plausible activation story as causal proof by default.
I inspected the Anthropic report, paper and public repository on September 11. I did not run the pipeline. The repository exposes code, datasets and experiment instructions, but the complete reproduction needs model APIs and GPU-backed infrastructure. The operating rule below is my translation for production reviews, not a result CHIVE measured on enterprise incidents.
Make the suspected cause move
When an agent fails, write the proposed cause as an intervention before writing it as the conclusion:
- Name the behaviour precisely: frequency, affected task, model version and harness version.
- Change one suspected cause: remove an instruction, rename the misleading field, replace retrieved evidence, revoke one tool or alter one permission.
- Rerun a fixed prompt set enough times to see whether the behaviour actually moves; keep decoding settings and scoring fixed.
- Record competing explanations and one edit that should leave the outcome unchanged. A convenient story should survive comparison, not only confirmation.
This is not laboratory purity for its own sake. It stops a plausible postmortem from assigning the fix to the wrong layer. If removing a tool changes nothing but changing retrieved evidence collapses the failure, the access-control ticket was never the root-cause fix.
The intervention can still be imperfect. Prompt edits may change several things at once, stochastic behaviour needs repeated samples, and a result on one incident does not generalise automatically. Write those limits into the receipt. The alternative is worse: a paragraph that sounds mechanistic but cannot tell the on-call engineer what to change.
Explanations are leads, not closures
This extends the archive's evaluation rule. Perfect accuracy cannot reveal the algorithm; now the inverse matters too. A convincing mechanism story cannot certify that it produced the observed output.
The gap is prediction. A useful explanation should help you anticipate what happens when the suspected cause is removed, strengthened or replaced. Until it does, keep the incident open and the language honest: “consistent with” is not “caused by.”
Do not close a model incident on the explanation that sounds right; close it on the smallest intervention that makes the failure move.
