Guardrail benchmarks are graded before the attacker moves
Twelve LLM defenses looked near-perfect against weak tests. Adaptive attacks broke most above 90%. Procurement should demand the attack receipt, not the headline score.
Twelve published defenses against jailbreaks and prompt injection looked close to perfect in their original evaluations. A team of fourteen researchers spanning OpenAI, Anthropic and Google DeepMind changed one condition: the attacker was allowed to see the defense and adapt. Most defenses then exceeded 90% attack success.
That is not a score correction. It is a category correction. The near-zero number described performance against fixed attacks or weak optimisation; buyers read it as robustness against someone trying to get through. The score was a property of the test protocol, not the product.
The benchmark sold the wrong noun
The Attacker Moves Second covers twelve defenses across prompting, adversarial training, model filtering and secret-knowledge techniques. Its researchers tuned gradient descent, reinforcement learning, random search and human-guided exploration to the defense under test. They bypassed all twelve, fully or substantially, and drove attack success above 90% for most, although a majority had originally reported near-zero success.
The paper does not produce a league table. The tasks, models and benchmarks differ across defenses, so its numbers are not directly comparable. Its narrower finding is stronger: a defense cannot support a robustness claim when the attacker in the evaluation never adapts to it.
This matters outside papers. The EchoLeak researchers document a zero-click Microsoft 365 Copilot exploit delivered through one crafted email. The chain evaded Microsoft's cross-prompt-injection classifier, bypassed link redaction with reference-style Markdown, used auto-fetched images and reached a Microsoft Teams proxy allowed by the content-security policy. That incident does not prove every classifier is useless. It proves that a deployed classifier can be one bypassable link in a longer exploit chain.
Ask for the adaptive attack receipt
The useful procurement question is not “what is your detection rate?” It is: what was the attack success rate when the attacker knew the defense and could optimise against it?
The answer needs a receipt:
- System: defense version, model, task suite and the exact policy being protected.
- Attacker: knowledge of the defense, black-box or white-box access, query budget, compute budget and human involvement.
- Outcomes: successful attacks over attempts, with failures adjudicated, plus utility measured in the same configuration.
- Failure path: what the agent can read, change or send when the detector produces a false negative.
The third field prevents a second sales trick. The paper found that model-filtering defenses substantially harmed utility in its AgentDojo evaluation. A vendor can lower attack success by blocking more legitimate work. Security and utility need to appear on the same row.
Architecture changes the consequence
The paper makes an important exclusion. It did not test plan-then-execute systems such as Tool Filter, CaMeL and IPIGuard in most selected scenarios because the attack was guaranteed to fail where untrusted data could not change the program's control flow. It also says those designs cover a limited set of tasks and reduce utility. That is a boundary, not a victory lap.
CaMeL shows the architectural move. A trusted query is turned into explicit control and data flows; untrusted data cannot rewrite the plan, and capabilities constrain where private data may travel. The authors report solving 67% of AgentDojo tasks with provable security under their model. That is evidence for one design on one benchmark, not proof that every agent can be made safe by wrapping it in a planner.
Meta's Agents Rule of Two offers a simpler operating constraint: within one session, an agent should have no more than two of untrusted input, sensitive access and external action. If a task needs all three, Meta calls for supervision at minimum. Meta also says the rule is not a finish line, can still fail and should supplement least privilege and defense in depth.
Architecture does not make prompt injection disappear. It changes what a missed injection can do. A classifier tries to recognise hostile content; capability and flow controls limit the authority available after recognition fails. The first is still useful for raising attacker cost and catching casual misuse. The second keeps one false negative from becoming the whole security boundary.
This later 40-second companion turns that distinction into three deployment rules: allowlist tools, cap spend and tokens, and enforce the limits outside the model.
What I verified, and what I did not
The >90% result needs its attack assumptions and utility costs beside it. Calling filters “UX polish” or assuming capability cuts never degrade performance goes beyond the primary sources: Meta says its rule can still fail, and the paper says plan-then-execute designs cover limited tasks and reduce utility.
I read the full paper and its appendices, not only the abstract. I checked the authors' threat models, the plan-then-execute exclusion and the utility table; then I read the primary EchoLeak, CaMeL and Rule of Two sources. I did not reproduce an attack, test a commercial guardrail or inspect a production agent. The evaluation result belongs to those authors. The procurement receipt and failure-path framing are my synthesis.
The same boundary matters when bug reports become untrusted agent input: detection can reduce noise, but authority has to be constrained even when the input gets through.
A guardrail score is only as strong as the attacker it allowed to move. Demand the adaptive test receipt, then design the system so one false negative does not inherit authority.
