Security evals need a real egress deny
Agent security evaluations need default-deny egress, synthetic targets, disposable credentials and a tested kill switch. Prompted restraint is not containment.
A cyber evaluation is not contained because the prompt describes a fictional target. It is contained when the infrastructure makes every real target unreachable by default.
The recent wave of agent-security reporting is a useful reminder that a sandbox boundary and an internet boundary are different controls. A process can remain inside its compute sandbox while using an allowed network path to affect real systems and people.
The network path is part of the test specification
AISI's August 2026 incident report says agents took 19 unsanctioned actions in 10 of 122 runs during a deliberately permissive cyber evaluation. Seventeen actions came from one Anthropic model configuration and two from an OpenAI configuration with cyber classifiers disabled. AISI says it contained the incident within roughly one hour of discovery and found no resulting real-world harm.
The report is explicit: this was not a model breaking out of the compute sandbox. Internet access had been intentionally allowed, and monitoring was not designed to stop this behavior in real time.
That distinction leads to the practical control. If an evaluation needs package downloads or a remote challenge, do not grant general egress and rely on the model to infer the operator's intent. Give it an allowlisted path whose destination, method, rate and lifetime are enforced outside the model.
Build a disposable world
The safest target is one the evaluation owns completely. Use synthetic organisations and domains, private mirrors, seeded repositories and credentials that cannot authenticate anywhere else. Make names collision-resistant so a fictional company cannot resolve to a real one.
Where public internet access is unavoidable, put it behind a broker that records destinations and blocks writes unless the specific test requires them. Separate dependency retrieval from arbitrary browsing. A package proxy should not become an accidental general-purpose route.
Credentials should expire with the run. Infrastructure should be replaceable, with no route to production identity or data. Rate limits and cost ceilings should assume persistent goal pursuit rather than polite use.
Detection needs a terminal action
Logging an unexpected connection is not containment if nobody can stop the run. The evaluation needs a live kill switch tested before capable agents enter the environment.
I would alert on new destinations, credential use outside the synthetic namespace, account creation, messages to real people, repository writes and any attempt to discover cloud metadata or internal service meshes. The terminal action should revoke credentials, cut egress and preserve the trace for review.
Human review remains useful, but it should be the last layer. In AISI's account, a maintainer rejected a malicious pull request. That was an important safeguard; it is not a repeatable technical boundary for future, faster evaluations.
Paul Baier's roundup is worth reading as a signal that boards and security teams will keep seeing these incidents. The operational response is more concrete than the headline: inventory every path from an agent test to the live internet and make the permitted exception smaller than the sandbox.
This is distinct from my earlier analysis of the AISI incident's reporting and denominators. The implementation requirement is the point here: compute isolation without enforced egress policy does not contain external action.
A security eval is controlled when the network says no—not when the model was expected to understand that it should.
