How to keep multi-step AI agents under control
In short: Give every agent a host-enforced budget of calls, time and spend, allowlisted tools and an explicit stop reason; verify the final state, not its success message.
Anthropic's "Building effective agents", published 19 December 2024, walks through five workflow patterns — prompt chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer — before it gets to the autonomous agent, and when it does, the advice is to add stopping conditions "such as a maximum number of iterations". That ordering is the whole discipline in one sentence: control comes from the host, not from the model's plan.
So here is the rule I use for keeping multi-step AI agents under control. An agent earns each additional decision step inside a budget the host enforces — calls, wall-clock time, spend — against an allowlist of tools with least privilege, and it exits with an explicit stop reason. What matters at the end is the state of the world and the side effects you can verify, not the length of the plan or the sentence "task complete".
Control is a runtime contract, not a prompt
A system prompt that says "stop after ten steps" is a request. A loop that refuses to issue the eleventh call is a control. The difference shows up precisely when the model is confused, and a confused model is the case you are designing for. So the contract lives in the harness: how many model calls, how many seconds, how many dollars, which tools, which arguments are allowed, and what counts as done. Most of these features are, as I argued in when you need an agent at all, workflows with one judgment point — which means most of the contract is ordinary code.
| Control | Enforced by | Stop reason it produces |
|---|---|---|
| Call budget | loop counter in the host | budget_calls |
| Time budget | deadline on the whole run | budget_time |
| Spend budget | token/cost meter per run | budget_spend |
| Tool allowlist + argument schema | tool router | tool_denied |
| Least-privilege credentials | per-run scoped tokens | permission_denied |
| Completion check | outcome assertion, not the model | done / not_done |
| Escalation | approval queue with context | needs_human |
Each row is a stop reason the run can end with; "the model said it finished" is deliberately not one of them.
Side effects are the part you cannot retry casually
Durable progress is where frameworks help and also where they mislead. LangGraph's persistence layer, for example, checkpoints state per thread so a run can resume after an interruption, recover from a failure, pause for a human, or rewind — its docs even carry a troubleshooting entry for checkpoints growing without bound. Persistence proves you can resume. It does not prove that resuming is safe: if the last step before the crash issued a refund, replaying the step issues it again.
So every external write gets an idempotency key derived from the run and the step, the host records the state before and after the call, and a resumed run checks the recorded state before acting. Agent concurrency makes this worse, not better; an abort budget and a single owner per side effect are the only ways I know to keep it honest.
The test I would run before trusting any of it
No fault-injection test was run for this piece, so this is the lab plan, labelled as such. Take a support-ticket agent whose one real side effect is an external write. Interrupt it immediately after that write, resume from the checkpoint, and verify the write happened exactly once. Then inject three faults in turn — a tool timeout, a denied permission, and a tool response containing text that tells the agent to do something else — and check that each produces the matching stop reason rather than a creative workaround. Grade the final state of the ticket system, not the transcript; Anthropic's eval guide draws exactly that line between a transcript and an outcome, and it is the same discipline as evaluating the feature before production.
Injection defences belong in that test and deserve a sober label: allowlists, argument schemas and least privilege reduce what a hijacked step can do; nothing here guarantees immunity. The design goal is that a successful injection ends in tool_denied, not in a refund.
Approvals, traces and the human on the end
Three operational pieces finish the contract. Approvals need context: the queue item should carry the proposed action, the budget consumed and the state diff, because an approver who sees only "approve?" approves everything — and, as independent policy checks argue, the check should not share the model's reasoning. Traces need redaction before storage, since the transcript is where credentials and customer data leak. And human recovery needs a procedure that is cheaper than the incident: a documented way to find the run, read its state records and reverse its writes.
Where this holds and where it breaks
It holds for agents with countable side effects — tickets, records, payments, provisioning — where budgets and outcome assertions are easy to write. It is weaker for research or drafting agents, where "done" is a judgment and the budget is the only hard control you have. It breaks if the tools themselves are not idempotent and cannot be made so; then the correct design is a workflow with a human at the write step, not a better-prompted agent.
FAQ
What is a sensible first budget?
Small enough to be embarrassing. Ten calls, two minutes and a spend cap below one support ticket's cost; raise each only when the eval shows runs ending in done rather than budget_*.
Does checkpointing make an agent safe to retry?
No. It makes it resumable. Safety to retry comes from idempotency keys and before/after state records on every external write.
Where should the stop reason go?
Into the run record and the trace, as a field — and onto the dashboard. A fleet whose runs mostly end in budget_calls is telling you something a success rate hides.
What's in it for you
- A seven-row runtime contract — budgets, allowlist, privilege, completion, escalation — each with the stop reason it must produce.
- A resumable-versus-safe distinction that keeps a replayed checkpoint from repeating a refund.
- A four-fault lab you can run on any agent before it touches production state.
An agent is under control when the host, not the model, decides how many steps it gets, which tools it may touch, and what "done" means in the state of the world.
