Coding agents need a refinement budget
In short: A coding-agent refinement budget should continue review only while each cycle adds test evidence, removes a defect or advances acceptance.
On 20 January 2026, a study of 30 ChatDev software-development tasks reported that code review consumed an average 59.4% of all tokens. Input made up 53.9% of the total. The expensive part was not the first pass at the code. It was the agents talking through refinement and verification.
That does not make review waste. It makes review the place where a coding-agent system needs a refinement budget: continue while another cycle produces new test evidence, removes a distinct defect or moves the change toward acceptance. When the same context returns and the test state does not move, longer iteration is not diligence. It is an unpriced loop.
Coding agents spend heavily on refinement
The Tokenomics study mapped ChatDev's phases onto design, coding, code completion, code review, testing and documentation. Code review dominated the average token distribution at 59.4%. Initial coding averaged 8.6%; design averaged 2.4%.
The distribution matters because review repeatedly carries the specification, current code, earlier feedback and revised output back into the model. The study calls the input-heavy pattern a communication tax. That diagnosis still leaves an operating question: which repeated tokens bought a better change?
The paper is preliminary and narrow. It used one multi-agent framework and one GPT-5 reasoning model across 30 tasks. Code completion appeared in only 6 tasks and testing in 12, so those phase estimates rest on smaller samples. I did not reproduce its runs or token accounting.
| Reported measure | Tokenomics study | Eight-model SWE-bench study |
|---|---|---|
| Tasks or benchmark scope | 30 ChatDev tasks | SWE-bench Verified trajectories |
| Models | 1 GPT-5 reasoning model | 8 frontier models |
| Largest reported variation | 59.4% in code review | Up to 30× on the same task |
| Input-token finding | 53.9% overall | Input tokens drove overall cost |
| Does more token use always improve accuracy? | Not tested as the main question | No; accuracy often peaked at intermediate cost |
The studies measure different systems and cannot be treated as one pooled benchmark.
More tokens do not guarantee a better accepted change
A second paper, submitted on 24 April and revised on 2 October 2026, analysed trajectories from eight frontier models on SWE-bench Verified. Runs on the same task varied by as much as 30× in total tokens. Higher consumption did not translate monotonically into higher accuracy; the authors report that accuracy often peaked at intermediate cost and then saturated.
The models could not reliably price their own work in advance either. Their self-predictions had weak-to-moderate correlations with actual consumption, topping out at 0.39, and systematically underestimated it. A task label such as “small bug” or “hard refactor” is therefore a poor budget by itself. The loop needs live evidence from the run.
These results do not establish one universal token ceiling. The studies use different agents, tasks and measurements. Together they support a smaller conclusion: iteration count and token spend are weak proxies for useful progress. A review cycle has to earn the next one.
My refinement receipt for a coding-agent loop
I am proposing this as editorial judgment, not claiming a measured Andy, Trigent or client deployment. I would give each review cycle a receipt with six fields: cycle number, input and output tokens, new failing-to-passing tests, distinct defects removed, repeated context share, and acceptance state.
The receipt gives the next-cycle decision observable inputs. Continue when the cycle produces a new failing test, makes one pass, removes a named defect or satisfies an acceptance condition. Reset context when the agent is carrying a large history but repeating an earlier diagnosis. Stop and escalate when two cycles add cost without changing tests, defects or acceptance.
That rule is deliberately about evidence, not a round token number. A difficult change may need five productive cycles. A simple change may stall after one. The budget controls unproductive refinement while preserving the review that catches real errors.
It also advances an earlier point. I have argued that task cost belongs to the harness and that generated code moves pressure into human review queues. The refinement receipt sits inside the agent run. It asks whether the automated review loop is producing evidence before that work reaches a person.
Where a refinement budget holds and where it breaks
The rule works best when acceptance has observable artefacts: tests, static-analysis findings, review comments resolved, benchmark cases passed or a merge decision. It is weaker for exploratory architecture work, where the useful output may be a rejected option or a clarified trade-off rather than a passing test.
Do not turn the receipt into a demand that every cycle change a metric. Some investigations need a bounded diagnostic pass before code moves. Record that purpose in advance and cap it separately. The failure to avoid is an agent interpreting its own continued conversation as proof that progress continues.
The two studies' token and accuracy results were not independently verified in this run. They do give engineering teams a better starting question than “how many tokens should this task need?” Ask what new evidence the last cycle purchased, then decide whether another one deserves a budget.
What's in it for you
- Add tests changed, defects removed and acceptance state beside tokens in the next coding-agent trace.
- Set a reset or escalation rule for cycles that repeat context without changing evidence.
- Compare cost per accepted change only after separating productive review from stalled refinement.
Let a coding agent spend another review cycle only when the last one bought evidence.
