Agent swarms need aggregation evals

Every branch came back green. A research swarm, one worker per subtask, every subtask completed, one final memo — and the memo cited a policy version that had been superseded months earlier, because only one worker had pulled the current document and the merge step quietly preferred the majority that agreed with each other. Nothing failed. The dashboard was entirely honest. The answer was still wrong.

That is the shape of the problem swarms have now, and the numbers being published make it more urgent, not less.

What the swarm vendors are actually reporting

Kimi's own Agent Swarm documentation is specific: up to 300 subagent instances deployed simultaneously, more than 4,000 tool calls in a single task, roughly 4.5× faster than sequential execution by one agent, and a 3× to 4.5× reduction in the minimum critical steps needed to reach a goal. On BrowseComp, accuracy goes from 15.9% with a single agent to 33.3% with the swarm. Moonshot shipped K3 on July 16, 2026, and the swarm feature carries forward from the K2.6 release of April 20. There is a GTC session on the catalogue titled "How We Scaled Kimi K2.5", and a widely forwarded founder masterclass doing the rounds this week — though the version that landed in my inbox is a paywalled preview, so its teaser about "3 reward functions that stop agent swarms from collapsing or cheating" is a headline, not evidence. I'm not treating it as one.

Read Kimi's number again, though, because it is more interesting as a confession than as a boast. Doubling from 15.9% to 33.3% means that with 300 agents and 4,000 tool calls, two-thirds of the benchmark is still answered incorrectly. And the docs are careful about where the gains live: large-scale retrieval, batch downloads, 100+ document processing. Parallelisable search. These are self-reported vendor benchmarks on the workloads that suit swarms best, and I could not verify them independently.

The failure lives downstream of the worker

Set that next to the best empirical work we have on why these systems break. The MAST paper (Cemri et al., arXiv 2503.13657) hand-annotated 150+ multi-agent execution traces — later expanded to 1,600+ across seven frameworks, κ = 0.88 between annotators — and produced 14 failure modes in three buckets: specification and system design (41.8%), inter-agent misalignment (36.9%), and task verification (21.3%).

Do the arithmetic the paper doesn't put in a headline: 58.2% of catalogued failures sit in misalignment and verification — the seam where independently-completed work has to be reconciled, not inside the work itself. The workers are mostly fine. The joins are where it goes wrong. That's my read of their table, and it's the number I'd defend.

Now put the two facts together. Parallelism is a multiplier on the seam. Going from 3 agents to 300 doesn't multiply the difficulty of any single subtask — each worker's job stays about as hard as it was. It multiplies the number of pairwise contradictions, duplicate entities, incompatible date assumptions and unequal-quality citations the orchestrator has to resolve before it can say anything. You bought a 4.5× speedup on the easy part and a combinatorial increase in the hard part.

Correcting myself

Earlier this year I argued that agent count is not a production metric — that "we run 40 agents" tells you nothing a latency chart wouldn't tell you better. I still think that's right, and I had the corollary wrong. I treated count as merely uninformative. It isn't neutral: raising it moves the failure surface. Each agent you add shifts probability mass out of the workers and into the merge, so a swarm scaled without a matching aggregation eval doesn't get less accurate in a way you'd notice — it gets more confidently wrong, faster, with a completion rate that looks better than the quarter before.

So here's the rule I now hold myself to: a swarm does not ship until its eval set contains poisoned inputs, and worker pass-rate is not allowed on the dashboard by itself. If subtask completion is the only number on the wall, the number is decorative.

Poisoned means deliberately constructed, not sampled. Two sources that flatly contradict each other on the same fact. The same company under three name variants. A superseded document sitting next to its replacement, both plausible. Two branches with a shared dependency where only one of them has the current version. Then score these separately from worker success:

• Attribution — does each claim in the output trace to the specific source that supports it, not to a plausible neighbour?
• Contradiction handling — when sources conflict, does the orchestrator surface the conflict, or silently pick the majority?
• Coverage — what did the merge drop? Minority-but-correct findings are the ones that vanish.
• Final-decision quality — graded end to end, by someone who never saw the branch outputs.

At Trigent, the swarm I described at the top was a vendor-security research pipeline for a client — one branch per questionnaire domain, running fine for weeks. The stale-policy answer is what made me sit down and build the poisoned set: a handful of hand-made cases, half an afternoon's work, deliberately contradictory. Worker pass rate on them stayed comfortably high. Final-answer accuracy came in far below it, by a margin big enough that I stopped the rollout. That gap had been invisible for the entire period we were only measuring branches, and the branch numbers had looked good the whole time.

Run this before Friday if you have a swarm in production: take ten of your real tasks, corrupt one source in each, and grade only the final answer. If your worker metrics don't move and your output quality does, you have been measuring the wrong half of your system.

Parallelism is cheap; reconciliation is the product — score the merge or you're grading a swarm on its handwriting.