Coding-agent vendor choice needs a regression test
In short: A coding agent's vendor choice is a versioned build input; regression-test it across repositories, prompts and procurement constraints.
On 3 September 2026, Armature reported 16,893 coding-agent runs across 75 repositories, 1,163 prompt variants and three agents. Claude Code, Codex and Cursor selected the same tool in only 42% of evaluated cells.
That makes coding-agent vendor choice a versioned build input, not a neutral recommendation. My rule is to regression-test the selected dependency across repository, prompt and procurement constraints before allowing implementation.
Armature found that repository context changes vendor choice
1/ Armature's published study kept 5,292 sessions it judged valid, covering 51 codebases and 18 sectors. A Gemini 3.7 Flash instance played the user; another judged validity and recorded the selected tool. These are controlled synthetic runs, not production purchasing decisions.
2/ The same email-provider ask produced four reported winners. Resend led on TypeScript in 55 of 89 runs, SendGrid on Python in 22 of 24, Postmark on Go in 20 of 24, and Azure Communication Services on Java in 22 of 23.
3/ Search behaviour also differed. Armature reports that Codex searched the web in 94% of sessions, Cursor in roughly two-thirds, and Claude Code in about 30%, rising to roughly 80% in newer sandbox categories. A recommendation therefore carries the agent's retrieval path alongside the repository's cues.
A vendor-choice regression test makes the input reviewable
4/ Treat the recommendation like generated code. Pin the agent and model version, repository commit, exact ask, hard requirements, prohibited vendors, procurement rules and allowed evidence sources. Run the same matrix again before a dependency upgrade or model migration.
5/ The acceptance check is not “did the agent install something?” It is whether the selected option satisfies the written constraints, whether the evidence is current, whether rejected alternatives were rejected for true reasons, and whether the implementation can be removed without rewriting the product.
6/ Keep the choice separate from the build. First produce a short decision record with the top options, decisive evidence and unresolved risks. Only an approved choice should unlock package installation, credentials, schema changes or infrastructure provisioning.
The test I would run before accepting an agent-picked dependency
7/ I have not reproduced Armature's experiment or deployed this matrix for a Trigent client. My labelled editorial test is smaller: take one real repository, two equivalent prompts and the team's approved-vendor list; run them across the current and proposed agent versions, then diff the recommendation, citations and requested permissions before any code lands.
8/ Fail the test when a hard constraint disappears, a cited page is stale, the winner changes without a new decisive fact, or the implementation asks for broader authority. A different winner is not automatically wrong. An unexplained winner is not ready to become architecture.
Where Armature's results hold and where they break
Armature publishes its leaderboard methodology and session traces, which makes the reported choices inspectable. It also sells discoverability services to developer-tool vendors and says this study supports that commercial work. The 11,601 unpublished runs were not independently audited here.
The results do not establish vendor quality, market share or universal agent behaviour. They establish the narrower engineering point: repository, prompt, agent and search path can change the selected dependency even when the requested job looks the same.
What's in it for you
- Stop a model migration from silently changing your software supply chain.
- Give procurement and security a reproducible record before credentials or data move.
- Review the decision separately from the generated implementation.
If an agent can choose a dependency, that choice deserves a regression test before it earns authority to build.
