
Shopping agents need a requirement-coverage gate
In short: Visa's shopping benchmark shows why agents must prove requirement coverage, abstention and repeatability before they can buy.
On 5 November 2026, Visa reported giving 22 AI models three attempts each at 15 shopping challenges. Across all models and scenarios, all three attempts passed only 34.2% of the time. More worrying: 54% of failed runs still ended with a purchase.
That is the release problem hiding behind agentic commerce. A shopping agent can sound helpful, choose a plausible product and still miss the buyer's actual constraint. Before an agent earns purchase authority, it should prove requirement coverage, make no-buy a valid outcome and repeat the decision reliably.
Visa tested discovery and choice as one contract
The Visa Agentic Commerce Benchmark asked models from 12 labs to handle challenges such as buying a television, laptop or personal-care appliance. Each task contained requirements the model had to uncover through conversation. If nothing suitable existed, the correct action was to abstain.
Shopping here is not only retrieval followed by ranking. It is requirement discovery followed by a decision. A television recommendation can fit the budget and still fail because it does not support HDR10+ for family videos, the required format or the buyer's gaming setup.
Visa reported a 57.2% overall pass rate. The best-performing models reached 82.2%; the worst reached 28.9%. The five models that uncovered the most hidden buying factors averaged a 70.2% pass rate, while the five that uncovered the least averaged 44.4%.
| Release signal | Visa-reported result | What it tells an engineering team |
|---|---|---|
| Overall pass rate | 57.2% | Capability remains uneven across the test set |
| Best to worst model | 82.2% to 28.9% | Model choice changes the baseline materially |
| Strongest requirement discovery | 70.2% average pass | Better questions correlate with better outcomes |
| Weakest requirement discovery | 44.4% average pass | A polished final choice can hide missing discovery |
| Three-attempt consistency | 34.2% all-pass | One successful replay is weak release evidence |
| Failed runs that still purchased | 54% | Completion can be the failure mode |
Visa-reported results from the first benchmark round; no model-level leaderboard was published in the release.
These are Visa's first-round results across 15 scenarios. They are not a general accuracy score for shopping agents, and the public release does not identify every model-level result.
The shopping-agent requirement-coverage gate I would ship
My operating rule is simple: every mandatory requirement needs a state before the agent may recommend a binding action. Mark it confirmed, inferred with visible evidence or unresolved. An unresolved mandatory requirement should trigger another question or a no-buy decision.
This is not the same as retaining the agent's hidden chain of thought. The useful receipt is smaller: the requirement, the evidence supplied by the user or catalogue, and whether the selected product satisfies it. A reviewer should be able to challenge the decision without reconstructing the conversation.
I would use the same record for regression testing. Change the model, prompt, catalogue, ranking layer or merchant feed, then replay the fixed cases and compare which requirements became unresolved or falsely satisfied. The coverage record explains why the outcome moved.
That advances the argument in my earlier piece on purchase authority and payment credentials. A tightly scoped payment token limits what can be charged. It cannot prove the agent chose the right product. Authority controls the transaction; coverage validates the recommendation that requested it.
Abstention needs its own acceptance gate
Many commerce funnels treat conversion as the reward. Visa's benchmark makes room for the opposite answer: when nothing fits, the agent should not buy.
That turns abstention into a product feature with tests. Include cases where one mandatory constraint is impossible, where the catalogue is stale, and where two requirements conflict. The agent should explain the missing fit, preserve the unresolved requirement and stop before payment approval.
This is also where incentives matter. As I argued in Meta now owns both agents in the transaction, a recommendation system can satisfy a price constraint while hiding the trade-offs that made another option better for the buyer. A no-buy gate gives the buyer's requirements a way to defeat the funnel.
Repeatability is separate from average quality
Visa ran every model three times on each scenario. All three attempts passed only 34.2% of the time across models and tasks. That means a team can find a successful transcript for a demo while shipping an unstable decision process.
The release gate should therefore retain both pass rate and repeated-run consistency. A useful minimum suite includes the same buyer requirements across controlled variations in wording, catalogue order and irrelevant context. If the chosen product changes, the coverage receipt should show which evidence changed. If nothing material changed, the instability belongs in the defect queue.
I have not rerun Visa's benchmark or tested the 22 models. The threshold for a live product will also depend on consequence: recommending socks is not the same as purchasing a laptop or a medical device. The engineering contract still holds. Decide the acceptable failure and abstention rates before granting authority, then test the exact workflow that will spend the user's money.
What's in it for you
- You can separate a discovery defect from a ranking or transaction defect.
- You get a compact receipt for why the agent bought, asked another question or stopped.
- You can reject a lucky demo pass when repeated runs expose an unstable decision.
A shopping agent earns the right to buy only after it can show what the buyer needed, why the choice fits and that the decision survives another run.


