← back to the archiveCover illustration for “Shopping agents need a requirement-coverage gate”
ESSAYday 172·3d ago·Published ·by Andy Padia

Shopping agents need a requirement-coverage gate

In short: Visa's shopping benchmark shows why agents must prove requirement coverage, abstention and repeatability before they can buy.

On 5 November 2026, Visa reported giving 22 AI models three attempts each at 15 shopping challenges. Across all models and scenarios, all three attempts passed only 34.2% of the time. More worrying: 54% of failed runs still ended with a purchase.

That is the release problem hiding behind agentic commerce. A shopping agent can sound helpful, choose a plausible product and still miss the buyer's actual constraint. Before an agent earns purchase authority, it should prove requirement coverage, make no-buy a valid outcome and repeat the decision reliably.

Visa tested discovery and choice as one contract

The Visa Agentic Commerce Benchmark asked models from 12 labs to handle challenges such as buying a television, laptop or personal-care appliance. Each task contained requirements the model had to uncover through conversation. If nothing suitable existed, the correct action was to abstain.

Shopping here is not only retrieval followed by ranking. It is requirement discovery followed by a decision. A television recommendation can fit the budget and still fail because it does not support HDR10+ for family videos, the required format or the buyer's gaming setup.

Visa reported a 57.2% overall pass rate. The best-performing models reached 82.2%; the worst reached 28.9%. The five models that uncovered the most hidden buying factors averaged a 70.2% pass rate, while the five that uncovered the least averaged 44.4%.

Release signalVisa-reported resultWhat it tells an engineering team
Overall pass rate57.2%Capability remains uneven across the test set
Best to worst model82.2% to 28.9%Model choice changes the baseline materially
Strongest requirement discovery70.2% average passBetter questions correlate with better outcomes
Weakest requirement discovery44.4% average passA polished final choice can hide missing discovery
Three-attempt consistency34.2% all-passOne successful replay is weak release evidence
Failed runs that still purchased54%Completion can be the failure mode

Visa-reported results from the first benchmark round; no model-level leaderboard was published in the release.

These are Visa's first-round results across 15 scenarios. They are not a general accuracy score for shopping agents, and the public release does not identify every model-level result.

The shopping-agent requirement-coverage gate I would ship

My operating rule is simple: every mandatory requirement needs a state before the agent may recommend a binding action. Mark it confirmed, inferred with visible evidence or unresolved. An unresolved mandatory requirement should trigger another question or a no-buy decision.

This is not the same as retaining the agent's hidden chain of thought. The useful receipt is smaller: the requirement, the evidence supplied by the user or catalogue, and whether the selected product satisfies it. A reviewer should be able to challenge the decision without reconstructing the conversation.

I would use the same record for regression testing. Change the model, prompt, catalogue, ranking layer or merchant feed, then replay the fixed cases and compare which requirements became unresolved or falsely satisfied. The coverage record explains why the outcome moved.

That advances the argument in my earlier piece on purchase authority and payment credentials. A tightly scoped payment token limits what can be charged. It cannot prove the agent chose the right product. Authority controls the transaction; coverage validates the recommendation that requested it.

Abstention needs its own acceptance gate

Many commerce funnels treat conversion as the reward. Visa's benchmark makes room for the opposite answer: when nothing fits, the agent should not buy.

That turns abstention into a product feature with tests. Include cases where one mandatory constraint is impossible, where the catalogue is stale, and where two requirements conflict. The agent should explain the missing fit, preserve the unresolved requirement and stop before payment approval.

This is also where incentives matter. As I argued in Meta now owns both agents in the transaction, a recommendation system can satisfy a price constraint while hiding the trade-offs that made another option better for the buyer. A no-buy gate gives the buyer's requirements a way to defeat the funnel.

rendering diagram…

Repeatability is separate from average quality

Visa ran every model three times on each scenario. All three attempts passed only 34.2% of the time across models and tasks. That means a team can find a successful transcript for a demo while shipping an unstable decision process.

The release gate should therefore retain both pass rate and repeated-run consistency. A useful minimum suite includes the same buyer requirements across controlled variations in wording, catalogue order and irrelevant context. If the chosen product changes, the coverage receipt should show which evidence changed. If nothing material changed, the instability belongs in the defect queue.

I have not rerun Visa's benchmark or tested the 22 models. The threshold for a live product will also depend on consequence: recommending socks is not the same as purchasing a laptop or a medical device. The engineering contract still holds. Decide the acceptable failure and abstention rates before granting authority, then test the exact workflow that will spend the user's money.

What's in it for you

  • You can separate a discovery defect from a ranking or transaction defect.
  • You get a compact receipt for why the agent bought, asked another question or stopped.
  • You can reject a lucky demo pass when repeated runs expose an unstable decision.

A shopping agent earns the right to buy only after it can show what the buyer needed, why the choice fits and that the decision survives another run.

#agents#agentic-commerce#evaluation#shopping#reliability

Sources

  1. investor.visa.com/news/news-details/2026/Visa-Releases-First-of-Its-Kind-Industry-Benchmark-for-AI-Shopping-Almost-Half-of-Test-Runs-Fail
  2. corporate.visa.com/en/products/intelligent-commerce/earning-trust-report.html
← older drop
Agent simulators need a promotion contract
newer drop →
Large language model (LLM): what it is and how it works

related drops

explore all 557 drops →
← back to the archiveday 175