
Query rewrites need separate acceptance contracts
In short: Instacart split query rewrites into three classes; release each with its own intent invariant, failure budget and recovery path.
On 21 November 2025, Instacart reported that its session-mined query rewrites covered only 50% of search traffic. Its LLM-based replacement pushed coverage above 95%, with more than 90% precision across three rewrite classes: synonyms, broader queries and substitutes.
Those numbers are useful. The more important design choice is the split itself. A synonym, a broader query and a substitute make different promises to the shopper. They should not share one release score just because one model generates all three.
Query rewrites make three different promises
Instacart's first general rewrite prompt exposed the problem. For “1% milk”, the model could return “one percent milk”. The answer is linguistically valid, but it adds little discovery value. The team responded with dedicated prompts for three tasks, backed by few-shot examples and semantic-relevance filters in its query-understanding system.
The three tasks do not fail alike:
| Rewrite class | Acceptance contract |
|---|---|
| Synonym | Preserve the requested intent; reject changes to product, brand or attribute. |
| Broader query | Relax only permitted constraints; reject silent loss of a required constraint. |
| Substitute | Offer a labelled, available alternative; never present a different product as equivalent. |
The promise and acceptance columns are my operating interpretation of Instacart's three reported classes.
A single precision number can hide an expensive mix. A wrong synonym corrupts intent. An over-broad rewrite may turn “gluten-free bread” into “bread”. A substitute may be sensible and still be unacceptable for an allergy, brand requirement or retailer inventory.
Instacart's guardrails still need class-specific error budgets
Instacart's wider architecture is disciplined. Conversion history proposes category candidates, an LLM reranks them with catalog context, and a semantic-similarity guardrail rejects weak category paths. For rewrites, specialised prompts separate synonyms, broader queries and substitutes. The company reports the coverage and precision gains, but not class-specific false-accept rates, online conversion lift or the thresholds behind the filters.
That is a legitimate evidence boundary, not a defect in the engineering post. It also means the public result cannot set your launch threshold. A grocery substitute, a medicine search and an enterprise catalog lookup carry different costs when a constraint disappears.
This is upstream of the problem I covered in debugging RAG retrieval. There, the task is to find the first stage that lost a known passage. Here, the system may alter the user's request before retrieval begins. A perfect retriever can faithfully execute the wrong rewrite.
The acceptance contract I would use
I did not run Instacart's system or test production query logs. My editorial judgement is to release each rewrite class against its own fixture and harm budget.
Each fixture row should keep the original query, inventory snapshot, rewrite class, allowed change, forbidden attribute changes, expected product set and fallback behaviour. Review false accepts and false rejects per class, not only in aggregate. When a rewrite broadens or substitutes, keep the untouched query available and label the change in the interface.
This is the same stop rule I use elsewhere: match the intervention to the observed failure. If synonym coverage is weak, do not loosen the substitute boundary to improve a blended dashboard.
Where separate acceptance contracts hold and break
The rule holds when rewrites can change retrieval scope, product attributes or customer choice. It is especially useful for long-tail queries, where behaviour data is sparse and an LLM's world knowledge contributes more of the interpretation.
It is less useful for spelling normalisation that provably preserves meaning. It also does not replace online measurement. Offline fixtures catch known boundary failures; live search still needs outcome and complaint signals, segmented by rewrite class and retailer context.
What's in it for you
- You can tell whether coverage improved because the system found more intent or merely changed more queries.
- You get a reviewable boundary for synonyms, broadening and substitutes before they reach retrieval.
- You preserve a recovery path when the rewrite is plausible but wrong.
Score query rewrites by the promise they change, not by the model that generated them.


