Quantized retrieval needs slice-level evals
NVIDIA's 4-bit Nemotron 3 Embed keeps 99% of aggregate retrieval accuracy. An average across 16 public tasks says nothing about which of your query classes absorbed the loss — slice before you swap.
On July 16, 2026, NVIDIA released Nemotron-3-Embed-1B-NVFP4, a 4-bit variant of its embedding model, reporting an average RTEB NDCG@10 of 72.00 versus 72.38 for the BF16 parent across 16 public tasks — and up to 2x the throughput on Blackwell hardware while retaining more than 99% of BF16 retrieval accuracy.
The consensus read: free lunch. Half the memory, double the throughput, a rounding error of quality. Swap it in.
Here is the problem with that read: 99% retention is an average, and averages do not have addresses. A 0.38-point aggregate loss spread across 16 public benchmark tasks tells you nothing about which query classes absorbed it. Quantization error is not uniformly distributed — it concentrates somewhere. Maybe in rare vocabulary. Maybe in long documents. Maybe in the one query type your compliance team cares about most. The fleet-wide number cannot tell you, by construction.
NVIDIA's own model card is more careful than the headline. It recommends validating on a representative sample of your workload before switching, and it flags known issues with this checkpoint family on vLLM 0.23.x and 0.24.x. Read the calibration detail too: quantization-aware distillation used 20,000 samples, and calibration used 512 CNN/Daily Mail query-passage pairs. News prose. Whatever that establishes, it is not fitness for a private enterprise corpus full of policy clauses, product acronyms, and mixed-language tickets. And a hedge on top: both evaluations are vendor-authored, and neither the throughput nor the accuracy claim has been independently reproduced that I have seen.
Where the 1% lives
At work, this exact failure pattern cost me a debugging cycle on a client retrieval pipeline — not with this model, but with an earlier compressed-embedding swap. Aggregate recall on our regression set moved by under a point, so it shipped. Two weeks later, support escalations clustered around one document family: policy pages dense with internal acronyms. The compressed model had quietly collapsed several acronym embeddings toward each other. The average never flinched, because acronym-heavy queries were maybe 4% of the eval set. They were closer to 30% of the queries that mattered.
That taught me the gate I now apply, and it is the rule I would defend: a quantized retriever ships only after slice-level evaluation on the deployment corpus, and the slices are named in advance. Mine are rare policy terms, multilingual queries, acronyms and internal jargon, and long documents — plus one more dimension most teams skip: the downstream cost of each slice failing. A 3-point recall drop on chitchat queries and a 3-point drop on regulatory lookups are not the same event, and an aggregate treats them identically.
Steal this before you touch the swap. Pull your real query logs and build four or five slices of about 50 queries each around your corpus's sharp edges — the acronyms, the non-English tickets, the 40-page documents, the terms that appear in fewer than ten chunks. Run BF16 and NVFP4 side by side, compare per-slice recall@k, and weight each gap by what a miss costs downstream. If every slice holds within your tolerance, take the 2x throughput with a clear conscience — this is a genuinely attractive artifact. If one slice craters, you have found where the 99% headline hid the bill, at the cost of an afternoon instead of a production incident.
An aggregate accuracy number is a press release; a slice table is a deployment decision — quantize after the slices pass, not after the average does.
