Quantization damage hides in the flips, not the average

ByteByteGo published a clean guide to shrinking language models on September 1. It says dropping to 4-bit might have a large quality cost, then points readers to QLoRA, whose central result is that 4-bit finetuning can preserve 16-bit task performance.

Both statements can be true. The deployment question is not whether the average benchmark score held. It is which answers changed, for which users, at what context length. For a quantized model, I would keep the aggregate score, add paired-output flips, then human-check the slices where a quiet change has an expensive consequence.

Count substitutions before admiring the mean

Microsoft Research's Accuracy is Not All You Need compared compressed models with their 16-bit baselines. Across seven benchmark tasks, the paper reports accuracy differences of 0–2% while some schemes produced up to 13.6% flips: answers moving correct-to-incorrect and incorrect-to-correct.

Those movements can cancel in the final score. If ten answers are lost and ten different answers are gained, aggregate accuracy is unchanged. The product is not unchanged for the users who depended on the lost ten.

The paper is careful about the boundary. A flip is evidence that behaviour changed, not proof that every change becomes downstream harm. If the benchmark closely matches the real task, accuracy may be enough. For a drop-in replacement used across many tasks, the authors argue that distance from the baseline belongs beside capability—and show that flip rate correlates with KL divergence in their experiments.

That gives the average a proper job. It says whether total measured capability moved. It cannot say whether the compressed model preserved the same behaviour.

The failures have addresses

The address can be a language. Cohere's multilingual study evaluated Command R/R+ and Aya 23 models across more than 20 languages. In Japanese, a 1.7% average deterioration across automatic tasks sat beside a 16.0% drop in human judgments on realistic prompts. Non-Latin-script languages were hit hardest in the automatic benchmarks, and difficult work such as mathematical reasoning degraded fastest.

It can also be context length. A separate long-context study tested 9,700 examples, five models and five quantization methods. FP8 and GPTQ-Int8 averaged roughly 0.2% and 0.8% drops, respectively. One 4-bit setting lost 59% on the tested long-context work. The same paper shows why a blanket verdict is lazy: on OneRuler, Qwen-2.5 72B gained 0.6% with BNB-NF4 while Llama-3.1 70B lost 59%.

There is a useful counterweight. Borgersen and Goodwin's Llama 3.3 70B study tested K-quantization importance matrices in English, Norwegian and Malayalam, then evaluated English and Norwegian. After the paper's multiple-comparison correction, that setup did not show disproportionate multilingual harm. The defensible claim is narrower than “quantization hurts non-Latin languages.” Model, method, task and evaluation instrument decide where the damage appears.

QLoRA is not the exemption card

QLoRA is excellent work. It showed how to finetune a frozen 4-bit base through adapters and reported preserving full 16-bit finetuning task performance. But its famous 99.3% number is not a universal retention score. It is Guanaco 65B's performance relative to ChatGPT on 80 Vicuna prompts, rated by GPT-4, with a reported 4.4% confidence interval.

The paper itself reports ordering effects in the GPT-4 judge and only moderate agreement with human rankings. That does not invalidate QLoRA. It invalidates using one precise-looking number as permission to skip an acceptance test on a different model and workload.

Correcting my own gate

In July I published Quantized retrieval needs slice-level evals. The instinct was right: name the workload's sharp edges before swapping the model. The unit was incomplete. Per-slice accuracy is still a mean, and balanced correct-to-incorrect and incorrect-to-correct movement can hide inside a smaller bucket too.

Run the paired acceptance test

Before approving a quantized build, run one baseline-versus-candidate receipt. This is a proposed gate distilled from the studies above, not a deployment result I ran for this article:

1. Freeze the model versions, prompt set, decoding settings and scoring code. Use real workload samples, not only public benchmarks.
2. For gradable answers, record correct-to-incorrect and incorrect-to-correct flips separately. Do not net them into one score.
3. Split results by predeclared risk slices: non-Latin scripts, the hardest reasoning tasks, longest production contexts, rare vocabulary and any workflow where a wrong answer is costly.
4. For free-form output, add blind paired human review or a domain rubric. A changed paragraph is not automatically worse, and exact-string difference is not a quality metric.

Keep latency, memory and aggregate quality in the same receipt. They answer different questions. The decision is not “did quantization work?” It is “did this exact method buy enough capacity without crossing any slice-level behaviour threshold we set before seeing the result?”

Memory and quality need separate ledgers

The ByteByteGo arithmetic—140 GB of 16-bit weights against a 24 or 48 GB card—is a useful explanation of weight pressure. A serving plan needs another line for KV cache: the PagedAttention paper describes a per-request cache that grows dynamically and can limit batch size. Weight quantization may create room for longer contexts; it does not prove the model will remain accurate on them, or that the resulting concurrency fits.

That extends an earlier AndyMental rule: active parameters do not size the cluster. Residency, runtime state and behavioural acceptance are separate budgets. Compressing one does not certify the other two.

Keep the average for capacity-versus-quality tradeoffs; ship only after paired flips and human-checked risk slices show where the changed answers landed.