
Quantization: shrinking a model by using fewer bits
In short: Quantization stores model weights in fewer bits, such as 4 instead of 16, so a model needs less memory and can run faster, with some loss of accuracy.
Quantization is reducing the numerical precision of a model's weights, and sometimes its activations, for example from 16-bit floats to 4-bit integers, so each weight uses fewer bits and the model takes less memory and can run faster. The GPTQ paper from October 2022 quantized models with 175 billion parameters down to 3 or 4 bits per weight. Some accuracy is usually lost along the way.
Quantization in plain words
Saving a photo as a smaller JPEG. For most uses you cannot tell the difference; zoom into fine detail and you can.
How it works
Each weight is mapped to a smaller set of values, with scaling factors shared across a group of weights. The method decides how to choose that mapping so the error stays small. GPTQ does it in one shot after training, using approximate second-order information.
In practice you rarely run the method yourself. The llama.cpp quantize tool takes a GGUF model file in a high-precision format such as F32 or BF16 and writes a lower-precision one such as Q4_K_M. Its documentation says the process may introduce some accuracy loss, usually measured as perplexity or KL divergence.
Example
As plain arithmetic, 8 billion weights at 16 bits each need about 16 GB. The llama.cpp table for Llama 3.1 8B, read in November 2026, lists the Q8_0 build at 7.95 GiB and the Q4_K_M build, about 4.89 bits per weight, at 4.58 GiB. The smaller file fits in the memory of an ordinary laptop, which the 16 GB version does not comfortably do.
Where you'll meet it
- Model files labelled Q4, Q8, INT8, FP8, AWQ or GPTQ on Hugging Face
- Local tools such as Ollama, LM Studio and llama.cpp
- Production inference servers serving FP8 or INT8 models
- Capacity planning and GPU purchasing decisions
What it changes for you
For developers, quantization is what makes local and on-device AI practical. For infrastructure teams, it lowers the memory each model copy needs, which changes the capacity plan for the same model. For quality owners, quantised models need their own evals, because perplexity on a benchmark does not tell you how your task behaves.
Quantization vs distillation
Quantization makes the same model use fewer bits. Distillation trains a different, smaller model to imitate a bigger one. They are often combined.
Common mistake
My view: picking the smallest quantisation that runs and assuming it is "the same model". Benchmark the quantised build on your own tasks; a 4-bit model that is fine for chat can be noticeably worse at structured extraction.
Related terms
Open-weights model · Small language model · On-device AI · Inference server · LoRA
From the archive: Quantization damage hides in the flips, not the average
Benchmark the quantised build on your own tasks before you treat it as the same model.


