← back to the archiveCover illustration for “Quantization: shrinking a model by using fewer bits”
TERM8mo ago·Published · Updated ·by Andy Padia

Quantization: shrinking a model by using fewer bits

In short: Quantization stores model weights in fewer bits, such as 4 instead of 16, so a model needs less memory and can run faster, with some loss of accuracy.

Quantization is reducing the numerical precision of a model's weights, and sometimes its activations, for example from 16-bit floats to 4-bit integers, so each weight uses fewer bits and the model takes less memory and can run faster. The GPTQ paper from October 2022 quantized models with 175 billion parameters down to 3 or 4 bits per weight. Some accuracy is usually lost along the way.

Quantization in plain words

Saving a photo as a smaller JPEG. For most uses you cannot tell the difference; zoom into fine detail and you can.

How it works

Each weight is mapped to a smaller set of values, with scaling factors shared across a group of weights. The method decides how to choose that mapping so the error stays small. GPTQ does it in one shot after training, using approximate second-order information.

In practice you rarely run the method yourself. The llama.cpp quantize tool takes a GGUF model file in a high-precision format such as F32 or BF16 and writes a lower-precision one such as Q4_K_M. Its documentation says the process may introduce some accuracy loss, usually measured as perplexity or KL divergence.

Example

As plain arithmetic, 8 billion weights at 16 bits each need about 16 GB. The llama.cpp table for Llama 3.1 8B, read in November 2026, lists the Q8_0 build at 7.95 GiB and the Q4_K_M build, about 4.89 bits per weight, at 4.58 GiB. The smaller file fits in the memory of an ordinary laptop, which the 16 GB version does not comfortably do.

Where you'll meet it

  • Model files labelled Q4, Q8, INT8, FP8, AWQ or GPTQ on Hugging Face
  • Local tools such as Ollama, LM Studio and llama.cpp
  • Production inference servers serving FP8 or INT8 models
  • Capacity planning and GPU purchasing decisions

What it changes for you

For developers, quantization is what makes local and on-device AI practical. For infrastructure teams, it lowers the memory each model copy needs, which changes the capacity plan for the same model. For quality owners, quantised models need their own evals, because perplexity on a benchmark does not tell you how your task behaves.

Quantization vs distillation

Quantization makes the same model use fewer bits. Distillation trains a different, smaller model to imitate a bigger one. They are often combined.

Common mistake

My view: picking the smallest quantisation that runs and assuming it is "the same model". Benchmark the quantised build on your own tasks; a 4-bit model that is fine for chat can be noticeably worse at structured extraction.

Open-weights model · Small language model · On-device AI · Inference server · LoRA

From the archive: Quantization damage hides in the flips, not the average

Benchmark the quantised build on your own tasks before you treat it as the same model.

#field-guide#inference#local-models#models#optimization

Sources

  1. arxiv.org/abs/2210.17323
  2. github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md
← older drop
Context rot: when long context makes models worse
newer drop →
Agent sandbox: an isolated place for AI agents to act

related drops

explore all 557 drops →
← back to the archiveday 175