← back to the archiveCover illustration for “Distillation: teaching a small model from a big one”
TERM7mo ago·Published · Updated ·by Andy Padia

Distillation: teaching a small model from a big one

In short: Distillation trains a smaller student model on a larger teacher model's outputs, keeping much of the quality on chosen tasks at lower cost.

Distillation is a training technique in which a smaller "student" model learns to reproduce the outputs of a larger "teacher" model, so it keeps much of the teacher's quality on target tasks while being cheaper and faster to run. The name comes from a March 2015 paper by Hinton, Vinyals and Dean, which built on earlier work compressing an ensemble of models into a single model.

Distillation in plain words

Think of a master chef teaching an apprentice: the chef cooks a thousand dishes while the apprentice watches and copies. The apprentice never becomes the master, but for the restaurant's regular menu, they get very close at a fraction of the wage.

How it works

With LLMs it usually means generating teacher answers and fine-tuning a small model on them. OpenAI's fine-tuning guide, read in November 2026, describes the steps:

  1. Tune a prompt for the larger model until its results meet your eval criteria.
  2. Capture those results.
  3. Build a dataset from the captured responses that meet the criteria.
  4. Fine-tune a smaller model on that dataset.

Then evaluate the student against the teacher on held-out cases. Where you control the teacher, the student can also learn from its probability distributions as well as its final answers.

Example

As an illustration, a company uses a frontier model to label 20,000 support emails with intent and urgency, checks a sample by hand, then fine-tunes a small open-weights model on the labels. The small model then classifies all incoming email at a much lower cost per call.

Where you'll meet it

  • Fine-tuning platforms that train a smaller model on a larger model's captured responses
  • Open-weights models trained on outputs of larger models
  • Policy debates about distilling competitors' models and licence terms

What it changes for you

For architects, distillation is the path from an expensive prototype to an affordable production system: prototype with the big model, then distil for the high-volume task. For legal teams, read the provider's terms before you train on a model's outputs. For quality, the student inherits the teacher's mistakes as well as its strengths.

Distillation vs fine-tuning

Fine-tuning trains on any curated data. Distillation is fine-tuning where the data comes from a stronger model's outputs.

Common mistake

My view: distilling the whole general capability and expecting a small model to keep up everywhere. Distil narrow, high-volume tasks, and route everything else to the bigger model.

Small language model · Fine-tuning · Model routing · Cost per task · Open-weights model

From the archive: Distilled models inherit their teacher's provenance · Distillation disputes are now trade policy, not license disputes

Distil narrow, high-volume tasks and route everything else to the bigger model.

#field-guide#models#training#inference#cost

Sources

  1. arxiv.org/abs/1503.02531
  2. developers.openai.com/api/docs/guides/supervised-fine-tuning
← older drop
AI observability: seeing what your AI system actually did
newer drop →
Foundation model: one big model, adapted to many tasks

related drops

explore all 557 drops →
← back to the archiveday 175