
Distillation: teaching a small model from a big one
In short: Distillation trains a smaller student model on a larger teacher model's outputs, keeping much of the quality on chosen tasks at lower cost.
Distillation is a training technique in which a smaller "student" model learns to reproduce the outputs of a larger "teacher" model, so it keeps much of the teacher's quality on target tasks while being cheaper and faster to run. The name comes from a March 2015 paper by Hinton, Vinyals and Dean, which built on earlier work compressing an ensemble of models into a single model.
Distillation in plain words
Think of a master chef teaching an apprentice: the chef cooks a thousand dishes while the apprentice watches and copies. The apprentice never becomes the master, but for the restaurant's regular menu, they get very close at a fraction of the wage.
How it works
With LLMs it usually means generating teacher answers and fine-tuning a small model on them. OpenAI's fine-tuning guide, read in November 2026, describes the steps:
- Tune a prompt for the larger model until its results meet your eval criteria.
- Capture those results.
- Build a dataset from the captured responses that meet the criteria.
- Fine-tune a smaller model on that dataset.
Then evaluate the student against the teacher on held-out cases. Where you control the teacher, the student can also learn from its probability distributions as well as its final answers.
Example
As an illustration, a company uses a frontier model to label 20,000 support emails with intent and urgency, checks a sample by hand, then fine-tunes a small open-weights model on the labels. The small model then classifies all incoming email at a much lower cost per call.
Where you'll meet it
- Fine-tuning platforms that train a smaller model on a larger model's captured responses
- Open-weights models trained on outputs of larger models
- Policy debates about distilling competitors' models and licence terms
What it changes for you
For architects, distillation is the path from an expensive prototype to an affordable production system: prototype with the big model, then distil for the high-volume task. For legal teams, read the provider's terms before you train on a model's outputs. For quality, the student inherits the teacher's mistakes as well as its strengths.
Distillation vs fine-tuning
Fine-tuning trains on any curated data. Distillation is fine-tuning where the data comes from a stronger model's outputs.
Common mistake
My view: distilling the whole general capability and expecting a small model to keep up everywhere. Distil narrow, high-volume tasks, and route everything else to the bigger model.
Related terms
Small language model · Fine-tuning · Model routing · Cost per task · Open-weights model
From the archive: Distilled models inherit their teacher's provenance · Distillation disputes are now trade policy, not license disputes
Distil narrow, high-volume tasks and route everything else to the bigger model.


