
Fine-tuning: adapting a trained model with your examples
In short: Fine-tuning continues training a pre-trained model on your own examples to change its behaviour, style or task skill, and it suits behaviour more than facts.
Fine-tuning is further training of an already pre-trained model on a smaller, task-specific dataset, so its weights change to favour the behaviour you want: a format, a tone, domain vocabulary or a narrow skill such as classification. Prompting changes only the input, and RAG adds knowledge at query time. OpenAI's supervised fine-tuning guide, read in November 2026, accepts as few as 10 examples.
Fine-tuning in plain words
Prompting is giving a skilled cook a recipe card each time. Fine-tuning is sending them on a short course so they cook your house style without the card.
How it works
- Prepare examples: inputs paired with ideal outputs. OpenAI's guide recommends starting with 50 well-crafted demonstrations and then evaluating the results.
- Train starting from the base model's weights.
- Evaluate on data you held back, against the base model plus a good prompt. The same guide says to invest in fine-tuning only after setting up evals.
- Deploy the tuned model or adapter, and re-evaluate when the base model changes.
With open models you run this yourself. Hugging Face's TRL library provides a supervised fine-tuning trainer and lists PEFT, the home of adapter methods such as LoRA, among its integrations.
Example
As an illustration, an insurer wants claim notes turned into a strict internal summary format with its own codes. A good prompt gets most outputs right but keeps slipping on the codes. Fine-tuning a small model on a few thousand approved examples is the next thing to test, and the shorter prompt would also cut cost per call.
Where you'll meet it
- Fine-tuning APIs from model providers
- Open-source tooling such as Hugging Face TRL and Unsloth
- Domain models in legal, medical and finance
- Distilling large-model behaviour into small models
What it changes for you
For teams, fine-tuning is a data project first: collecting, cleaning and approving examples is most of the work. For architects, a tuned model is an artefact to version, evaluate and retrain when the base model is retired. For most knowledge problems, RAG is the better tool; fine-tune for behaviour, not facts.
Fine-tuning vs RAG
RAG supplies facts at question time and updates instantly when documents change. Fine-tuning changes behaviour and needs retraining to change.
Common mistake
My view: fine-tuning to teach a model company facts. It memorises poorly, cannot cite sources and goes stale. Put facts in retrieval and use fine-tuning for format, tone and task skill.
Related terms
LoRA · Retrieval-augmented generation · Distillation · Golden dataset · Few-shot prompting
From the archive: Fine-tune to change behaviour, use RAG to change knowledge · Choose prompts, RAG, tools or fine-tuning
Put facts in retrieval, and fine-tune only for format, tone and task skill.


