Choose prompts, RAG, tools or fine-tuning
In short: Match the fix to the observed failure: prompts for ambiguity, retrieval for changing facts, tools for live state, schemas for output shape, and training last.
The paper that named retrieval-augmented generation, submitted by Lewis and colleagues on 22 May 2020, framed it as combining a model's parametric memory with an external one — a repair for missing facts. OpenAI's structured-outputs documentation, inspected on 30 September 2026, makes an equally narrow claim: a schema guarantees the shape of an output, and a schema-conforming answer can still be wrong. Each technique fixes one kind of failure and is silent about the others.
That is the whole method. Before choosing between prompts, RAG, tools or fine-tuning, name the failure you actually observed. An ambiguous instruction is repaired with a prompt and examples. Evidence that changes is repaired with retrieval. A need for live state or an action is repaired with a tool. A broken output contract is repaired with a schema. Only a persistent task behaviour — tone, format habits, a domain's way of reasoning — that survives all four is a candidate for training, and only after evaluation shows it.
The failure-to-intervention matrix
| Observed failure | What it looks like | First intervention | What it cannot fix |
|---|---|---|---|
| Ambiguous instruction | right facts, wrong emphasis or format | prompt + worked examples | missing or stale facts |
| Changing evidence | confident answer from last year's policy | retrieval over the current corpus | behaviour, output shape |
| Live state or action needed | "I can't check that" or a guessed status | an authorised tool | correctness of what it says |
| Output contract broken | valid prose, invalid JSON | schema-constrained output | factual accuracy |
| Persistent task behaviour | same style failure across good prompts | fine-tuning, after evals | knowledge, freshness |
Read a row across, not the table down: each intervention has a blind spot, and the blind spot is usually the next failure you will see.
The pair everyone argues about already has a page: fine-tuning changes behaviour, RAG changes knowledge. This piece extends it to the two interventions that debate skips — tools and output contracts — because most production failures are one of those two wearing a knowledge problem's clothes. A support bot that "doesn't know the order status" does not need more documents; it needs an order-lookup tool it is allowed to call.
These are layers, not rivals
The reason the choice feels hard is that it is framed as a choice. In a working system the four compose: a prompt sets the task, retrieval supplies the evidence, a tool fetches live state or performs the action, and a schema fixes the shape of what comes back — with a validator that checks the content, not just the structure, because constrained JSON does not constrain the tool choice or the facts inside it.
Fine-tuning is absent from that diagram on purpose. It is not a layer in the request path; it is a change to the model that the other four layers sit on, and it should enter only when the layers are in place and a held-out evaluation still shows the same behavioural failure. One caveat belongs here, labelled: the OpenAI optimisation guide inspected for this piece carries legacy and deprecation notices, so provider training availability must be rechecked at decision time rather than assumed.
The stop rule
Keep the simplest configuration that passes the held-out task criteria, and stop. Two misuses this rule prevents are common enough to name. Do not fine-tune to obtain valid JSON; a schema does that deterministically and a trained model still produces malformed output on a bad day. And do not describe retrieval as a guarantee of supported citations; retrieval puts documents in front of the model, and whether the answer actually rests on them is a property you measure — retrieval recall is a corpus-scope problem before it is a model problem. The same honesty applies to tools: a tool description is not a guardrail; authorisation is enforced by the host.
The diagnostic exercise I would run
No comparative intervention experiment was run for this piece; this is the exercise, labelled as such. Take four failed support requests from last week's logs. For each, write one sentence on what was observed — not what you suspect — and place it in a matrix row. A request answered from a superseded refund policy is a changing-evidence row. A request where the bot asserted a delivery date it could not know is a live-state row. A reply that was correct but ignored the required format is an ambiguous-instruction row, or an output-contract row if the format is machine-read. A reply that was wrong in the same characteristic way across all four, with good prompts and the right documents in context, is the only one that earns a training conversation — and it earns it only after the evaluation suite can reproduce the failure on demand.
Most teams who run this find no training candidate among the four. That is the result, not a disappointment.
Where this holds and where it breaks
It holds for feature work with a checkable task — support, extraction, drafting from policy — where failures can be named from logs. It is weaker for stylistic or voice problems, where "persistent behaviour" is a judgment and the training case is genuinely stronger. It breaks when nobody can say what was observed; then every intervention is a guess, and the first fix is logging and an eval, not any row in the matrix.
FAQ
Can a prompt replace retrieval?
Only for facts that do not change and fit in context. The day the policy changes, the prompt is the stale document.
Does structured output make answers correct?
No. It makes them parseable. OpenAI's own documentation says schema-conforming outputs can still contain mistakes; a validator for content sits after the schema.
When is fine-tuning the right first move?
Almost never first. When the behaviour fails consistently across good prompts, current evidence and the right tools, and an eval can reproduce it, training becomes a measured bet rather than a reflex.
What's in it for you
- A five-row failure-to-intervention matrix that turns "RAG or fine-tune?" into "what did we observe?".
- A composed reference flow — prompt, retrieval, tool, schema, validator — showing the techniques as layers, not rivals.
- A stop rule and two named misuses that keep training as the last, evaluated option.
Name the failure first: prompts for ambiguity, retrieval for changing facts, tools for live state, schemas for shape — and training only when an eval proves the behaviour persists.
