How to evaluate an LLM feature before production

Anthropic published its agent-evaluation guide on 9 January 2026; OpenAI's own Evals platform goes read-only on 31 October 2026. Tooling keeps changing; the thing worth owning is the method for deciding whether a feature may ship.

Here is the method, stated before the nuance. The first evaluation suite you write should be a short list of the failures that would cost you a release — an invented citation, a duplicate refund, a tool call that never happened — each paired with a grader and an explicit hold rule. Only after that list exists do you collect the easy, representative examples. Build it the other way round and a rising average score can hide the one regression you cannot afford.

Decide what the eval is for before you collect cases

An eval is a release decision wearing a spreadsheet. Anthropic's guide makes a distinction that most teams skip: a transcript is the full record of a trial — outputs, tool calls, intermediate results — while the outcome is the final state of the environment. Its example is a flight-booking agent that says "your flight has been booked" while nothing was booked. If your graders only read the transcript, that failure scores as a pass.

So the first question is not "what should the model say" but "what state must be true afterwards, and what must never have happened on the way". Those two sentences become your outcome checks and your transcript checks; tone and formatting can wait until the gate works.

Build the failure-cost matrix first

Write the failures down with a cost attached. Here is the shape I use when planning one; the rows are illustrative, the columns are the point.

Failure class	Example	Grader type	Cost if shipped	Rule
Fabricated reference	cites a ticket ID that does not exist	code-based (lookup)	trust loss, hard to detect later	hold on 1
Duplicate side effect	issues the same refund twice	code-based (state diff)	money, support load	hold on 1
Silent partial completion	reports done, two steps skipped	outcome check	rework, missed SLA	hold above 2 %
Wrong refusal	declines a routine, in-policy request	model-based with rubric	lost throughput	hold above 5 %
Quality drift	correct but verbose, meandering	calibrated human sample	slower users	track, no hold

Illustrative matrix for a support feature; thresholds are placeholders for your own cost of a miss.

The expensive rows get deterministic graders, because a model grading a model is not how you catch a duplicate refund. The cheap rows get model-based graders with a written rubric and, per OpenAI's guidance, periodic human calibration so the judge does not drift. Every row carries its own rule, which is what turns a score into a decision.

The five steps in the order that earns trust

Anthropic describes three grader families (code-based, model-based, human) and five construction steps; I keep its order and lean harder on the first.

Start with the checks you already do by hand before each release; they are your highest-value cases and they are free. Write each task so two domain experts would agree on the answer, and prove it is solvable with a reference solution that passes every grader. Build the set in both directions — should-do and must-not — because a one-sided suite trains a one-sided system. Pin the environment, version the dataset, and hold cases out; OpenAI's worked example holds out 1,000 reference transcripts behind a ROUGE-L floor of 0.40.

Then run repeated trials — Anthropic's pass@k exists because one green run is an anecdote — and report sample count and spread beside the number. Once a capability eval passes reliably it graduates into a regression suite that runs on every change, which is what makes eval history a version-control concern.

How I would run it on a support feature

I have not run this suite on a live system, so treat this as a plan, not a tested tutorial. The first experiment: take a small ticket-triage feature, seed the fixture with one ticket that must cite a real knowledge-base article and one whose correct action is a single refund, then make the eval fail deliberately — once on an invented article ID, once on a second refund call. If the suite goes green on either, the graders are wrong and nothing else about the eval can be trusted. Only after both red runs are reproducible would I compare prompts or models.

A suite that cannot fail on the failures you care about is a dashboard, not a gate. And as I argued in when you actually need an agent, most of these features are workflows with one judgment point, so most of the eval is ordinary state assertion with a small model-graded part.

Where this holds and where it breaks

It holds wherever a feature touches money, records or external actions: the costly rows are obvious and the graders deterministic. It is weaker for open-ended generation — drafting, summarising, advising — where the only honest grader is a calibrated human sample and the hold rule has to be a trend across releases. It breaks entirely if the harness differs from production: a different tool version, retrieval index or system prompt makes every number useless for the one decision you built it for.

FAQ

How many cases does a first eval suite need?

Fewer than you think: a dozen cases that encode the costly failures, each with a passing reference solution, beats hundreds of easy examples. Grow it from incidents afterwards.

Should the grader be a model or code?

Code wherever the check is a fact about state — a record exists, a call was made once. Model-based graders with a written rubric for judgement calls, calibrated against human review on a sample. Never a model alone for the rows that cost money.

When does an eval become a regression test?

When a capability eval passes at the rate your rule demands across repeated trials. At that point it stops asking "can we do this" and starts asking "can we still", and it should run on every change.

What's in it for you

• A failure-cost matrix you can fill in for your own feature in an afternoon, with the grader type and hold rule already decided per row.
• A build order — costly failures first, easy cases later — that stops a rising average from hiding the regression that matters.
• A first experiment that proves your graders can fail before you trust anything they pass.

Build the eval that can fail on the failures you cannot afford; the average score can earn your trust later.