How to choose an LLM for your actual workload
In short: Choose an LLM by eliminating every candidate that fails your task contract on your own cases first, and only then ranking the survivors on cost and latency.
OpenAI's own model-selection guide, as inspected on 30 September 2026, does not start with a leaderboard. It starts with four questions about the job: how often the workload runs, how urgently a response is needed, what the output is used for, and what level of quality that use demands. Its evaluation guide adds the other half — compare models on production-shaped, edge and adversarial cases with task-specific criteria, because a universal ranking cannot tell you which model fails your task.
So here is the method, stated plainly. Choose an LLM for your actual workload by elimination first and ranking second: run every candidate — frontier, smaller, open-weight, self-hosted — against the same held-out cases for each task you care about, strike every candidate that fails the task contract, and only then compare the survivors on cost, latency and operating effort. A cheaper model that fails one case in twenty is not cheaper; it is a different product.
Why one winner is the wrong output
"Which model is best?" collapses four jobs into one number. A model that writes clean code may hallucinate a field in extraction; a model that reasons well may be too slow for the support desk. The useful output of a comparison is therefore a scorecard per task, not a podium — and the useful input is a set of cases that look like production, including the awkward ones. OpenAI's guidance is explicit that evaluation cases should include edge and adversarial inputs, not just the happy path; the same applies to a model bake-off, where the happy path is exactly where every candidate looks the same.
Two neighbouring arguments on this site set the boundary. Premium models need failure-cost routing is about which model handles which request once you have chosen a set; local model benchmarks are not capacity plans is about what a self-hosted number leaves out. This piece is the protocol that produces the set in the first place.
The task-to-grader matrix
Each task gets its own grader, and the grader decides what "fails the contract" means. Cells are left for your measurements — this table is the protocol, not a result.
| Task | Grader | Contract (pass / fail) | Candidate A | Candidate B | Candidate C |
|---|---|---|---|---|---|
| Coding | hidden tests executed | all tests pass, no banned calls | — | — | — |
| Extraction | field accuracy + schema validity | schema valid, required fields correct | — | — | — |
| Reasoning | verifiable final answer | answer matches reference | — | — | — |
| Support | grounded in sources + correct escalation | no unsupported claim, escalates when policy says | — | — | — |
Score every cell as a pass rate across repeated runs and record the sample size next to it. A candidate that fails any contract row is eliminated for that task before cost is discussed.
Record the configuration, not the brand
A comparison is only repeatable if the row is a configuration: the exact model ID and its date, the prompt and tool definitions, the context limit you actually hit, quantization and hardware for anything self-hosted, the concurrency you tested at, the p95 latency you measured, and the full cost of each accepted answer including retries. "GPT" or "Claude" or "Llama" is not a row; a brand name moves under you between releases, and the same open-weight model at two quantizations is two candidates.
Open-weight candidates need two extra columns before cost: licence terms, and the operating effort someone will carry — the same column an open-source shortlist usually leaves out. Privacy, reliability and capacity fit are gates, not weights: a model you may not send the data to is eliminated, however it scores.
The comparison I would run
No cross-model benchmark and no current price comparison was run for this piece, so the table above stays unmeasured and no model is named as best. The comparison I would run: build four held-out sets — a dozen coding tasks with hidden tests, fifty extraction documents with gold fields, thirty reasoning problems with checkable answers, forty support tickets with source documents and an escalation policy. Hold them out from any prompt tuning, as OpenAI's guide does with its reference transcripts. Run each shortlisted configuration five times per case, log every retry and the tokens it consumed, and fill the matrix. Only then open the pricing pages — and price per accepted task, because retries are where the "cheap" model's bill lives.
The last step is the one teams skip: declare a tuning budget per candidate and keep it equal. A frontier model with a one-line prompt against an open-weight model with a week of prompt engineering is not a comparison; it is a preference with data attached.
Where this holds and where it breaks
It holds for workloads with a checkable contract — code, extraction, structured reasoning, grounded support — which covers most enterprise features. It is weaker for open-ended writing or advice, where the grader is a calibrated human and the contract is a trend rather than a pass rate. It breaks when the organisation cannot write the held-out cases; then the problem is not model choice but the absence of an evaluation suite, and any model you pick will be chosen by the demo.
FAQ
Should I just pick the frontier model and move on?
Run it through the same protocol. It may well survive every row — but you will know its p95 latency and cost per accepted task, which is what you need when a smaller candidate also survives.
How often should the comparison be repeated?
Whenever a candidate's model ID changes, and on a fixed cadence otherwise. Keep the held-out sets and the harness versioned so a re-run is an afternoon, not a project.
Does a public leaderboard count for anything?
As a shortlist filter, yes. As evidence that a model will meet your contract, no — the leaderboard did not run your cases.
What's in it for you
- A task-to-grader matrix with the contract per row, ready to fill with your own pass rates.
- A configuration record — model ID, date, prompt, tools, hardware, concurrency, p95, retry cost — that makes the comparison repeatable.
- A protocol that eliminates before it ranks, so a cheap failure never gets averaged into a good decision.
Eliminate every model that fails your task contract on your own cases, then rank what is left on cost; a model that fails the contract has no price.
