How much does an LLM feature cost in production?
In short: Budget an LLM feature per accepted task, with retries, tools and human recovery counted; cheaper tokens cost more when the workflow needs more attempts.
OpenAI's Batch API, as documented on 19 October 2026, charges 50% less than synchronous requests and promises completion within 24 hours. That is the kind of number that ends up on a slide: "we can halve our LLM bill". It is also the kind of number that hides what the bill is made of.
The claim I will defend: how much an LLM feature costs in production should be budgeted per accepted task, not per token. Add retries, tool calls, evaluation and the human minutes spent recovering from the tasks that fail, then divide by the tasks the business actually accepted. Do that and a 50% token discount usually turns out to be a rounding error next to the acceptance rate — which is the number a cheaper model can quietly make worse.
The unit is the accepted task, not the token
A token price tells you the cost of one attempt. A workflow does not pay for attempts; it pays for outcomes, and every attempt that does not become an outcome still costs the tokens, the tools it called, and — the expensive part — the human who notices and fixes it. So the denominator is accepted tasks, and the numerator has at least six lines: model tokens (input, output, cached), retrieval and tool calls, hosting and observability, the evaluation suite you maintain, human review of exceptions, and the incident cost when something goes wrong in front of a customer.
What the feature costs in production: the worksheet
Here is the shape, priced on illustrative unit rates, not vendor prices: $1 per million input tokens, $4 per million output tokens, 3,000 input and 400 output tokens per attempt, $0.002 of retrieval and tools per attempt, $30 per hour of human time, 6 minutes per escalation, and a flat $15 per 1,000 tasks for hosting, observability and eval runs. Per 1,000 tasks:
| Scenario | Attempts / task | Acceptance | Model + tools | Human recovery | Total | Per accepted task |
|---|---|---|---|---|---|---|
| Low (tight scope) | 1.1 | 97% | $7 | 30 × $3 = $90 | $112 | $0.12 |
| Base | 1.3 | 92% | $9 | 80 × $3 = $240 | $264 | $0.29 |
| High (loose scope) | 1.6 | 85% | $11 | 150 × $3 = $450 | $476 | $0.56 |
Arithmetic on the stated assumptions; replace every number with your own. Escalations are the tasks not accepted; "attempts" includes retries.
Read the table from right to left. The model-and-tools line is between $7 and $11 in all three cases. Human recovery is between $90 and $450. A 50% discount on the first line saves under $5 per thousand tasks; moving acceptance from 92% to 97% saves $150. That is the whole argument. The cheapest model is the one whose outputs get accepted, and the only way to know that is the evaluation suite measured on your cases — which is also why task cost is a harness property, not a model price.
What the optimisations actually buy
OpenAI's latency guide groups the techniques into seven principles — process tokens faster, generate fewer tokens, use fewer input tokens, make fewer requests, parallelise, make users wait less, and don't default to an LLM — and notes that cutting half your output tokens cuts roughly half your latency. Map them onto the worksheet and they sort into three kinds.
Some reduce the small line: prefix caching for repeated context, shorter outputs, routing bounded sub-tasks to a smaller model. Worth doing; expect single-digit percentages of the total. Some reduce the big line: fewer requests and parallel independent calls cut the retries and timeouts that turn into escalations. And one — "don't default to an LLM" — removes whole rows: a deterministic lookup has an acceptance rate of 100% and a human-recovery cost of nothing. The Batch API belongs in the first group with a condition attached: the 24-hour window makes it a tool for overnight evaluation runs and backfills, not for anything a user is waiting on.
Keep the quality checks after every change. A routing rule that saves 40% of tokens and loses two points of acceptance has made the feature more expensive, and the dashboard that only tracks spend will tell you the opposite.
The calculation I would run first
No production bill or benchmark was inspected for this piece, so this is a plan, labelled as such. Take one support answer and price three paths through it: the request that succeeds first time; the one that retries once; and the one that escalates to a person. Write the token counts, tool calls and minutes on each path from real traces, not estimates. Weight the three paths by last month's acceptance and retry rates, and you have a per-accepted-task cost you can defend — and, more usefully, a sensitivity: which single number, moved by five points, changes the total most. In every version of this I have sketched, it is the escalation rate, never the token price.
Where this holds and where it breaks
It holds for high-volume, bounded tasks — triage, extraction, drafting from templates — where acceptance is measurable and human recovery is a real cost line. It is weaker for creative or advisory features, where "accepted" is a judgment and recovery is a rewrite nobody logs. It breaks if you cannot measure acceptance at all; then the worksheet has no denominator, and the first investment is instrumentation, not a cheaper model.
FAQ
Is a 50% batch discount worth taking?
For evaluation runs, backfills and anything with a day of slack, yes. For an interactive feature it is irrelevant: the window is 24 hours, and the saving is on the smallest line of the worksheet.
Does a smaller model always lower the cost?
Only if acceptance holds. Two points of lost acceptance on the base scenario costs more than the entire model line; measure before switching.
What should the first dashboard show?
Cost per accepted task, acceptance rate, retries per task and escalation minutes, in that order. Token spend alone optimises the wrong line.
What's in it for you
- A six-line worksheet and three labelled scenarios you can re-price with your own volumes in an hour.
- A sensitivity rule — escalation rate before token price — that tells you where optimisation effort pays.
- A sorting of the seven latency principles into what shrinks the small line, the big line, or whole rows.
Price the feature per accepted task; the escalation rate, not the token price, is the number that moves the bill.
