
Batch inference: cheaper AI for work that can wait
In short: Batch inference runs many AI requests asynchronously as one job, usually at half price, for work such as classification and evals that can wait.
Batch inference is cheaper AI for work that can wait: you submit a large set of requests as one asynchronous job and collect the results later. The OpenAI and Anthropic batch documentation, read in November 2026, both quote 50 percent off standard prices and a 24-hour window. It suits classification, enrichment, evaluation and summarisation that nobody is waiting on.
Batch inference in plain words
It is posting a parcel by standard delivery instead of same-day courier. It arrives later, but it costs much less, and for most parcels that is fine.
How it works
- Prepare all requests, each with a unique ID. OpenAI takes a
.jsonlfile with one request per line; Anthropic takes arequestsarray. - Submit the job to the provider's batch endpoint.
- Poll for completion. Anthropic says most batches finish in under an hour, and a batch expires if it is not complete within 24 hours.
- Download results and match them back by ID, because OpenAI warns that output order may not match input order. Retry any failures.
Size limits when read: 50,000 requests or 200 MB per OpenAI batch, and 100,000 requests or 256 MB per Anthropic batch.
Example
As an illustration, two lines of an OpenAI-format input file for a job that re-classifies a year of support tickets overnight at half the standard price:
{"custom_id": "ticket-1001", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "...", "messages": [{"role": "user", "content": "Classify: 'Charged twice this month'"}]}}
{"custom_id": "ticket-1002", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "...", "messages": [{"role": "user", "content": "Classify: 'Export fails on Safari'"}]}}
Where you'll meet it
- Batch APIs from model providers and the cloud platforms that host their models
- Data enrichment and back-filling pipelines
- Running large evaluation suites
- Embedding generation for big document sets, which OpenAI's batch endpoint list includes
What it changes for you
For developers, sorting work into "needs an answer now" and "can wait" is one of the easiest cost savings available. For data teams, batch makes it more affordable to apply AI across whole historical datasets. For finance, my view is that a healthy AI bill shows a mix of real-time and batch spend.
Batch inference vs real-time inference
Real-time inference answers each request immediately at standard prices. Batch inference trades speed for lower cost and, on OpenAI, for separate and higher rate limits.
Common mistake
My view: running nightly jobs through the real-time API out of habit. If a person is not waiting, check whether batch is available and cheaper before writing the loop.
Related terms
Inference · Cost per task · Token pricing · Evals · Embeddings
From the archive: How much does an LLM feature cost in production?
If no person is waiting for the answer, check the batch API before you write the loop.


