← back to the archiveCover illustration for “Batch inference: cheaper AI for work that can wait”
TERM6mo ago·Published · Updated ·by Andy Padia

Batch inference: cheaper AI for work that can wait

In short: Batch inference runs many AI requests asynchronously as one job, usually at half price, for work such as classification and evals that can wait.

Batch inference is cheaper AI for work that can wait: you submit a large set of requests as one asynchronous job and collect the results later. The OpenAI and Anthropic batch documentation, read in November 2026, both quote 50 percent off standard prices and a 24-hour window. It suits classification, enrichment, evaluation and summarisation that nobody is waiting on.

Batch inference in plain words

It is posting a parcel by standard delivery instead of same-day courier. It arrives later, but it costs much less, and for most parcels that is fine.

How it works

  1. Prepare all requests, each with a unique ID. OpenAI takes a .jsonl file with one request per line; Anthropic takes a requests array.
  2. Submit the job to the provider's batch endpoint.
  3. Poll for completion. Anthropic says most batches finish in under an hour, and a batch expires if it is not complete within 24 hours.
  4. Download results and match them back by ID, because OpenAI warns that output order may not match input order. Retry any failures.

Size limits when read: 50,000 requests or 200 MB per OpenAI batch, and 100,000 requests or 256 MB per Anthropic batch.

Example

As an illustration, two lines of an OpenAI-format input file for a job that re-classifies a year of support tickets overnight at half the standard price:

{"custom_id": "ticket-1001", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "...", "messages": [{"role": "user", "content": "Classify: 'Charged twice this month'"}]}}
{"custom_id": "ticket-1002", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "...", "messages": [{"role": "user", "content": "Classify: 'Export fails on Safari'"}]}}

Where you'll meet it

  • Batch APIs from model providers and the cloud platforms that host their models
  • Data enrichment and back-filling pipelines
  • Running large evaluation suites
  • Embedding generation for big document sets, which OpenAI's batch endpoint list includes

What it changes for you

For developers, sorting work into "needs an answer now" and "can wait" is one of the easiest cost savings available. For data teams, batch makes it more affordable to apply AI across whole historical datasets. For finance, my view is that a healthy AI bill shows a mix of real-time and batch spend.

Batch inference vs real-time inference

Real-time inference answers each request immediately at standard prices. Batch inference trades speed for lower cost and, on OpenAI, for separate and higher rate limits.

Common mistake

My view: running nightly jobs through the real-time API out of habit. If a person is not waiting, check whether batch is available and cheaper before writing the loop.

Inference · Cost per task · Token pricing · Evals · Embeddings

From the archive: How much does an LLM feature cost in production?

If no person is waiting for the answer, check the batch API before you write the loop.

#field-guide#inference#cost#automation

Sources

  1. developers.openai.com/api/docs/guides/batch
  2. platform.claude.com/docs/en/build-with-claude/batch-processing
← older drop
OpenCode: an open-source AI coding agent for the terminal
newer drop →
llms.txt: a site map written for language models

related drops

explore all 557 drops →
← back to the archiveday 175