
Inference: running a trained AI model to get answers
In short: Inference is running a trained model on new input to produce output, and it is where most ongoing AI cost and latency sit.
Inference is the process of running an already trained AI model on new inputs to produce outputs: every chat reply, completion, embedding or classification is an inference call. Training happens once or occasionally; inference happens every time a user or agent uses the model. For most organisations, inference is where ongoing AI cost, latency and capacity planning concentrate.
Inference in plain words
Training is the years a doctor spends at medical school. Inference is every individual patient consultation afterwards.
How it works
For a language model, inference has two phases. Prefill processes the whole prompt at once. Decode generates the answer one token at a time. Serving systems such as vLLM batch many users' requests together and cache intermediate state to keep hardware busy. The PagedAttention paper reported in September 2023 that managing that cache better raised throughput 2 to 4 times at the same latency.
Example
As an illustration, a request with a 3,000-token prompt and a 300-token answer spends a short burst on prefill, which mostly sets time to first token, then 300 decode steps, which set how long the full answer takes to stream.
Where you'll meet it
- API pricing per input and output token
- Inference servers such as vLLM, SGLang, TensorRT-LLM and llama.cpp
- GPU and TPU capacity planning
- Latency metrics such as time to first token and tokens per second
What it changes for you
For product teams, inference cost scales with usage, so unit economics must be modelled per user or per task. For engineers, long prompts, verbose outputs and agent loops multiply inference spend. For the industry, falling inference prices and rising agent token use pull in opposite directions, so total bills often grow even as prices drop.
Inference vs training
Training creates or updates a model's weights and is done rarely at high cost. Inference uses fixed weights to answer requests, continuously.
Common mistake
My view: budgeting AI projects on the price per token and forgetting the number of tokens per task. Agents can use many times the tokens of a single chat reply. Measure tokens per completed task early.
Related terms
Time to first token · Tokens per second · KV cache · Inference server · Cost per task
From the archive: A 128K context window needs a concurrency budget
Budget inference by tokens per completed task; the price per token alone tells you little.


