← back to the archiveCover illustration for “Inference: running a trained AI model to get answers”
TERMday 172·3d ago·Published ·by Andy Padia

Inference: running a trained AI model to get answers

In short: Inference is running a trained model on new input to produce output, and it is where most ongoing AI cost and latency sit.

Inference is the process of running an already trained AI model on new inputs to produce outputs: every chat reply, completion, embedding or classification is an inference call. Training happens once or occasionally; inference happens every time a user or agent uses the model. For most organisations, inference is where ongoing AI cost, latency and capacity planning concentrate.

Inference in plain words

Training is the years a doctor spends at medical school. Inference is every individual patient consultation afterwards.

How it works

For a language model, inference has two phases. Prefill processes the whole prompt at once. Decode generates the answer one token at a time. Serving systems such as vLLM batch many users' requests together and cache intermediate state to keep hardware busy. The PagedAttention paper reported in September 2023 that managing that cache better raised throughput 2 to 4 times at the same latency.

Example

As an illustration, a request with a 3,000-token prompt and a 300-token answer spends a short burst on prefill, which mostly sets time to first token, then 300 decode steps, which set how long the full answer takes to stream.

Where you'll meet it

  • API pricing per input and output token
  • Inference servers such as vLLM, SGLang, TensorRT-LLM and llama.cpp
  • GPU and TPU capacity planning
  • Latency metrics such as time to first token and tokens per second

What it changes for you

For product teams, inference cost scales with usage, so unit economics must be modelled per user or per task. For engineers, long prompts, verbose outputs and agent loops multiply inference spend. For the industry, falling inference prices and rising agent token use pull in opposite directions, so total bills often grow even as prices drop.

Inference vs training

Training creates or updates a model's weights and is done rarely at high cost. Inference uses fixed weights to answer requests, continuously.

Common mistake

My view: budgeting AI projects on the price per token and forgetting the number of tokens per task. Agents can use many times the tokens of a single chat reply. Measure tokens per completed task early.

Time to first token · Tokens per second · KV cache · Inference server · Cost per task

From the archive: A 128K context window needs a concurrency budget

Budget inference by tokens per completed task; the price per token alone tells you little.

#field-guide#inference#infrastructure#cost

Sources

  1. docs.vllm.ai
  2. arxiv.org/abs/2309.06180
← older drop
Embeddings: turning text into numbers that capture meaning
newer drop →
Cost per task: the real unit of AI spend

related drops

explore all 557 drops →
← back to the archiveday 175