← back to the archiveCover illustration for “Tokens per second: how fast an AI model writes”
TERM8mo ago·Published · Updated ·by Andy Padia

Tokens per second: how fast an AI model writes

In short: Tokens per second measures how fast a model generates output, per user and across a server, and why the two numbers pull against each other under load.

Tokens per second (TPS, or output speed) measures how many tokens a language model generates each second. It sets how long a full answer takes to stream and how fast agents get through multi-step tasks. NVIDIA's benchmarking guide, published on 2 April 2025, separates two numbers: per-user speed for one request and system speed across every request a server handles at once.

Tokens per second in plain words

Think of typing speed. Time to first token is how long before someone starts typing; tokens per second is how fast they type once they do.

How it works

Each output token needs a pass through the model. Speed depends on model size, hardware, precision or quantisation, batch size and serving software.

NVIDIA's guide describes the trade-off under load: higher concurrency raises system tokens per second until the GPU saturates, while each user's speed falls as latency grows. It also computes per-user speed as output length divided by end-to-end latency, which includes the wait for the first token, so check which definition a benchmark uses before you compare numbers.

Serving software such as vLLM lists continuous batching, quantisation and speculative decoding among its features. Batching serves total throughput; speculative decoding is aimed at the speed of a single stream.

Example

As an illustration: a 600-token answer at 30 tokens per second takes 20 seconds to finish streaming; at 120 tokens per second it takes 5. For an agent making 40 calls per task, that difference compounds into minutes.

Where you'll meet it

  • Benchmarks comparing model providers and hardware
  • Self-hosted capacity planning
  • Coding agents and voice agents, where speed shapes usability
  • Specialised inference providers marketing very high output speed

What it changes for you

For developers, output speed matters most where responses are long or agents loop many times. For platform teams, the trade-off is per-user speed against total throughput and cost: tune batch sizes to your service-level targets. For product teams, it is worth testing whether a faster mid-size model serves an interactive feature better than a slower top model.

Tokens per second vs throughput

Per-user tokens per second is one stream's speed. Throughput is total tokens served per second across all users. Optimising one can hurt the other.

Common mistake

My view: comparing providers on a single quiet test request. Speeds under real concurrent load can be very different. Test at your expected concurrency and at your busiest hour.

Time to first token · Speculative decoding · Inference server · Quantization · Inference

From the archive: Local model benchmarks are not capacity plans

Measure tokens per second at your expected concurrency and your busiest hour before you compare providers.

#field-guide#inference#performance#latency

Sources

  1. developer.nvidia.com/blog/llm-benchmarking-fundamental-concepts
  2. docs.vllm.ai
← older drop
Token pricing: how AI model APIs charge you
newer drop →
Vector database: storage built for similarity search

related drops

explore all 557 drops →
← back to the archiveday 175