
KV cache: the memory that keeps LLM generation fast
In short: The KV cache stores attention keys and values for tokens already processed so a model need not recompute them, at a growing cost in GPU memory.
The KV cache (key-value cache) is the memory that keeps LLM generation fast: it stores the attention keys and values computed for every token already processed, so the model reuses them for the next token instead of recomputing the whole sequence. The PagedAttention paper, submitted in September 2023, describes this memory as huge and as growing and shrinking dynamically.
KV cache in plain words
It is like keeping your working notes from every page you have read, so you do not reread the whole book before writing each new sentence. That is fast, but the notes take up desk space.
How it works
In a transformer, each new token attends to all earlier tokens through their keys and values. Without a cache the model would repeat the same calculations at every step, which is how the Hugging Face cache guide explains it. During prefill, keys and values are computed for the whole prompt and stored. During decoding, each new token adds its own entry in every attention layer. So the cache grows with every token of context and with every concurrent request.
That growth is the serving problem. The PagedAttention authors found that badly managed cache memory is wasted through fragmentation and duplication, and their system, vLLM, allocates it in pages. The Hugging Face guide lists other trade-offs: offloading the cache to CPU or quantising it saves GPU memory at some cost in speed.
Example
As an illustration, serving long-context requests: doubling every user's context length roughly doubles each request's KV-cache memory, which can halve how many users fit on the same GPU at once, even though the model weights did not change.
Where you'll meet it
- Inference server settings for maximum context and concurrency
- Prefix caching, which reuses a stored cache for a repeated prompt prefix
- Research on cache compression and offloading
- GPU memory planning for self-hosted models
What it changes for you
For infrastructure teams, KV-cache memory sets serving capacity alongside model size, so long-context features need explicit capacity planning. For developers, shorter prompts and shared prefixes reduce memory and cost. For product teams, my view is that this is one reason long-context usage tends to be priced and rate-limited more carefully.
KV cache vs prompt caching
The KV cache is working memory inside one request. Prompt caching keeps computed state for a repeated prefix across requests, so the shared part is not processed again. The Hugging Face guide shows the idea as prefilling a cache with a common prompt.
Common mistake
My view: sizing GPUs from model weights alone. A model that fits comfortably can still run out of memory at your real concurrency and context lengths. Plan for weights plus KV cache at peak load.
Related terms
Prompt caching · Inference server · Context window · Time to first token · Inference
From the archive: A 128K context window needs a concurrency budget
Size GPUs for model weights plus KV cache at peak concurrency and context length.


