← back to the archiveCover illustration for “KV cache: the memory that keeps LLM generation fast”
TERM8mo ago·Published · Updated ·by Andy Padia

KV cache: the memory that keeps LLM generation fast

In short: The KV cache stores attention keys and values for tokens already processed so a model need not recompute them, at a growing cost in GPU memory.

The KV cache (key-value cache) is the memory that keeps LLM generation fast: it stores the attention keys and values computed for every token already processed, so the model reuses them for the next token instead of recomputing the whole sequence. The PagedAttention paper, submitted in September 2023, describes this memory as huge and as growing and shrinking dynamically.

KV cache in plain words

It is like keeping your working notes from every page you have read, so you do not reread the whole book before writing each new sentence. That is fast, but the notes take up desk space.

How it works

In a transformer, each new token attends to all earlier tokens through their keys and values. Without a cache the model would repeat the same calculations at every step, which is how the Hugging Face cache guide explains it. During prefill, keys and values are computed for the whole prompt and stored. During decoding, each new token adds its own entry in every attention layer. So the cache grows with every token of context and with every concurrent request.

That growth is the serving problem. The PagedAttention authors found that badly managed cache memory is wasted through fragmentation and duplication, and their system, vLLM, allocates it in pages. The Hugging Face guide lists other trade-offs: offloading the cache to CPU or quantising it saves GPU memory at some cost in speed.

Example

As an illustration, serving long-context requests: doubling every user's context length roughly doubles each request's KV-cache memory, which can halve how many users fit on the same GPU at once, even though the model weights did not change.

Where you'll meet it

  • Inference server settings for maximum context and concurrency
  • Prefix caching, which reuses a stored cache for a repeated prompt prefix
  • Research on cache compression and offloading
  • GPU memory planning for self-hosted models

What it changes for you

For infrastructure teams, KV-cache memory sets serving capacity alongside model size, so long-context features need explicit capacity planning. For developers, shorter prompts and shared prefixes reduce memory and cost. For product teams, my view is that this is one reason long-context usage tends to be priced and rate-limited more carefully.

KV cache vs prompt caching

The KV cache is working memory inside one request. Prompt caching keeps computed state for a repeated prefix across requests, so the shared part is not processed again. The Hugging Face guide shows the idea as prefilling a cache with a common prompt.

Common mistake

My view: sizing GPUs from model weights alone. A model that fits comfortably can still run out of memory at your real concurrency and context lengths. Plan for weights plus KV cache at peak load.

Prompt caching · Inference server · Context window · Time to first token · Inference

From the archive: A 128K context window needs a concurrency budget

Size GPUs for model weights plus KV cache at peak concurrency and context length.

#field-guide#inference#infrastructure#performance

Sources

  1. arxiv.org/abs/2309.06180
  2. huggingface.co/docs/transformers/kv_cache
← older drop
Semantic search: search that understands what you meant
newer drop →
Context rot: when long context makes models worse

related drops

explore all 557 drops →
← back to the archiveday 175