A 128K context window needs a concurrency budget

Two requests, each holding 128,000 tokens, can account for about 78.1 GiB of KV cache on Llama 3.1 70B with 16-bit cache storage. That is the cache alone. The weights haven't entered the calculation.

Meta's Llama 3.1 release gives the 70B model a 128K context window. Useful capability. But a product promise to serve several long requests at once needs another number: how much live state those requests may hold together.

My rule: long context should be an explicitly budgeted service tier, not a global checkbox nobody owns after launch. The maximum describes what the model can accept. It doesn't tell you how many such requests your deployment can serve within its latency target.

Start with the bytes, not the headline

The Llama architecture paper's Table 3 lists 80 layers, eight key/value heads, 64 attention heads and an 8,192-wide model dimension for 70B. The head dimension is therefore 128. With two bytes per stored value, the logical cache calculation for one sequence is:

2 × 80 × 8 × 128 × 2 × token count

The first factor covers keys and values. At 128,000 tokens, that is 41,943,040,000 bytes, or 39.0625 GiB. This is derived arithmetic, not a measured GPU benchmark. If your token limit is 131,072 instead, the same calculation gives exactly 40 GiB. Write the actual integer in the capacity sheet; “128K” is too loose for arithmetic.

Keep the assumptions beside the result. This is an unquantized 16-bit KV representation, without shared-prefix savings, and not a total serving-memory estimate. Sharding changes placement; implementations can add overhead or replication. Neither the number nor its precision makes it a hardware purchase recommendation.

For a planning illustration, compare a 16,000-token sequence with the 128,000-token case: about 4.9 GiB against 39.1 GiB under the same assumptions. One long sequence occupies the logical cache budget of eight shorter ones. That ratio is a useful product discussion even before anyone rents a GPU.

A limit is not an allocation

Here is the correction that keeps this argument honest: setting a large maximum does not mean every short request immediately consumes the maximum-sized cache.

PagedAttention allocates cache blocks on demand instead of requiring one contiguous maximum-sized reservation per request. It also supports sharing blocks where sequences have reusable prefixes. The authors reported 2–4× throughput improvements against their evaluated baselines at comparable latency; that result belongs to those experiments, not every deployment using vLLM.

Better packing is real progress. It doesn't make two unrelated, fully populated contexts occupy the space of one. This is why “we use paged attention” answers a memory-management question, while “what happens when the long requests arrive together?” remains a service-design question.

I wouldn't require a system to reserve the worst case for every idle chat. I would require it to explain how it handles growth. Does it admit fewer long requests, queue them, preempt work, or offer a slower lane? Choose deliberately, then test the choice. Quietly hoping the contexts stay small is not an overload policy.

Give long work its own admission decision

As an illustration, consider an internal assistant with routine question-answering and occasional whole-contract analysis. These are hypothetical workloads, not a report of a client deployment. I would give them separate context budgets and measure whether the contract lane affects the ordinary lane's response target.

The proposed flow is small enough to put beside the serving configuration:

rendering diagram…

The feedback arrow matters. Admission is not just counting prompt tokens at the door; output growth and concurrent completions change the available budget. A declared output allowance gives the scheduler something concrete to plan around. The diagram is my proposed operating policy, not a claim that every serving engine implements these exact steps.

I would also make any cheaper alternative visible to the caller. If the full document cannot enter the long lane now, offer a queue or a clearly described retrieval-based route. Don't silently truncate it and present the resulting answer as though the original request was fulfilled. Capacity management must not change the promised task without saying so.

Test the collision, not just the happy path

The earlier local-model capacity-planning argument asks for realistic prompts and concurrency instead of borrowing a vendor speedup. Here the test has a specific target: the point where long-context requests compete with one another and with ordinary work.

For the first trial, replay a representative short-request mix, then introduce long requests together rather than evenly spaced. Record peak cache occupancy, queue time, time to first token and output-token latency separately. Repeat with the expected output allowance, not just a prompt followed by a one-word answer. These are proposed tests; I haven't run this workload or measured a safe concurrency here.

Also run the case where expected prefix reuse is absent. Shared material can buy room, but a budget built entirely around cache hits needs a stated fallback when the requests are distinct. Keep that fallback beside the context tier in the product configuration.

The outcome needn't be a smaller context window. It may be fewer simultaneous long jobs, a separate pool, or an honest queue. What matters is that a model capability becomes a service promise you can actually defend.

Offer the long context—but name the concurrency, memory budget and waiting policy that come with it.