A cache selector needs a random baseline

Salesforce AI Research removed the intelligence from KV-cache eviction and the result barely moved. Its September 3 paper tested prompt-pinned random eviction across four reasoning models and six math, science and coding tasks. At roughly 4x compression, Random Attention matched the strongest prior selector across the main grid while avoiding the selector's scoring pass.

My rule is simple: a smart cache selector has not earned production complexity until it beats prompt-pinned random eviction at the same memory budget, workload and serving target. Otherwise the system may be paying to rediscover one structural fact: the prompt is the state it cannot reconstruct.

Random Attention separates protection from selection

Most cache-eviction methods score cached tokens and keep the apparent winners. The Random Attention paper asks whether that score is doing the work.

Its policy pins the whole prefill—the system prompt, chat template and question—then samples generated-trace positions independently inside each KV head. In the main comparison, Random Attention was significantly ahead in 31 of 60 baseline cells and significantly behind in one. The authors also report 32–43% more tokens per second than TriAttention in their vLLM experiment at 32k-token generations because random eviction skips the scoring pass.

Those are author-run results from arXiv v1, not numbers I reproduced. They cover Qwen3 at 4B, 14B and 32B plus Phi-4-reasoning, under the paper's tasks and cache budgets. They do not establish the best policy for a proprietary workload, short outputs or another serving stack.

Prompt protection explains much of the apparent intelligence

The useful experiment is the ablation. When the researchers gave competing methods the same prompt-protection rule, much of the gap closed. SnapKV gained as much as 22.5 percentage points on Phi-4-reasoning GPQA-D; methods that already retained more prompt gained less.

That changes how I read selector benchmarks. If one policy pins the irreplaceable input and another lets it compete for cache slots, the comparison mixes two questions: what must survive, and how should everything else be ranked. A stronger score may simply be a more expensive way to lose less of the question.

The paper's mechanism claim is narrower and interesting. Generated reasoning tends to restate working state, and every KV head carries its own copy. Random per-head retention can therefore preserve enough copies of repeated state. Rare facts stated once remain the hard case, and long code prompts can consume a large share of the cache budget before selection begins.

Make random eviction the release gate

For this article, I compared the paper and public repository with three archive arguments: context windows need concurrency budgets, cache eviction belongs in the whole-task cost ledger, and persistent context should not become an unquestioned container for agent state. The missing check was a null baseline for the selector itself.

I would test a proposed policy in this order:

• Pin the same prompt region for every policy, then match cache budget, model, prompts, output lengths and concurrency.
• Add prompt-pinned random eviction as the null, not only full attention as the ceiling.
• Measure task quality, tokens per second, tail latency and memory together; charge the scorer for its own runtime.
• Keep the scorer only where its gain survives repeated runs on the workload that will actually ship.

This is labelled editorial judgment, not a deployment result. The point is not that random eviction must win everywhere. It is that complexity needs evidence after structural protection is held constant.

What's in it for you

Before adopting another intelligent cache policy, add one embarrassingly simple competitor. If prompt-pinned randomness ties it, remove the scorer. If the scorer wins only on rare once-stated facts or long code inputs, route those workloads deliberately instead of taxing every request.

Protect the irreplaceable input first; make every clever selector beat randomness for the right to stay.