Letting a model route attention creates a new thing to audit

Declarative Attention lets a model emit instructions that change which parts of its context it attends to. In the September 2 paper, Gemma 4-31B uses 52.0% fewer attended tokens with a 1.27 percentage-point average accuracy drop across 15 long-context tasks. For Qwen3.6-27B, the reported trade-off is 31.1% fewer attended tokens and a 2.75-point drop. Paper and reported results.

Those are promising attention reductions, not equivalent reductions in an API bill or measured production latency. The distinction matters because the model spends additional decoding steps directing attention, while the remaining attention work becomes smaller.

My bet is that model-directed attention will need a decision record much like retrieval does. Once the model decides what deserves another look, an evaluation has to examine the evidence it skipped as well as the answer it returned.

Focus is a decision with consequences

The paper introduces global, focus and local modes that an inference engine interprets. Its implementation uses a custom vLLM metadata hook to adjust attention access. It avoids new kernels and scheduler changes, but that is not the same as requiring no serving integration. Implementation details.

The mechanism creates an appealing division of labour: the model can indicate where relevant information is, and the serving system can avoid some unnecessary attention work. The difficult case is information whose relevance becomes clear only after it is combined with something elsewhere.

Imagine a hypothetical contract-review task. The main agreement gives a delivery date, while a later amendment changes the acceptance condition that makes the date meaningful. A model that focuses confidently on the main agreement may produce a fluent answer while losing the qualification. I would want the evaluation to expose that failure, not reward the smaller attention footprint in isolation.

This is a test design I would propose, not an experiment I have run with the authors' implementation. The published averages establish a trade-off on the benchmark suite; they do not tell us the acceptable loss for a particular contract workflow.

Measure the work that remains

I would build paired examples where the answer is local in one version and requires two distant passages in the other. Keep the wording and total context broadly comparable, then inspect how the attention decisions change. A method that saves work on the first version should still recognise why the second deserves broader access.

For each pair, keep the answer, cited evidence, attention-mode transitions and actual end-to-end duration. That record can distinguish an efficient correct answer from an inexpensive miss. It also makes it possible to investigate whether an apparent regression comes from routing, additional decoding or the underlying model.

The paper's wall-clock analysis is projected rather than a production speed measurement. A deployment evaluation should therefore measure latency and resource use in the intended serving setup, including the routing overhead. Percentage reductions in one operation do not automatically become the same percentage improvement in the whole request. Evaluation scope.

There is a separate boundary worth preserving: attention selection is not access control. If a model can choose to focus elsewhere, that does not establish that sensitive material was never accessible to it. Data permissions need enforcement before material enters a context the model may inspect.

The useful next step is a workload-specific trial with an explicit tolerance for missed evidence. I would start with tasks where the required facts are known and deliberately scattered, then earn the right to use the savings on less inspectable work.

Count the attention saved, but keep a record of the evidence the model chose to stop considering.