KAIST-Google: 'Declarative Attention' Cuts KV Reads 31-52% in LLMs
TL;DR
- The method lets a model emit
, , or tokens per step so the inference engine skips unneeded KV cache reads. - Off-the-shelf Gemma-4-31B cut attended tokens 52.0% for a 1.27pp accuracy loss; Qwen-3.6-27B saw 31.1% cuts and 2.75pp loss.
- Wall-clock decode ran at 0.71x vanilla on Gemma and 0.77x on Qwen across 15 long-context tasks.
Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. That one-line observation opens a new paper from KAIST AI and Google DeepMind and drives its whole design.
The proposal, Declarative Attention, hands the model three tokens it can emit (`<global>`, `<focus>`, `<local>`) to declare which part of the context the next decode step needs. `<global>` reads everything. `<focus>` narrows to a specific region. `<local>` looks at recent output only. The inference engine reads the declaration and skips the rest of the KV cache.
On off-the-shelf Gemma-4-31B and Qwen-3.6-27B, the authors report the mechanism "significantly reduces total attended tokens during decoding (52.0%, 31.1%)" for "modest accuracy drops (1.27pp, 2.75pp)" across 15 long-context tasks. Wall-clock decode landed at 0.71x vanilla on Gemma and 0.77x on Qwen.
The focus mode is more fragile at smaller scale. Success rates in picking the right region run from 58% on Gemma-4-E4B up to 99% on Gemma-4-31B. The abstract publishes no per-task accuracy breakdown, and does not compare DA against sparse-attention baselines like sliding-window or landmark attention.
It sits in a run of research we've tracked this week where the load-bearing move is letting the model itself decide what compute to spend next. Yesterday's Cliff paper made the same bet on the training side, teaching RL to spend its gradient on the first mistake in a rollout.
Originally reported by huggingface.co
Read the original article →Original headline: Anthropic-Google Paper: 'Declarative Attention' Lets LLMs Skip 31–52% of Cache Reads at Minimal Accuracy Loss