KV cache gate holds 90.14 RULER at 75% compression on Qwen3-8B
TL;DR
- A new learnable gating approach for KV cache eviction places the gate inside the softmax normalizer, redistributing suppressed entries' attention mass rather than discarding it.
- Training the gate on standard language-modeling loss fails because the frozen backbone was already optimized for dense attention, so the loss prefers a no-op.
- At 75% cache compression on Qwen3-8B, the gate scores 90.14 on RULER at 8K and 46.03 on LongBench, 3.72 and 1.36 points below full cache.
Train a learnable gate on each cached entry's attention weight, then place the gate inside the softmax normalizer so suppressing one entry hands its mass to the others rather than discarding it. That is the proposal in a paper posted to arxiv, 'The Gating Mechanism of KV Cache Eviction.'
The plan runs into a trap. Trained on the standard language-modeling loss, the gate stays near a constant score while its loss falls. The authors' own line: 'a frozen backbone was fit for dense attention, so its loss prefers the no-op, and a near-constant score ranks nothing.' Their fix is to put dense attention beyond the gate's reach and make the dense model's predictions the target.
Under those two conditions, the paper argues, capacity stops being the missing ingredient: 'neither a larger scorer, nor explicit access to the past, nor a trainable backbone helps.'
At 75% KV cache compression on Qwen3-8B, the gate reaches 90.14 on RULER at 8K and 46.03 on LongBench, 3.72 and 1.36 points below full-cache inference. The abstract publishes no per-task breakdown and no numbers at other compression ratios or longer contexts.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: The Gating Mechanism of KV Cache Eviction