KVpop shrinks Qwen3 KV cache 88%, keeps 94% of quality

Almost every LLM serving stack today runs into the same wall at long context lengths: the KV cache grows linearly with the sequence and eats VRAM. The KVpop paper posted to Hugging Face proposes a post-training retrofit that bounds the cache to a fixed budget per attention head and reportedly holds most of the base model's quality on top.

Prior learned methods like DMS pick which tokens to keep by training binary eviction gates with a differentiable relaxation, letting the model figure out a scoring rule during retrofit. KVpop takes a different swing, defining the 'right' answer explicitly. A token's target score is the attention mass it will actually receive from queries that come later, after the token has left a small protected recent window. During training the authors compute that target with a transposed-attention pass so they never materialize the full attention matrix. The scorer then learns to rank tokens by their real future utility, with a stateless MLP variant and a stateful mLSTM variant that can accumulate context before scoring.

The reported numbers back the setup. On AIME and HMMT math benchmarks, a Qwen3-4B distilled with KVpop retains 95% of the dense teacher's pass@1 at 75% cache compression and 94% at 88% compression. On Qwen3-8B those figures move to 95% and 99%. The learned policy transfers to LiveCodeBench code generation and GPQA Diamond science questions even though the sparsification training data was math-only. On memory, decoding Qwen3-8B out to 131k tokens takes 36GB of peak VRAM under dense attention versus roughly 19GB under both DMS and KVpop, and the paper reports KVpop hitting lower latency than DMS at the longest generations.

The honest caveats are that the paper only shows Qwen3 results, the retrofit still requires a 2,000-step distillation run on the Nemotron-Math v2 dataset, and the authors excluded sparse-retrieval methods like Quest and DeepSeek Sparse Attention from the head-to-head by design, since those keep the full cache rather than bounding it. The paper positions KVpop as a post-training retrofit rather than a compressed-cache architecture trained from scratch.

If the policy transfers to other model families, the practical upside is that hosted long-context inference gets cheaper without touching the base weights, and small teams running open Qwen models on a single GPU push their usable context length further before hitting OOM.