Random Attention Matches Top KV Scorers 32-43% Faster in vLLM

"Large language models achieve superior performance on tasks that require extended reasoning, but long chains of thought make the KV cache a severe memory bottleneck," a new preprint on arXiv opens. Its answer is to stop scoring.

Ten authors, led by Heng Wang, argue that the entire selection-signal design of KV-cache compression contributes almost nothing. Their method, Random Attention, "keeps the prompt and evicts uniformly at random within each attention head, computing no score at all." Across four models and six reasoning tasks, they report it matches the strongest prior evictor while serving "32-43% higher throughput than it in vLLM deployment."

The paper offers two reasons the trick works. "The prompt is the fragile part of the cache, and most of the gap between selectors is just whether their selection signal happened to keep it." Beyond that, the reasoning trace has redundancy at two levels: the model "restates what it still needs as it works," and each attention head keeps its own copy of the trace. Once the prompt is safe, a random draw retains enough copies of what the model still needs.

The abstract does not name the four models, the six tasks, or the "strongest prior evictor" the comparison rests on. It lands during a busy stretch of KV-cache and serving work on our inference feed, alongside same-day work like Minima's NVFP4 quantization of Qwen3.8-27B GDN layers.