Baidu's Virtual Head Attention claims 4× KV cache cut vs GQA

Baidu researchers claim they can shrink an LLM's KV cache by 4× while nudging benchmark scores up, with no change to how the model runs at inference.

The method is called Virtual Head Attention. In a NeurIPS 2026 paper, Xibo Ding, Guoxia Wang and colleagues at Baidu describe two small linear operations grafted onto grouped-query attention: a Q Premix that "applies a near-identity transformation to queries within each KV group to recover virtual head diversity from halved physical query heads," and a Linear Postmix that "fuses inter-head features via a low-rank $I + AB^\top$ residual structure." Because neither step introduces a nonlinearity, both can be folded back into the standard query and output projections at inference. The paper's framing: Virtual Head Attention collapses to a "standard GQA-2 architecture that directly reuses all existing GQA inference optimizations."

The pitch is aimed squarely at what the authors call an "impossible triangle" among KV cache size, training and prefill FLOPs, and model quality.

Across four pretraining scales with 50 billion tokens at 4K context, VHA beats a GQA baseline by between +0.05% and +0.81%, while reducing KV cache 4× versus GQA-8 at dense scales and 2× versus GQA-4 at the MoE scale. Training and prefill FLOPs drop by 7.9%. On an industrial 30B-A3B mixture-of-experts model trained on 500B tokens at 8K context, VHA reportedly outperforms MLA and GQA-4 by +0.70% and +2.60% on a 12-benchmark base-model suite. A 10B-A1B MoE scores an average RULER of 84.4 across 8K-128K context versus 73.0 for GQA.

The quality gaps on the smaller pretraining scales start at +0.05%, tight enough to leave replication an open question. The 12-benchmark suite behind the MoE comparison is not identified in the abstract.