Found first: a primary source the press has not covered yet.
Baidu researchers have posted Virtual Head Attention (VHA), accepted to NeurIPS 2026, claiming to resolve what the paper calls the "impossible triangle" among KV cache size, training compute, and model quality. At dense scale, VHA reduces KV cache 4× versus GQA-8. On a 30B-A3B MoE model evaluated across 12 benchmarks, it outperforms GQA-4 by +2.60% and MLA by +0.70%.
What the source says
Authors Xibo Ding, Guoxia Wang, Jinle Zeng, Jiabin Yang, Sirui Tao, Chao Yang, and Dianhai Yu, all at Baidu, introduce two lightweight linear operations: Q Premix, a near-identity transform that recovers head diversity from halved physical query heads, and Linear Postmix, a low-rank residual that fuses inter-head features. Both fold into standard projection matrices at inference, adding no overhead beyond what GQA-2 already requires. Across four pretraining scales at 4K context and 50B tokens, VHA beats the GQA baseline by +0.05% to +0.81%. On a 10B-A1B MoE at contexts from 8K to 128K, VHA scores 84.4 on the RULER benchmark versus 73.0 for GQA. Training and prefill FLOPs fall 7.9% at the 1.7B scale at sequence length 4096.
Why it matters
GQA is the current default in most production LLM training runs. MLA, used in DeepSeek models, achieves comparable cache efficiency but requires a different inference path and adds prefill compute. VHA fits into a standard GQA-2 inference path with no new kernels and no nonlinearity, making it a potential drop-in replacement for labs currently training with GQA. The long-context result is the sharpest signal: 84.4 versus 73.0 on RULER across 8K to 128K covers the retrieval and recall tasks that drive most RAG and agent workloads. The largest published experiment is 30B parameters, so replication at larger scales remains the open question.