MHAR adds +3.2 GSM8K, +3.1 GPQA at 8B with zero new parameters

A quiet July paper on arxiv has stuck with me because it makes an unfashionable claim: you can still squeeze real capability out of the Transformer architecture without adding a single parameter. The technique, Multi-Head Attention Residuals, reportedly lifts an 8B model by +3.2 on GSM8K and +3.1 on GPQA when converted mid-training, and shows consistent validation-loss gains at 100M, 350M, and 1B when trained from scratch.

The diagnosis is more interesting than the fix. Standard Transformers push information forward through a single additive residual stream, so every sublayer reads only the most recent state. Attention residuals relaxed that by letting each sublayer attend across depth through a learned softmax. But that read used one query shared across the entire width, so every feature subspace had to agree on which past layers to consult. The authors argue that as models get wider, subspaces disagree more, and forcing them through one distribution costs you.

MHAR reshapes the routing query into H per-subspace heads, each with its own softmax over the depth history. The read becomes block-diagonal, the reshape adds zero parameters and negligible compute, and H=1 recovers attention residuals exactly. On an anneal corpus that is quality-filtered and STEM- and code-heavy, the authors report validation loss improvements of 0.061, 0.149, and 0.140 at the three scales, with the gain growing with size. Their fused Triton routing kernels take training throughput from 0.2-0.5x of baseline up to 0.55-0.88x, which is the more honest number to watch than the benchmark deltas.

The honest caveats are in the paper itself. Validation loss is U-shaped in H with a flat optimum at 4 or 8, and pushing to 16 gives back part of the gain, so this is a real design axis, not a free knob. The 8B benchmark numbers come from a mid-training conversion using delta attention residuals rather than a from-scratch 8B run, which is a meaningful asterisk. And what the reporting doesn't give you is downstream behavior beyond GSM8K and GPQA, or a head-to-head against other recent low-parameter residual-stream tricks.

If it holds up under replication, the practical read is that architectural refinements with zero parameter cost are still on the table at frontier scale, and any team already sitting on an 8B-class checkpoint has a cheap enough mid-training upgrade path to be worth prototyping on their own evals this quarter.