MSAR paper claims 1.58x compute speedup at 28B MoE scale

A new preprint from Xiaohan Qin and co-authors claims that a redesigned residual connection, called Multi-Stream Attention Residuals, delivers a 1.58x iso-loss compute speedup over the standard residual baseline when training mixture-of-experts models at 18B and 28B parameters. The prior best variants on the same benchmark, AttnRes and manifold-constrained Hyper-Connections, came in at 1.35x and 1.30x.

The method, posted on arXiv, generalises AttnRes from single-stream depth attention to multi-head depth attention. "MSAR draws stream diversity from sequence context through multi-scale causal convolutions, allowing different heads to retrieve historical representations through distinct contextual views while preserving the integrity of the hidden representation," the abstract reads.

The retrieved streams are combined through what the authors call "lightweight adaptive mixing" before being fed as the single-stream layer input. The claim worth sitting with: MSAR "supports multi-stream computation without requiring persistent multi-stream storage or communication." That is where the compute win is supposed to come from.

Across all 12 downstream benchmarks at both tested scales, the paper reports MSAR outperforming standard residual connections, mHC, and AttnRes. The abstract publishes no per-benchmark accuracy numbers, no author affiliations, and no wall-clock timings.