Shanghai Jiao Tong MSAR Hits 1.58x Iso-Loss Speedup at 18B-28B MoE

Multi-Stream Attention Residuals, a new residual-connection design from Shanghai Jiao Tong University, Xiaohongshu's Dots Studio and Peking University, reaches the same pre-training loss as a standard residual baseline using 1.58x less compute. In the paper, posted to Hugging Face this week, MSAR is pitted against two recent challengers to the standard residual: manifold-constrained Hyper-Connections (mHC) and Attention Residuals (AttnRes). On matched scaling-law curves, mHC's iso-loss speedup is 1.30x, AttnRes is 1.35x, and MSAR is 1.58x.

The authors frame their contribution in attention terms. "A natural question emerges: can we realize multi-head depth attention, bringing greater expressiveness to residual retrieval over depth?" they write. Their move is to run a small set of multi-scale causal depthwise convolutions over each layer output, use the resulting streams as separate heads that each do their own softmax retrieval across depth, and fuse the retrieved streams back into one layer input. The expanded streams only exist in memory while depth attention is computing; the paper puts persistent storage at the same O(Nd) as AttnRes.

At the 18B MoE scale, final training loss drops from Vanilla's 1.799 to MSAR's 1.750, with mHC at 1.776 and AttnRes at 1.769. At 28B MoE, the authors report MSAR "increases training FLOPs by only 0.5% and activated parameters by 0.3%."

The 1.58x figure comes from scaling-law experiments rather than a direct wall-clock measurement on a full 28B run. The abstract does not break out per-task numbers for the 12 downstream benchmarks it tests against. Residual-architecture tweaks have been a steady beat in our open-source tracker lately; see yesterday's note on an SFT-efficiency paper out of Nanjing and Fudan.