Muon-RT cuts persistent memory 42% by reusing EMA teacher
TL;DR
- Muon-RT removes Muon's separate momentum buffer by reusing the teacher-student displacement already stored by EMA self-distillation.
- On Qwen2.5-7B-Instruct with SDFT, it beats AdamW by 2.6 accuracy points while using 42% less persistent memory.
- On Olmo-3-7B-Instruct with SDPO, it beats AdamW by 2.0 accuracy points; the abstract reports no per-benchmark numbers.
If a training run already keeps an exponential-moving-average teacher around, Muon's separate momentum buffer is redundant, three researchers argue in a new preprint: the teacher-student gap carries the same signal.
In the paper, Aryan Das, Vinay Kumar Verma and Aakansha Mishra describe Muon-RT, which 'reuses the teacher-student displacement to eliminate momentum buffers for hidden weight matrices.' Mechanically, as stated: the method 'removes its radial component, normalizes it spectrally, and mixes it with the current gradient after Newton-Schulz (NS) orthogonalization.' The claim is that this displacement 'encodes an exponentially weighted history of applied updates,' which is what a momentum buffer is for.
The reported numbers come from two recipes on two 7B bases. Under Self-Distillation Fine-Tuning on Qwen2.5-7B-Instruct, Muon-RT 'improves average accuracy over AdamW by 2.6 points with 42% less persistent memory,' and over momentum Muon by 5.4 points with 27% less persistent memory. Under Self-Distillation Policy Optimization on Olmo-3-7B-Instruct, it improves average accuracy over AdamW by 2.0 points. The abstract gives no per-benchmark breakdown, no wall-clock figures, and no results outside these two base models.
Originally reported by paper
Read the original article →Original headline: Muon-RT Cuts LLM Self-Distillation Training Memory 42% by Reusing EMA Teacher as Momentum Buffer