SMELT loops middle MoE layers twice, saves 6.8-18% FLOPs
TL;DR
- SMELT reuses the middle half of a Mixture-of-Experts Transformer's layers twice while matching baselines on per-token FLOPs, non-embedding parameters, and KV cache.
- The recipe saves 6.8 to 18.0 percent of training FLOPs on the compute-optimal frontier across four scales up to 54B non-embedding parameters.
- Gains transfer to downstream benchmarks beyond validation loss, are largest on Code, and grow with sample length and in-context examples.
The claim in a new paper posted to Hugging Face: looping the middle half of a Mixture-of-Experts Transformer's layers twice saves 6.8 to 18 percent of training FLOPs on the compute-optimal frontier, without adding per-token FLOPs, non-embedding parameters, or KV cache.
The authors call the recipe SMELT (Sparse MoE Transformer, middle layers Loop Twice) and scale it across four sizes up to 54B non-embedding parameters, fitting a separate Chinchilla-style scaling law for each architecture. "SMELT's loss drops faster with compute, saving 6.8--18.0% of training FLOPs on the compute-optimal frontier," the abstract states. The advantage "transfers to downstream benchmarks beyond what validation loss predicts, is largest on Code, and grows with sample length and the number of in-context examples."
Why the second pass helps, in the authors' words: "the second visit reduces the attention sink and redirects mass toward content-relevant tokens." The paper lands amid a run of looping and MoE work moving through the open-source research feeds we track, including this week's 305B V4-Flash-Vision-Exp release from DeepSeek.
Originally reported by huggingface.co
Read the original article →Original headline: SMELT Paper Shows Compute-Matched Looped Transformers Save 6.8–18% of Training FLOPs