Hugging Face publishes 70,000-word MoE training handbook

Hugging Face has published a long-form training manual for Mixture-of-Experts models, built on roughly 800,000 GPU hours of its own experiments and running to about 70,000 words and 55 interactive figures. The Ultimate MoE Training Handbook is credited to Nouamane Tazi, Leandro von Werra and Thomas Wolf.

"18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over," Wolf wrote on X, framing the release as an attempt to move know-how out of a handful of frontier teams and into public reference material.

The handbook is organized around the practical questions a team hits during an MoE run: how many experts and how large, which router function to use and why to keep it in fp32, whether to pick DeepSeek's loss-free bias instead of an auxiliary loss, whether to upcycle a dense checkpoint or train from scratch, and whether a model can be served with fewer experts per token than it was trained with.

Several of the numbers come from a single eight-H100 benchmark. On that run, half a step was consumed by exposed all-to-all communication; swapping the learned router for forced-even routing improved an iteration by about 40%, according to a newsletter summary of the handbook's diagnostic section. Two separate runs that fell back to sockets instead of the requested EFA transport ran about 25 times slower, with no error in the logs.

Two of the researchers we track posted the link the day it dropped.