Motif Details 314B Motif 3 MoE with 13.2B Active per Token

Motif Technologies has posted the technical report for Motif 3, a 314-billion-parameter sparse mixture-of-experts model that activates 13.2 billion parameters per token. The paper on Hugging Face describes 384 routed experts with top-8 routing across 53 transformer layers, plus one shared expert per layer.

The architectural bet is Grouped Differential Latent Attention, or GDLA, which stitches together grouped differential attention with the latent key-value compression of Multi-head Latent Attention. The paper claims GDLA reaches a 3.2 loss with 9.2% fewer tokens than an MLA baseline. Each expert also gets its own learnable polynomial activation, called Expert-Specific PolyNorm, meant to encourage specialization rather than the shared SiLU nonlinearity most MoEs use.

Training ran on roughly 12.5 trillion tokens with MoE experts stored in MXFP8, activations in BF16, and optimizer states in FP32. The context window schedule expanded from 4K to 32K to 256K using window-aware Ring Attention. Reported base-model scores include 86.20% on MMLU (5-shot), 93.93% on GSM8K, 73.70% on HumanEval, and 70.58% on MATH. Post-training uses Multi-teacher On-Policy Distillation across seven specialized teachers spanning 13 verifier domains.

These are self-reported numbers from the paper page, not third-party evaluations, and nothing in the retrieved summary covers contamination screening, safety testing, or a head-to-head against other contemporary MoEs under matched conditions. The report describes an architecture and a training recipe; teams considering it will want independent runs on their own eval harness before committing GPUs.

Motif's earlier Beta release already appeared on AI Weekly's open-source tracker, one of 289 open-source releases we've logged there in the last 90 days, and the more useful question for anyone building sparse MoEs of their own is whether GDLA and Expert-Specific PolyNorm hold up when other labs try to reproduce the token-efficiency claim.