Router Prior Bias preserves MoE routing during post-training
TL;DR
- Router Prior Bias anchors router logits softly to the frozen base router while keeping the router itself trainable during post-training.
- On math post-training of Moonlight-16B-A3B, RPB reached 45.77 in-domain accuracy versus 31.91 for re-applied load-balancing loss and 29.44 for unanchored fine-tuning.
- The authors replicate on Qwen3-30B-A3B-Base and argue softness of the constraint matters more than the specific prior chosen.
A new arXiv paper argues that the load-balancing loss used to keep experts evenly used during MoE pretraining should not be re-applied during post-training. On math fine-tuning of Moonlight-16B-A3B, the authors report 45.77 in-domain accuracy for their Router Prior Bias method against 31.91 with re-applied LBL and 29.44 for unanchored fine-tuning.
The technique the authors call "soft router anchoring" "gently constrains router logits toward the frozen base router while keeping the router itself trainable." They replicate the finding on Qwen3-30B-A3B-Base and conclude that neither fully flattening the inherited routing nor rigidly enforcing it produces optimal results. The softness of the constraint, they write, matters more than the specific prior used.
The paper's abstract publishes no per-expert utilization data, no wall-clock overhead, and no results beyond the two named checkpoints. Two researchers we track had circulated the link by early this week.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: Router Prior Bias: Preserving Base Routing Structure in MoE Post-Training