FreeBalance cuts MoE prefill latency 13.1% via pre-routing
TL;DR
- FreeBalance predicts MoE routing distributions before the router fires, so expert-weight migration overlaps with pre-routing computation instead of stalling the critical path.
- The authors report a 13.1% cut in end-to-end prefill latency and a 32.8% drop in the max-to-mean rank load ratio.
- The scheme hides balancing overhead averaging 5.1 expert swaps per layer, otherwise about 8.5% of critical-path latency.
A 13.1% cut in end-to-end prefill latency for Mixture-of-Experts inference. That is the headline claim in a new preprint on arxiv from Pengfei Chen and four co-authors, posted August 14.
In distributed MoE inference, whichever rank draws the heaviest routing load stalls the whole batch. Standard fixes wait for the router to fire and then migrate expert weights, which puts the migration squarely on the critical path. FreeBalance, the authors' method, predicts the routing distribution first and starts moving weights while attention is still running. The paper's pitch: it uses 'cross-layer similarities in hidden representations within the residual network to build a lightweight workload predictor,' and a cost model caps how many swaps get scheduled so they fit inside the available window.
The headline numbers: a 32.8% reduction in the max-to-mean rank load ratio, and a method that hides 'balancing overhead of an average of 5.1 experts per layer, which would otherwise account for about 8.5% of the critical-path latency.' The authors call the framework 'lossless.' Two AI researchers we follow passed the paper around this week.
The abstract does not name the specific MoE models or datasets used in the evaluation, and it reports no absolute throughput, hardware, or wall-clock figures beyond the percentages.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: FreeBalance: Pre-Routing Online Moe Load Balancing via Residual Workload Prediction