ACE halves MoE expert use without training or calibration
TL;DR
- ACE claims to skip half of a mixture-of-experts model's experts per token with no training, no calibration data, and no checkpoint edits.
- At 50% skipping on Qwen3.6-35B-A3B, the paper reports a 7.96% cut in WikiText-2 perplexity and 4.15 point accuracy gain over the strongest baseline.
- The authors also measure up to 2.25× prefill and 1.41× decoding speedups from optimized expert dispatch.
Half the experts inside a mixture-of-experts language model can be skipped per token with no calibration data, no retraining, and no checkpoint edits, according to a preprint posted to arxiv. The method, called ACE, pairs a Global Spectral Proxy over the coupled gate, up and down projections with a Router-Conditioned Refinement that scores expert responses along routing-preferred directions. An expert slot is dropped only when both views agree it is low-contribution, 'while always retaining the top-1 expert.'
The authors report tests across 'three MoE-based LLMs and eight benchmarks.' At a 50% skipping ratio on Qwen3.6-35B-A3B, they write that ACE 'reduces WikiText-2 perplexity by 7.96% and improves average downstream accuracy by 4.15 percentage points over the strongest competing method,' with 'up to 2.25× prefill and 1.41× decoding speedups' credited to optimized expert dispatch.
One unreviewed submission, no third-party replication yet; two researchers we follow had already circulated the link by the time we pulled the paper. Code is posted at github.com/xzktx003/moe-prune.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs