Zhejiang, PolyU paper reframes MoE expert overlap as coherent

Something interesting out of Hangzhou this week: a paper posted to Hugging Face's papers hub by researchers at Zhejiang University and the Hong Kong Polytechnic University argues that the standard mental model of sparse mixture-of-experts routing has been quietly wrong. The intuition most teams carry, that when a token is routed to several experts each one contributes a different slice of the representation, does not survive a careful test.

The authors introduce an Expert Subspace Separation Index (ESSI) and a prefix-controlled 2x2 factorial protocol, and run them across six open MoEs including OLMoE, Mixtral, DeepSeek-MoE, Qwen3, Gemma4 and Qwen3.6. Two findings stand out. Expert subspaces overlap substantially, with ESSI landing near one across the geometry survey (range 0.776 to 1.060, median 0.969). And when they cross candidate experts with actual and matched-alternative prefixes across 39 factorial cells, the interaction is negative in every cell, with every 95% confidence interval below zero. The actual co-selection context narrows a candidate expert's advantage rather than amplifying it, which is the opposite of what geometric complementarity would predict.

Why this matters if you are working with MoE stacks: a chunk of the MoE compression and pruning literature treats input-subspace overlap as a proxy for redundancy. The paper's frozen-route interventions push back on that assumption. Adding later experts reduces next-token NLL in 24 of 39 cases, and a compute-matched Top-1 versus Top-2 training study favors Top-2 in all three seeds. Overlap in geometry does not mean overlap in function, which the authors label 'coherent overlap'.

The honest caveat is scope. The full functional protocol only ran on OLMoE, Mixtral and DeepSeek; the newer stacks in the survey were only checked for geometry, and the paper does not tell you what a router objective that would produce genuinely useful geometric complementarity should look like. What the reporting doesn't give you is any test on the very largest deployed MoEs, nor any inference-latency or kernel-design implications.

The forward-looking read is that anyone building compression, merging or expert-skipping tooling now has a rigorous diagnostic to run before shipping, and a fresh reason to test at the output level on the retained route rather than trusting subspace similarity as a shortcut.