Meta Expert matches vanilla MoE accuracy with 70-80% of experts

A new preprint from researchers at Tsinghua University and Shanghai AI Laboratory, posted to arXiv on October 20, proposes a change to how Mixture-of-Experts models store knowledge: share experts across depth rather than keep a private set at every layer.

The paper, titled "Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE," opens on the redundancy problem. Mainstream models, the authors write, "adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content." Their alternative is a pool of "Meta Experts" that any layer can route into.

The headline result: at matched activated-expert counts and granularity, the Meta Expert configuration "matches the performance of Vanilla MoE using only 70-80% of the experts." The authors also report "lower cross-entropy loss and higher downstream task accuracy" against conventional Local Expert layouts, and argue theoretically that cross-layer sharing produces "more flexible residual pathways" for information retrieval.

The experimental sweep spans depth-ladder runs at 12, 16, 20 and 24 layers, with 128 routed experts per layer and top-8 routing, trained for 200B tokens at a 6B-A1.6B configuration. Evaluations include MMLU, C-Eval, CMMLU, BBH, OpenBookQA and TriviaQA.

The abstract publishes no per-benchmark comparison against named frontier MoE systems. The 70-80% figure is against the paper's own vanilla baseline at the 6B-A1.6B scale, not DeepSeek, Qwen-MoE or Mixtral.