E-MoE turns MoE routing into shared latent for diffusion LMs

Masked diffusion language models have a factorization problem. At each denoising step they sample several tokens from a distribution that treats positions as independent, which the authors argue is the thing holding quality down in the few-step regime, the regime where diffusion's speed advantage over autoregressive decoding is supposed to actually matter.

Enhanced Mixture-of-Experts (E-MoE), a method posted to arXiv by Arseny Ivanov, Alexander Kolesov, Alexander Korotin, Ivan Oseledets and Mikhail Goncharov, reframes the reverse process as 'a mixture of factorized distributions over a discrete shared latent given by the expert-routing decisions of a Mixture-of-Experts (MoE) backbone, without increasing active parameters over the factorized baseline.' The MoE routing choice itself becomes the shared latent each position can condition on.

The authors position the move against an earlier line of work using a continuous Gaussian latent trained as a variational autoencoder, which they describe as 'prone to posterior collapse, where the latent is silently ignored.' Across 'synthetic multi-modal benchmarks, binarized MNIST, and LM1B,' the paper says E-MoE 'improves few-step generation over factorized baselines.'

The abstract publishes no MAUVE scores, no NFE values, and no head-to-head against autoregressive sampling at matched compute.