ByteDance SAE finds Qwen3-30B-A3B experts split by subdomain

ByteDance Seed researchers trained a sparse autoencoder on the output of Qwen3-30B-A3B's mixture-of-experts layers and found that most learned features are principally driven by just one or two experts, with distinct specialists for Java and front-end web code and separate ones for geometry versus arithmetic and polynomials.

The method, laid out in a paper on Hugging Face, leans on a structural fact about MoE routing: because a layer's output is a router-weighted sum of expert outputs and the SAE encoder is affine, the pre-activation of every latent 'decomposes exactly over the experts.' That lets each expert be assigned a dominant set of latents whose activation it principally drives, rather than being profiled by how often the router selects it on a pre-labeled corpus.

On layers 4, 30 and 46 of Qwen3-30B-A3B, 'contribution is far more concentrated than gating, most latents have only one or two dominators, and the overlap between dominant sets concentrates on a few expert pairs,' the abstract reports. Auto-interpreting those dominant sets surfaces 'a division of labor below coarse domains,' with the coarse domains themselves taking 'very unequal numbers of experts.'

The authors (Anyuan Zhuo, Shaohua Zhang, Yuan Lin and Pinyan Lu) present the move as a correction to earlier expert-attribution work, which 'fixes the taxonomy in advance, stays coarse, and records selection rather than contribution.'

Code, the trained SAEs and the Qwen3-30B-A3B-MoE-SAE model are public under the project's ByteDance-Seed GitHub and Hugging Face repos.