Copenhagen team maps MoE experts that carry persona features

A new mechanistic study from the University of Copenhagen locates personalization inside Mixture-of-Experts language models to a small, stable set of routing paths, and shows that boosting those experts in an unmodified base model shifts it toward personalized behavior. Pepa Atanasova and Isabelle Augenstein ran the experiments on three open MoE systems (OLMoE-1B-7B, Qwen3-30B-A3B and Mixtral-8x7B) and released code and data on GitHub. The paper is posted on Hugging Face.

Across the three families, the personalized MoEs matched or beat dense baselines of comparable total parameter count while activating only around a fifth of their weights per token. When the authors trained crosscoders on paired activations of the personalized and base models, fewer than 0.25% of latents dominated the reconstruction of personalized behavior: 157 of 65,536 for Qwen3, 132 of 131,072 for Mixtral and 111 of 65,536 for OLMoE. Most fired on the system message carrying the user preference rather than on the response, 85% for OLMoE and 82% for Qwen3, with Mixtral the exception at 41% response-side.

Causal interventions tested whether the mapped experts actually do the work. Patching the identified latents into the non-personalized model recovered the personalized model's next-token distribution better than random or shared-latent baselines, and steering just the top experts per preference dimension reduced KL divergence from the personalized counterpart with p<0.001 under Wilcoxon tests. "MoE encodes distinct personalization features via different routing paths, enabling better localization and subsequent intervention for controlled personalization," the authors conclude.

Three caveats sit on the surface of the paper. Expert attribution is correlational, crosscoder reconstruction explains 0.80 of variance for OLMoE but only 0.65 for Mixtral, and the preference signal comes from synthetic Multifaceted system messages in English rather than real user histories. It fits a steady run of expert-level MoE interpretability and PEFT work; our fine-tuning feed alone has carried 66 stories in the past 90 days.