What Is Mixture of Experts (MoE)? How Modern LLMs Get Efficient
Mixture of Experts (MoE) is an architecture that lets AI models have trillions of parameters while only using a small fraction of them for any given input. It is the design pattern behind the most capable and cost-effective large language models in production today, including GPT-4, DeepSeek-V3, Mixtral, and Llama 4.
The core idea is simple: instead of processing every input through the entire network, split the computation among specialized sub-networks called experts. A routing mechanism selects which experts handle each input token, activating only a few while the rest stay idle. The result is a model that has the knowledge capacity of a massive network but the computational cost of a much smaller one.
In 2026, nearly all frontier language models use some form of MoE architecture. Understanding how it works is essential for anyone building with, deploying, or evaluating modern AI systems.
Why Mixture of Experts Exists
The motivation for MoE comes from a fundamental tension in AI scaling.
The Scaling Problem
Larger models are more capable. This has been one of the most consistent findings in deep learning research. A model with 70 billion parameters generally outperforms a model with 7 billion parameters on a wide range of tasks. The relationship between scale and capability is so reliable that it is formalized as scaling laws.
But larger models are proportionally more expensive to run. A dense model, where every parameter is active for every input, requires computational resources that scale linearly with parameter count. Doubling the parameters doubles the cost of each forward pass. At the scale of frontier models with hundreds of billions or trillions of parameters, this becomes prohibitively expensive for real-time applications.
The MoE Solution
MoE breaks the link between total parameters and computational cost. A model can have 671 billion total parameters, like DeepSeek-V3, but only activate 37 billion for each token. The model stores knowledge across all 671 billion parameters but pays the compute cost of a 37 billion parameter model at inference time.
This is called sparse activation. The model is sparse because most of its parameters are inactive for any given input. Only the relevant experts fire. The irrelevant ones contribute nothing and cost nothing.
How Mixture of Experts Works
An MoE layer replaces the standard feed-forward network (FFN) in a transformer block. Instead of one large FFN that processes every token, there are multiple smaller FFNs (the experts) and a gating mechanism (the router) that decides which experts process each token.
The Expert Networks
Each expert is a standard feed-forward neural network, identical in architecture but with different learned weights. In a typical MoE configuration, a layer might have 8, 16, 64, or even 256 experts. Each expert has the same input and output dimensions as the FFN it replaces, but its hidden dimension is smaller proportionally.
Think of experts as specialists. One expert might become particularly good at processing scientific text. Another might specialize in dialogue. A third might handle code. This specialization emerges naturally during training without being explicitly programmed.
Researchers have observed that trained MoE models develop distinct expert roles: syntactic experts handle grammatical structures and verb conjugations, semantic experts process meaning and conceptual relationships, domain experts specialize in topics like scientific text or dialogue, and numerical experts handle arithmetic, dates, and quantities.
The Router (Gating Network)
The router is a small neural network, typically a single linear layer followed by a softmax or sigmoid function, that takes each input token and produces a score for every expert. These scores represent how relevant each expert is for that particular token.
The router then selects the top-k experts with the highest scores. Most architectures use k=1 or k=2, meaning one or two experts process each token. The outputs of the selected experts are combined using the router scores as weights.
The routing decision is made independently for each token. In a single sentence, different words might be routed to different experts. The word "photosynthesis" in a sentence about biology might activate a different expert than the word "however" in the same sentence.
The Complete Forward Pass
Here is how a token flows through an MoE transformer layer:
- The token enters the transformer block and passes through the attention mechanism as normal.
- At the FFN stage, the token is sent to the router.
- The router computes scores for all experts and selects the top-k.
- The token is processed by each selected expert independently.
- The expert outputs are weighted by the router scores and summed.
- The combined output continues to the next layer.
Attention layers are not replaced by MoE. They remain dense, meaning every token attends to every other token as normal. Only the FFN layers become sparse. This hybrid design preserves the attention mechanism's ability to model relationships between tokens while making the computationally expensive FFN layers efficient.
The Load Balancing Challenge
MoE architectures face a critical practical problem: load balancing. If the router sends most tokens to a few popular experts while others sit idle, the system loses its efficiency advantage. Popular experts become bottlenecks, and the computational savings from sparsity disappear.
Why Load Imbalance Happens
Routers tend to develop preferences. During training, an expert that happens to get good early gets more tokens, learns faster, and becomes even better, attracting even more tokens. This positive feedback loop, sometimes called expert collapse, can result in most tokens being routed to just one or two experts while the rest are barely used.
Traditional Fix: Auxiliary Losses
The standard solution has been to add an auxiliary loss term to the training objective that penalizes uneven expert utilization. This loss encourages the router to distribute tokens more evenly across experts. The downside is that the auxiliary loss competes with the main language modeling objective. Pushing for balanced routing can hurt model quality because sometimes the best expert for a token is an already-popular one.
DeepSeek's Innovation: Auxiliary-Loss-Free Balancing
DeepSeek-V3 introduced a fundamentally different approach that eliminates auxiliary losses entirely. Instead, they add a dynamic bias term to each expert's routing score. The bias is adjusted after each training step based on the expert's load: if an expert is overloaded, its bias is decreased, making it less likely to be selected. If an expert is underutilized, its bias is increased.
This approach has a key advantage: the bias term is used only for routing decisions and is not included in the training loss. The load balancing mechanism does not compete with the quality optimization objective. The result is better model quality with balanced routing, a significant engineering achievement.
Fine-Grained Experts and Shared Experts
Not all MoE architectures are created equal. The design of the expert structure significantly impacts performance.
Fine-Grained Expert Segmentation
DeepSeek's MoE architecture uses more fine-grained experts than traditional designs. Instead of N large experts, DeepSeek uses mN smaller experts, where each expert has 1/m the hidden dimension of a standard expert. More experts are activated per token, but each expert is smaller.
The rationale is that finer-grained experts allow knowledge to be decomposed more cleanly. Instead of one expert handling all of biology, separate experts might handle molecular biology, ecology, and anatomy. This finer decomposition leads to better specialization and more efficient use of model capacity.
Shared Experts
DeepSeek introduced the concept of shared experts that process every token regardless of routing decisions. The insight is that some computations are universally useful. Basic language understanding, common syntactic patterns, and general knowledge benefit every token. Shared experts handle these common computations while routed experts handle specialized processing.
In DeepSeek-V3, shared experts run on every token while the router selects additional routed experts based on the token's content. This ensures that foundational capabilities are always available while specialized knowledge is activated on demand.
Routing Mechanisms: Softmax vs. Sigmoid
The choice of gating function in the router has significant implications.
Softmax Routing
Traditional MoE architectures use softmax gating, where expert scores are normalized to sum to 1. This creates competition between experts: if one expert's score goes up, others must go down. While this produces clean routing decisions, the forced competition can prevent a token from strongly activating multiple relevant experts.
Sigmoid Routing
DeepSeek-V3 uses per-expert sigmoid gating, where each expert's score is computed independently. A token can independently select multiple relevant experts without forced competition. If two experts are both highly relevant, both get high scores. This independent scoring allows more flexible routing and better utilization of expert capacity.
Major MoE Models in 2026
MoE has moved from experimental curiosity to the default architecture for frontier models.
GPT-4
GPT-4 is widely believed to use an MoE architecture, though OpenAI has never officially confirmed its design. Reports suggest approximately 1.8 trillion total parameters across 16 experts, with a subset activated per token. The MoE design would explain how GPT-4 achieves its broad capabilities while maintaining reasonable inference costs.
Mixtral 8x7B and 8x22B
Mistral's Mixtral models made MoE accessible to the open-source community. Mixtral 8x7B has 8 experts per layer with 2 active per token. While the model has 47 billion total parameters, it only uses about 13 billion per token during inference. This gives it the quality of a much larger model at the cost of a much smaller one. Mixtral demonstrated that MoE could work at scales accessible to individual researchers and small companies.
DeepSeek-V3
DeepSeek-V3 is a 671 billion parameter MoE model with 37 billion activated per token. It competes with GPT-4o on quality benchmarks while being significantly cheaper to train and run. DeepSeek's innovations in auxiliary-loss-free load balancing, fine-grained expert segmentation, shared experts, and sigmoid routing represent the state of the art in MoE architecture design.
DeepSeek-V3 was trained for roughly $5.5 million in compute, a fraction of what comparably capable models cost. The MoE architecture is a key enabler of this efficiency.
Llama 4
Meta's Llama 4, released in 2025, adopted MoE for the first time in the Llama family. This signals that MoE has become the expected architecture for frontier open-weight models, not an exotic alternative.
Mistral Large 3
Mistral Large 3 uses MoE architecture, continuing the company's investment in sparse models. The model targets enterprise deployment where inference cost is a primary concern.
Advantages of MoE
MoE provides several concrete benefits over dense architectures.
Computational Efficiency
The headline advantage: a model with N total parameters and k active experts per token costs roughly k/N as much to run as a dense model of the same total size. DeepSeek-V3 activates about 5.5% of its parameters per token. This makes it feasible to deploy trillion-parameter models in production settings where latency and cost matter.
Knowledge Capacity
Total parameter count determines how much knowledge a model can store. MoE allows models to store vastly more knowledge without proportional compute cost. This is why MoE models tend to outperform dense models with the same inference budget: they simply know more.
Training Efficiency
MoE models achieve a given quality level with less training compute than dense models of equivalent inference cost. During training, each batch of tokens updates only the active experts, but all experts learn over the course of training. The combination of high capacity and selective activation leads to faster convergence.
Specialization
Experts naturally develop distinct capabilities during training. This specialization allows the model to apply domain-specific processing to different types of input rather than using one-size-fits-all computation. The right expert for a math problem is different from the right expert for a poetry request.
Disadvantages and Challenges
MoE is not without tradeoffs.
Memory Requirements
While inference compute scales with active parameters, memory scales with total parameters. A 671 billion parameter MoE model requires enough memory to store all 671 billion parameters, even though only 37 billion are active at any time. This creates a gap between compute efficiency and memory efficiency that requires large GPU clusters or sophisticated offloading strategies.
Training Instability
MoE models are harder to train stably than dense models. The router introduces a discrete selection decision into an otherwise continuous optimization process. Expert collapse, routing oscillation, and training loss spikes are common issues that require careful hyperparameter tuning and engineering.
Communication Overhead
In distributed training and inference, tokens must be routed to the GPUs hosting their selected experts. This requires all-to-all communication between GPUs, which can become a bottleneck at scale. Efficient MoE deployment requires careful placement of experts across hardware and optimized communication patterns.
Fine-Tuning Complexity
Fine-tuning MoE models is more complex than fine-tuning dense models. The router must adapt to new data distributions, and experts may need to re-specialize. Techniques like expert-wise learning rates and router warm-up have been developed to address these challenges, but the process is less straightforward.
MoE Beyond Language Models
The MoE pattern is spreading beyond text.
Vision Models
Vision MoE (V-MoE) applies sparse expert routing to image recognition tasks. Different experts specialize in different visual features or object categories, achieving strong accuracy with reduced computation.
Video Generation
Wan-AI's Wan2.2 uses a Mixture-of-Experts diffusion architecture for video generation, routing specialized experts across different denoising timesteps. This shows MoE adapting to generative tasks beyond language.
Multimodal Models
Modern multimodal models that process text, images, audio, and video simultaneously use MoE to handle the diverse computational requirements of different modalities. Different experts can specialize in processing different types of input.
The Future of MoE
MoE architecture continues to evolve rapidly.
Increasing expert count. Models are trending toward more, smaller experts. DeepSeek's fine-grained approach demonstrates that more experts with finer specialization yields better results. Future models may use hundreds or thousands of experts per layer.
Expert retrieval. Rather than keeping all experts in GPU memory, future systems may store experts on slower storage and retrieve them on demand, enabling models with millions of experts and petabyte-scale knowledge capacity.
Routing beyond top-k. Research into more sophisticated routing mechanisms, including hierarchical routing, multi-stage routing, and content-addressable routing, aims to improve expert utilization and specialization.
MoE at every layer type. Current architectures apply MoE only to FFN layers. Emerging work explores MoE attention heads, MoE embedding layers, and MoE output layers, extending sparsity throughout the entire model.
Conclusion
Mixture of Experts has become the architecture of choice for frontier language models in 2026. By splitting computation among specialized expert networks and routing each token to only the most relevant experts, MoE breaks the link between model capacity and computational cost. A model can store knowledge across hundreds of billions of parameters while paying the inference cost of a fraction of that size.
The innovations driving MoE forward, from DeepSeek's auxiliary-loss-free load balancing to fine-grained expert segmentation and sigmoid routing, are making these models more efficient, more capable, and more practical to deploy. As every major frontier model adopts MoE, understanding this architecture is no longer optional for anyone working seriously with large language models. It is the engineering foundation on which the current generation of AI is built.