Mixture of Experts Architectures
Mixture of Experts, or MoE, is one way frontier models scale parameter count without activating all parameters for every token.
Primary sources:
Dense Versus Sparse
In a dense transformer, every token runs through the same MLP parameters in every layer.
In an MoE transformer, some MLP layers are replaced by multiple expert MLPs. A router chooses which expert or experts process each token.
Example:
token -> router -> top-k experts -> weighted combination -> residual stream
Why MoE Exists
MoE tries to improve the parameter-capacity versus compute tradeoff.
Benefits:
- More total parameters.
- Similar active parameters per token.
- Potential specialization across tokens/domains.
- Better quality at a given inference FLOP budget.
Costs:
- Routing instability.
- Load balancing.
- Expert collapse.
- Communication overhead across devices.
- More complex training and serving.
Top-k Routing
The router produces scores over experts. The model sends each token to the top k experts.
Common choices:
- Top-1 routing: cheaper, simpler, more brittle.
- Top-2 routing: more compute, often better quality/stability.
- Shared experts plus routed experts: combines common capacity with specialized capacity.
Load Balancing
If the router sends too many tokens to one expert, that expert bottlenecks while others sit idle.
Training often adds auxiliary losses or routing constraints to spread tokens across experts.
Key question:
How do we encourage balanced routing without preventing useful specialization?
Capacity Factor
Hardware requires batching tokens per expert. Each expert can process only a capacity-limited number of tokens. Overflow tokens may be dropped, rerouted, or handled by fallback logic.
This is a systems constraint that becomes an architecture constraint.
MoE And Frontier Labs
MoE is attractive because frontier labs are constrained by:
- Training compute.
- Inference cost.
- Memory.
- Latency.
- Model quality.
- Hardware topology.
Sparse models can have enormous total parameter counts while keeping active compute lower than a dense model of the same total size.
Failure Modes
- Router learns degenerate expert usage.
- Experts do not specialize usefully.
- Communication cost removes compute savings.
- Small batches make experts inefficient at inference.
- Fine-tuning destabilizes routing.
- Evaluation gains come from parameter count, not mechanism.
Research Questions
- Do experts specialize semantically, syntactically, by domain, or mostly by optimization accident?
- How should routing work for long context and multi-token reasoning?
- Can MoE improve continual learning by isolating new knowledge?
- How should post-training update sparse experts without damaging routing?
- What is the right comparison: same total params, same active params, same FLOPs, same latency, or same dollars?