Mixture of experts: routing tokens with top-k gates
Mixture-of-experts models have many feed-forward blocks but use only a few per token. A small router decides which. Type your own text and see who goes where.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
Each word on the left is linked to the experts it is routed to; line thickness is the gate weight, dashed lines are routes dropped because the expert was full. Bars inside each expert show its load against capacity. Click a word to see its router probabilities.
Things to try
- Lower the capacity factor below 1 and watch tokens get dropped.
- Switch the router to untrained and compare the load balance.
- Type a sentence full of numbers or names.
How it works in real systems
Mixtral 8x7B, DeepSeek-V3 and many frontier models are mixtures of experts. Training adds a load-balancing loss; DeepSeek also uses shared experts and fine-grained routing.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · A top-k MoE layer
import torch, torch.nn as nn, torch.nn.functional as F
class MoE(nn.Module):
def __init__(self, d, hidden, n_experts=8, k=2):
super().__init__(); self.k = k
self.router = nn.Linear(d, n_experts, bias=False)
self.experts = nn.ModuleList(nn.Sequential(nn.Linear(d, hidden), nn.SiLU(), nn.Linear(hidden, d)) for _ in range(n_experts))
def forward(self, x): # x: (tokens, d)
probs = self.router(x).softmax(-1)
gate, idx = probs.topk(self.k, dim=-1)
gate = gate / gate.sum(-1, keepdim=True) # renormalise over the chosen experts
out = torch.zeros_like(x)
for e, expert in enumerate(self.experts):
rows, slot = (idx == e).nonzero(as_tuple=True)
if len(rows): out[rows] += gate[rows, slot, None] * expert(x[rows])
# Switch-Transformer load-balancing loss: fraction routed × mean probability, per expert
frac = F.one_hot(idx[:, 0], len(self.experts)).float().mean(0)
aux = len(self.experts) * (frac * probs.mean(0)).sum()
return out, aux
y, aux = MoE(64, 256)(torch.randn(10, 64)); print(y.shape, aux.item())
Step 2 · Add the auxiliary loss when training
# inside the training step, for every MoE layer:
# loss = task_loss + 0.01 * sum(aux_losses)
Without it the router collapses onto a few experts, the others stop learning, and capacity is wasted.
Step 3 · Count total and active parameters
def moe_params(d, hidden, n_experts, k, layers):
expert = 2 * d * hidden
return layers * n_experts * expert, layers * k * expert
tot, act = moe_params(d=4096, hidden=14336, n_experts=8, k=2, layers=32) # Mixtral-8x7B-like feed-forward blocks
print(f"experts: {tot/1e9:.1f}B total, {act/1e9:.1f}B active per token")
Mixtral's real expert MLPs are SwiGLU (three matrices), and attention and embeddings are shared, which is how the published totals of about 47B and 13B arise.