# Mixture of experts: routing tokens with top-k gates

Mixture-of-experts models have many feed-forward blocks but use only a few per token. A small router decides which. Type your own text and see who goes where.

<div data-sim="moe"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

Each word on the left is linked to the experts it is routed to; line thickness is the gate weight, dashed lines are routes dropped because the expert was full. Bars inside each expert show its load against capacity. Click a word to see its router probabilities.

## Things to try

- Lower the capacity factor below 1 and watch tokens get dropped.
- Switch the router to untrained and compare the load balance.
- Type a sentence full of numbers or names.

## How it works in real systems

Mixtral 8x7B, DeepSeek-V3 and many frontier models are mixtures of experts. Training adds a load-balancing loss; DeepSeek also uses shared experts and fine-grained routing.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · A top-k MoE layer

```python
import torch, torch.nn as nn, torch.nn.functional as F
class MoE(nn.Module):
    def __init__(self, d, hidden, n_experts=8, k=2):
        super().__init__(); self.k = k
        self.router = nn.Linear(d, n_experts, bias=False)
        self.experts = nn.ModuleList(nn.Sequential(nn.Linear(d, hidden), nn.SiLU(), nn.Linear(hidden, d)) for _ in range(n_experts))
    def forward(self, x):                                   # x: (tokens, d)
        probs = self.router(x).softmax(-1)
        gate, idx = probs.topk(self.k, dim=-1)
        gate = gate / gate.sum(-1, keepdim=True)            # renormalise over the chosen experts
        out = torch.zeros_like(x)
        for e, expert in enumerate(self.experts):
            rows, slot = (idx == e).nonzero(as_tuple=True)
            if len(rows): out[rows] += gate[rows, slot, None] * expert(x[rows])
        # Switch-Transformer load-balancing loss: fraction routed × mean probability, per expert
        frac = F.one_hot(idx[:, 0], len(self.experts)).float().mean(0)
        aux = len(self.experts) * (frac * probs.mean(0)).sum()
        return out, aux
y, aux = MoE(64, 256)(torch.randn(10, 64)); print(y.shape, aux.item())
```

### Step 2 · Add the auxiliary loss when training

```python
# inside the training step, for every MoE layer:
# loss = task_loss + 0.01 * sum(aux_losses)
```

Without it the router collapses onto a few experts, the others stop learning, and capacity is wasted.

### Step 3 · Count total and active parameters

```python
def moe_params(d, hidden, n_experts, k, layers):
    expert = 2 * d * hidden
    return layers * n_experts * expert, layers * k * expert
tot, act = moe_params(d=4096, hidden=14336, n_experts=8, k=2, layers=32)   # Mixtral-8x7B-like feed-forward blocks
print(f"experts: {tot/1e9:.1f}B total, {act/1e9:.1f}B active per token")
```

Mixtral's real expert MLPs are SwiGLU (three matrices), and attention and embeddings are shared, which is how the published totals of about 47B and 13B arise.

<div class="cl-end"></div>

## More simulations

[← Quantisation](/ai-simulations/quantization) · [All AI Simulations](/ai-simulations) · [ReAct tool calling →](/ai-simulations/react-tool-calling)
