Özer Cem KELAHMETKELAHMET
AI Simulations · Language models

Mixture of experts: routing tokens with top-k gates

Mixture-of-experts models have many feed-forward blocks but use only a few per token. A small router decides which. Type your own text and see who goes where.

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

Each word on the left is linked to the experts it is routed to; line thickness is the gate weight, dashed lines are routes dropped because the expert was full. Bars inside each expert show its load against capacity. Click a word to see its router probabilities.

Things to try

  • Lower the capacity factor below 1 and watch tokens get dropped.
  • Switch the router to untrained and compare the load balance.
  • Type a sentence full of numbers or names.

How it works in real systems

Mixtral 8x7B, DeepSeek-V3 and many frontier models are mixtures of experts. Training adds a load-balancing loss; DeepSeek also uses shared experts and fine-grained routing.

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · A top-k MoE layer

import torch, torch.nn as nn, torch.nn.functional as F
class MoE(nn.Module):
    def __init__(self, d, hidden, n_experts=8, k=2):
        super().__init__(); self.k = k
        self.router = nn.Linear(d, n_experts, bias=False)
        self.experts = nn.ModuleList(nn.Sequential(nn.Linear(d, hidden), nn.SiLU(), nn.Linear(hidden, d)) for _ in range(n_experts))
    def forward(self, x):                                   # x: (tokens, d)
        probs = self.router(x).softmax(-1)
        gate, idx = probs.topk(self.k, dim=-1)
        gate = gate / gate.sum(-1, keepdim=True)            # renormalise over the chosen experts
        out = torch.zeros_like(x)
        for e, expert in enumerate(self.experts):
            rows, slot = (idx == e).nonzero(as_tuple=True)
            if len(rows): out[rows] += gate[rows, slot, None] * expert(x[rows])
        # Switch-Transformer load-balancing loss: fraction routed × mean probability, per expert
        frac = F.one_hot(idx[:, 0], len(self.experts)).float().mean(0)
        aux = len(self.experts) * (frac * probs.mean(0)).sum()
        return out, aux
y, aux = MoE(64, 256)(torch.randn(10, 64)); print(y.shape, aux.item())

Step 2 · Add the auxiliary loss when training

# inside the training step, for every MoE layer:
# loss = task_loss + 0.01 * sum(aux_losses)

Without it the router collapses onto a few experts, the others stop learning, and capacity is wasted.

Step 3 · Count total and active parameters

def moe_params(d, hidden, n_experts, k, layers):
    expert = 2 * d * hidden
    return layers * n_experts * expert, layers * k * expert
tot, act = moe_params(d=4096, hidden=14336, n_experts=8, k=2, layers=32)   # Mixtral-8x7B-like feed-forward blocks
print(f"experts: {tot/1e9:.1f}B total, {act/1e9:.1f}B active per token")

Mixtral's real expert MLPs are SwiGLU (three matrices), and attention and embeddings are shared, which is how the published totals of about 47B and 13B arise.

More simulations

← Quantisation · All AI Simulations · ReAct tool calling →

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open