Mixture-of-Experts

Overview

Mixture-of-Experts, usually shortened to MoE, is a way to build neural networks in which many specialist sub-networks, called experts, sit side by side, and a small router decides which few of them handle each input. Only the chosen experts run. The rest stay idle for that input. This means a model can hold a very large number of parameters while spending the compute of a much smaller model on every token or row.

The payoff is capacity per unit of compute. A dense model uses all its weights for every input. A sparse MoE model uses a fraction of its weights per input, so for the same cost per input it can store more knowledge. The price is complexity: routing must be trained, load must be balanced across experts, memory must hold all the experts, and communication between devices can become a bottleneck. MoE is now a core tool in large language models and is relevant to models that learn from tables. It relates closely to Transformers, Optimization and Meta-learning.

What it is

An MoE layer replaces one big block, most often the feed-forward block inside a transformer layer, with several parallel copies called experts plus a gating network, also called the router.

The parts

• Experts. Independent sub-networks with the same shape. In transformers each is typically a feed-forward network.
• Router (gate). A small network, often one linear layer followed by a softmax, that scores every expert for the current input.
• Combination rule. The outputs of the chosen experts are mixed using the router weights.

Dense versus sparse

Conditional computation

The broader idea behind MoE is conditional computation: run different parts of a network for different inputs. It saves compute and allows specialization.

What MoE is not

• It is not an ensemble of separate trained models whose predictions are averaged at the end. The experts are trained together inside one network, along with the router.
• It is not the same as having many models in production. MoE is a layer type.
• It does not shrink memory needs. It reduces compute per input, not parameter storage.

Where it came from

Early ideas

The idea is old. In 1991 a paper titled "Adaptive Mixtures of Local Experts" proposed training several expert networks and a gating network together, so that each expert learned to handle a part of the input space. A few years later, work on hierarchical mixtures of experts extended this to trees of gates, and linked it to statistical models fitted with the expectation maximization algorithm.

In that early form, every expert ran on every input, and the gate just weighted their outputs. The gains were in modeling, not in saving compute.

Sparse gating arrives

In 2017 the paper "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" showed that you can pick only the top few experts per input and get a model with thousands of experts and over a hundred billion parameters at a practical cost. It introduced noisy top-k gating and auxiliary losses for balancing load.

Scaling with transformers

• In 2020, GShard showed how to scale sparse transformers for translation across many devices, with automatic sharding.
• In 2021, the Switch Transformer simplified routing to a single expert per token, called top-1 routing, and studied how to train such models stably at very large scale.
• Also in 2021, GLaM demonstrated large sparse language models that matched dense ones at lower inference cost.
• Later work introduced expert choice routing, where experts pick tokens instead of tokens picking experts, and stability techniques such as router z-loss.
• By 2023 and 2024, openly released MoE language models showed that sparse designs with a small number of experts and top-2 routing could compete with much larger dense models. Later designs added shared experts that always run, plus many fine-grained routed experts.

Why it is back in focus

Training compute is limited. Inference cost per token matters for serving. MoE offers a different point on the trade-off curve: more parameters for the same compute. As hardware and software for expert parallelism matured, the approach became practical.

How it works

The hospital analogy

A hospital has many specialists: a heart doctor, a bone doctor, a skin doctor, and so on. A triage nurse looks at each patient and sends them to one or two relevant specialists. No patient sees all the doctors. The hospital as a whole holds a huge amount of expertise, yet each patient uses only a small part of it.

The nurse is the router. The doctors are the experts. The hospital's knowledge is the total parameter count. The cost of one visit is the active parameter count.

Two things can go wrong. If the nurse sends everyone to the same popular doctor, there is a queue and the other doctors sit idle. That is load imbalance. If the nurse is bad at triage, patients reach the wrong specialist. That is poor routing.

Step by step for one MoE layer

1. An input vector x arrives, for example one token embedding.
2. The router computes a score for each expert: a linear map of x, giving one number per expert.
3. The scores go through a softmax to give probabilities, or the top scores are selected first.
4. The top-k experts are chosen. Common values are k = 1 or k = 2.
5. The chosen experts each process x independently.
6. The outputs are combined as a weighted sum, using the router probabilities of the chosen experts, renormalized to add up to one.
7. The result passes on to the next layer.

The routing formula in words

Let the router score for expert i be s_i. Keep only the k largest scores. Apply a softmax over those k to get weights g_i. The layer output is the sum over chosen experts of g_i times Expert_i(x).

Why top-k and not all

If all experts ran, you would pay the full dense cost. Choosing k out of N gives a cost of k over N of the full expert pool. With 64 experts and k = 2, each token uses about 3 percent of the expert parameters.

Making routing trainable

Choosing the top k is not smooth. But the weights g_i of the chosen experts are smooth functions of the router scores, so gradients flow to the router through those weights. Experts that are not chosen get no gradient for that input. Some designs add noise to the scores during training to encourage exploration, so that less popular experts also get used.

Load balancing

Without a countermeasure, a few experts get most of the traffic, receive most of the training, become better, and attract even more traffic. The others stay weak. This rich-get-richer loop is called expert collapse.

The usual fix is an auxiliary loss added to the main loss. For each expert i, let f_i be the fraction of tokens routed to it, and P_i the average router probability given to it. The loss is proportional to the number of experts times the sum of f_i times P_i. It is smallest when load is spread evenly. A small coefficient, such as 0.01, is typical.

Other tools:

• Capacity factor. Each expert gets a fixed buffer of slots per batch, about the average load times a factor such as 1.25. Tokens that overflow are dropped or passed along the residual path.
• Router z-loss. Penalizes very large router logits, which improves numerical stability.
• Expert choice routing. Each expert selects its top tokens, which guarantees balance but changes the guarantees for each token.
• Bias-based balancing. Adjust a per-expert bias on the router score over time, without an auxiliary loss, to even out traffic.

Shared and fine-grained experts

Some recent designs split experts into smaller pieces and route among more of them, with a larger k. This allows more combinations of specialists. They also keep one or a few shared experts that every token uses, to hold common knowledge, so that the routed experts can specialize more sharply.

Parallelism in practice

Large MoE models are spread over many devices. The standard approach is expert parallelism:

1. Each device holds some of the experts.
2. After routing, tokens are sent across the network to the devices that hold their chosen experts. This is an all-to-all exchange.
3. Experts process their tokens.
4. The results are sent back in a second all-to-all exchange.

Communication can dominate time. Systems work hard to overlap it with compute, to keep traffic local, and to limit the number of devices a token visits.

Training and fine-tuning notes

• MoE models are more prone to overfitting when fine-tuned on small data sets, because of their size.
• Training can be unstable. Careful initialization, lower precision handling, and z-loss help.
• Upcycling is a common trick: start from a trained dense model, copy its feed-forward block into N experts, add a router, and continue training.
• Some work freezes most experts and tunes the router and a few experts for adaptation.

Inference

• Memory must hold all experts, even though few run.
• Small batches under-use experts, so throughput can be lower than the parameter count suggests.
• Large batches spread tokens across experts and improve use.
• Quantization and offloading of cold experts can help to fit models on limited hardware.

Hard versus soft routing

There are two broad ways to combine experts.

• Soft routing. Every expert runs and the outputs are mixed with smooth weights. This is easy to train but saves no compute.
• Hard (sparse) routing. Only the top-k experts run. This saves compute but makes the choice discrete.

Most large systems use hard routing at the layer level, with soft weights among the chosen few.

Where MoE layers sit

• Usually every second or fourth transformer block uses an MoE feed-forward layer, while the others stay dense.
• Attention layers are normally dense, though some research routes attention heads too.
• Putting MoE in every layer raises memory and communication cost sharply.

Key concepts and terms

A worked example

Counting parameters

Take a transformer feed-forward block with model width 1024 and hidden width 4096. It has two weight matrices, so about 2 times 1024 times 4096, which is roughly 8.4 million parameters.

Now build an MoE layer with 8 such experts and top-2 routing:

So the layer holds 8 times the parameters of the dense block, yet costs only twice as much compute per token. If a dense block with the same total capacity were used, compute would be 8 times higher.

Routing numbers

Say the router gives these scores for a token across four experts: 2.0, 1.0, 0.1, and minus 1.0. With top-2 routing we keep 2.0 and 1.0. Softmax over just those two:

• e to the 2.0 is about 7.389.
• e to the 1.0 is about 2.718.
• The sum is about 10.107.
• Weights are about 0.731 and 0.269.

The layer output is 0.731 times the output of expert 1 plus 0.269 times the output of expert 2. Experts 3 and 4 do no work for this token.

Load balance by hand

Imagine 100 tokens and 4 experts, with top-1 routing. If the counts are 70, 20, 5, 5, then f is 0.7, 0.2, 0.05, 0.05. If the average router probabilities P are similar, say 0.6, 0.25, 0.1, 0.05, the sum of f times P is 0.42 plus 0.05 plus 0.005 plus 0.0025, about 0.4775. Multiply by 4 experts to get 1.91. A perfectly balanced layer has f and P both equal to 0.25, giving a sum of 0.25 and a loss value of 1.0. The auxiliary loss pushes the layer from 1.91 toward 1.0.

A minimal code example

import torch
import torch.nn as nn
import torch.nn.functional as F

class Expert(nn.Module):
    def __init__(self, d_model, d_hidden):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(d_model, d_hidden),
            nn.GELU(),
            nn.Linear(d_hidden, d_model),
        )

    def forward(self, x):
        return self.net(x)

class MoELayer(nn.Module):
    def __init__(self, d_model=256, d_hidden=1024, n_experts=8, k=2):
        super().__init__()
        self.k = k
        self.n_experts = n_experts
        self.router = nn.Linear(d_model, n_experts, bias=False)
        self.experts = nn.ModuleList(
            [Expert(d_model, d_hidden) for _ in range(n_experts)]
        )

    def forward(self, x):                       # x: (tokens, d_model)
        logits = self.router(x)                 # (tokens, n_experts)
        top_vals, top_idx = logits.topk(self.k, dim=-1)
        weights = F.softmax(top_vals, dim=-1)   # (tokens, k)

        out = torch.zeros_like(x)
        for slot in range(self.k):
            for e in range(self.n_experts):
                mask = top_idx[:, slot] == e
                if mask.any():
                    out[mask] += weights[mask, slot, None] * self.experts[e](x[mask])

        # load balancing loss
        probs = F.softmax(logits, dim=-1)
        frac_tokens = torch.zeros(self.n_experts, device=x.device)
        for e in range(self.n_experts):
            frac_tokens[e] = (top_idx[:, 0] == e).float().mean()
        mean_probs = probs.mean(dim=0)
        aux_loss = self.n_experts * (frac_tokens * mean_probs).sum()
        return out, aux_loss

This loop-based version is for clarity. Production code groups tokens by expert and uses batched matrix multiplies and fast communication.

Using it in training

moe = MoELayer()
x = torch.randn(512, 256)
y, aux = moe(x)
loss = y.pow(2).mean() + 0.01 * aux
loss.backward()

The coefficient 0.01 keeps the balancing term small compared with the main loss.

What to monitor

• The fraction of tokens per expert, each step. Look for experts near zero.
• The router entropy. A very low entropy means the router is over-confident.
• The share of dropped tokens.
• The gap between training and validation loss, since large MoE models can overfit.

Strengths and limitations

Strengths

• More capacity for the same compute. Total parameters grow without a matching rise in cost per input.
• Faster pre-training for a target quality. Sparse models often reach a given loss with less compute than dense ones.
• Specialization. Different experts can learn different regimes of the data.
• Cheaper inference per token. Active parameters set the compute.
• Flexibility. Experts can be added, and dense models can be upcycled.

Limitations

• Memory. All experts must be stored, so memory scales with total parameters.
• Communication. All-to-all traffic can limit speed on clusters.
• Training instability. Routing adds a discrete choice and a risk of collapse.
• Load imbalance. Needs auxiliary losses and capacity tuning.
• Small-batch inefficiency. Few tokens per expert means poor hardware use at low batch sizes.
• Fine-tuning difficulty. Overfitting on small tasks is more likely.
• Interpretability myths. Experts rarely specialize in clean human concepts. They often split by surface features such as syntax, token type or position.
• Engineering burden. Kernels, sharding and debugging are harder.

How it compares with alternatives

Choosing

• Pick MoE when you are compute bound in training or serving, have enough memory and fast interconnect, and have large data to feed many experts.
• Stay dense when memory is tight, batch sizes are small, or the data set is small.
• Consider adapters when you only need to specialize a large model for a task.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed numeric and categorical types, missing cells, and sometimes no useful column names. Models such as TabPFN and TabICL are pre-trained on large collections of synthetic or real tables and use in-context learning: labeled rows are given as context and predictions come out of a single forward pass with no weight updates. See /model/research, /model/architecture and /model/benchmarks.

Tables are wildly diverse. A model must handle a ten-column medical table, a two-hundred-column sensor table, a table of mostly categories, and a table of mostly noise. That diversity is a natural reason to consider experts. The following points describe how MoE relates to such models in concrete terms. They describe design options, not claims about any specific release.

Diversity of table types as a case for specialization

Different tables call for different mechanisms:

• Smooth numeric relationships versus sharp threshold rules.
• High-cardinality categories versus binary flags.
• Small tables with few rows versus large tables with many.
• Strong feature interactions versus nearly additive effects.

A single dense network must cover all of these with the same weights. A mixture of experts can let different experts take on different regimes, and the router can learn to send each cell, column or row to the expert that suits it.

What a token is in a table

In a language model a token is a word piece. In a tabular model the unit could be a cell, a column embedding, a row embedding, or a pooled summary. Routing can happen at any of these levels:

• Cell level. Each cell embedding goes to experts chosen by its value and column role.
• Column level. Whole columns are routed, so a numeric column and a categorical column use different experts.
• Row level. Each row goes to experts that match its overall profile.

This matches the row and column structure of tables, and fits with attention along rows and along columns as in axial attention.

Mixed types

Numeric and categorical cells need different processing. A router that sees a type signal can direct numbers to experts that learned scale-sensitive transformations, and categories to experts that learned identity-based lookups and rare-level handling. This is a sparse alternative to building separate branches by hand.

Missing values

Missingness patterns are informative. An expert can specialize in rows with many gaps, learning to rely on available columns, while another handles complete rows. In imputation networks and masked cell modeling, the masking rate varies, and a router can adapt capacity to the amount of missing information.

No column names

When names are missing, a tabular model identifies a column by its values and its relation to others. A router can use those learned column signatures to pick experts, without needing any text. When names are present, they can be embedded and fed to the router as extra context, so that the same mechanism adds semantic knowledge.

Pre-training on synthetic tables

The synthetic generator can produce families of tables: linear, tree-like, clustered, periodic, noisy. Experts can align with these families, and diversity in the prior encourages specialization. Monitoring which experts activate on which synthetic family is a useful diagnostic, and a way to detect that the prior covers too little.

In-context learning

In-context learners compute predictions for query rows by attending to context rows. Adding MoE layers on top of such attention increases the capacity to represent many different kinds of dependencies without increasing the cost of each forward pass much. Since tabular forward passes may need to run many times, for example over feature permutations in an ensemble, low compute per pass matters.

Continual learning

Sparse experts are attractive for continual learning. New data can be handled by new or lightly tuned experts while old experts stay fixed, which limits forgetting. A router can be updated to send new kinds of tables to the new experts. This relates to transfer learning and meta-learning, where reuse of earlier knowledge is central. Care is needed, since the router itself can drift and re-route old inputs.

Imputation

Filling a missing cell may need different strategies depending on the column type, the correlations available, and the missingness pattern. Experts can specialize in these strategies. Routing by the target cell's column signature is a direct way to realize this.

Anomaly detection

Experts are trained on typical data. A row that the router finds hard to place, because its scores are flat or because it sends the row to rarely used experts, can be a sign of an anomaly. Router entropy and expert usage statistics can be extra inputs for a detector such as isolation forest. This is a heuristic that needs validation, not a guarantee.

Sector and semantic classification

Assigning rows or columns to sectors or semantic classes (semantic sector heads) benefits from specialists. One expert can learn the patterns of financial columns, another those of logistics, another those of health. The router then acts as a soft pre-classifier. Because the active experts are observable, they can also serve as a clue when reviewing a prediction.

Explaining sparse models

Feature attribution with SHAP still applies to the whole model as a function. Combining it with logs of which experts fired gives two complementary views: which inputs mattered and which internal specialists processed them.

Tuning

Number of experts, k, capacity and balancing weights interact. Because each run is costly, systematic search with Bayesian optimization on small proxy models is a sensible way to choose them before a large run.

Practical guidance and best practices

Choosing the number of experts and k

• Start with 8 to 16 experts and k = 2 for a first study. Larger counts help at large scale with fine-grained experts.
• Top-1 routing is cheaper and simpler, top-2 is often more stable and better in quality.
• Keep expert size and count balanced against memory and communication limits.

Balancing and stability

• Use a load balancing loss with a small weight and watch usage histograms.
• Add router z-loss if logits grow large.
• Use a capacity factor of about 1.0 to 1.5 in training and look at the drop rate.
• Initialize the router with small weights so that routing starts near uniform.
• Consider adding noise to the router during training.
• Keep the router in higher numeric precision than the experts.

Starting from dense

• Upcycle a trained dense model when you have one. It is cheaper than training an MoE from scratch.
• Add small random noise to expert copies so they can diverge.
• Train long enough for experts to specialize.

Systems

• Use expert parallelism with efficient all-to-all kernels.
• Group tokens per expert and use batched matrix multiplies.
• Overlap communication and compute.
• Prefer larger batches at inference to keep experts busy.
• Consider quantization of experts for memory savings.

Fine-tuning and adaptation

• Use lower learning rates and stronger regularization for small tasks.
• Consider tuning only the attention layers, router, or a subset of experts.
• Evaluate with held-out tasks, since overfitting is a real risk.

Evaluating

• Compare against a dense model with the same active parameters and the same total parameters. Both comparisons matter.
• Report compute per token, memory, and throughput, not only quality.
• Inspect expert usage on different data slices.

Common mistakes

• Comparing only by total parameters. An MoE with 8 times the parameters is not an 8 times bigger dense model in compute or quality.
• Ignoring memory. Idle experts still occupy memory.
• No balancing. Leads to expert collapse and wasted capacity.
• Too strong balancing. Forces uniform routing that blocks specialization.
• Tiny batches at inference. Leaves most experts idle and wastes hardware.
• Dropping too many tokens. Hurts quality silently. Track the drop rate.
• Expecting human-readable experts. Specialization is statistical, not semantic.
• Fine-tuning like a dense model. Overfits quickly.
• Using MoE on small data and small models. The overhead outweighs the benefit.
• Forgetting to log routing statistics. Without them, failures are invisible.
• Neglecting the network. Slow interconnect turns sparse compute savings into communication delays.

Frequently asked questions

What is a Mixture-of-Experts model in simple terms?

It is a neural network with many specialist sub-networks and a router that picks a few of them for each input. Only the chosen experts do any work for that input, so the model can be very large in total while staying cheap to run per input. The router and the experts are trained together.

Does Mixture-of-Experts reduce the memory needed?

No. It reduces the compute used per input, not the number of stored parameters. All experts must be available in memory, or loaded on demand, because any input might need any expert. This is why MoE models often need many devices or careful offloading.

What does the router actually learn?

It learns a scoring function that maps an input representation to a preference over experts. Over training it settles into patterns that help the loss, for example sending certain token types, positions, or data regimes to certain experts. These patterns are often statistical rather than neat human categories.

Why is load balancing needed?

Without it, the router tends to favor a few experts that happen to learn faster, which then get even more traffic. The rest stay undertrained and idle, wasting capacity and slowing systems that wait for the busiest expert. An auxiliary loss, capacity limits and other tricks keep traffic spread out.

How many experts are active per input?

Most designs activate one or two experts per token, although newer fine-grained designs activate several small experts plus shared ones. The choice sets the compute per token. A larger k raises quality and cost together.

Is a Mixture-of-Experts model the same as an ensemble?

No. An ensemble combines the outputs of separately trained models, and every member runs on every input. An MoE layer lives inside one model, its experts are trained jointly with a router, and only a few experts run per input. The goal is efficiency and specialization within a single network.

Can Mixture-of-Experts be used for tabular data?

Yes in principle. Tables vary a great deal in type, size and structure, so specialists can help, and routing can operate on cells, columns or rows. It is a design option for tabular foundation models rather than a requirement, and its benefit depends on scale and on how diverse the training tables are.

What is upcycling?

Upcycling means starting from a trained dense model and turning its feed-forward blocks into several experts by copying them, adding a router, and continuing training. It saves compute compared with training a sparse model from scratch, because the experts begin with useful weights and then diverge.

Why can MoE models be harder to fine-tune?

They have many parameters relative to the active compute, and small fine-tuning sets can lead to overfitting. The router can also shift in unhelpful ways. Lower learning rates, stronger regularization, and tuning only some parts, such as attention layers or the router, usually help.

How do I know if my experts are specializing?

Log the fraction of inputs per expert and break it down by data slices, such as column type, table family or position. If experts show different usage profiles across slices and the balance loss stays low, specialization is happening. Flat usage across all slices suggests the experts are redundant, and extreme usage suggests collapse.

Glossary

• Active parameters: the parameters used to process a single input.
• All-to-all communication: an exchange in which every device sends data to every other device, used to move tokens to experts.
• Capacity factor: a multiplier that sets the number of tokens each expert can handle per batch.
• Conditional computation: running different parts of a network for different inputs.
• Expert: one of several parallel sub-networks in an MoE layer.
• Expert collapse: a failure in which a few experts handle almost all inputs.
• Gating network: another name for the router.
• Load balancing loss: an auxiliary loss that spreads traffic evenly across experts.
• Router z-loss: a penalty on large router logits that improves stability.
• Sparse activation: using only a small part of a model for each input.
• Top-k routing: sending each input to the k highest scoring experts.
• Upcycling: converting a dense model into an MoE by copying its blocks.

Summary

Mixture-of-Experts splits a layer into many experts and uses a trained router to activate only a few per input, so a model can hold far more parameters than it uses on any one input. It needs load balancing, capacity control, enough memory and fast communication, but it can reach a given quality with less training compute and serve tokens more cheaply than a dense model of equal total size. For tabular foundation models, which must cover many table types, mixed column types, missing values, nameless columns and several tasks such as imputation, anomaly detection and sector classification, experts offer a sparse way to specialize while keeping each forward pass cheap. Monitor routing, compare with dense baselines fairly, and tune with care using tools like Bayesian optimization and SHAP. See /model/architecture for how such components fit together in table models.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: