Initialisation, batch normalisation and dropout in a deep network
Why do deep networks need careful initialisation? Send a real batch of 128 examples through a stack of random layers and watch the scale of the signal, layer by layer, then fix it.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The top chart is the standard deviation of the activations after every layer, on a log scale; the histogram below is the distribution at the selected layer. Every number is computed on a real batch through real random weight matrices.
Things to try
- Pick 'too small' init: the signal dies within a few layers.
- Pick 'too large': it explodes.
- Keep 'too small' and switch on batch normalisation: the problem disappears.
- Set dropout to 0.5 and toggle training mode: the share of exact zeros jumps.
How it works in real systems
PyTorch layers use Kaiming-uniform initialisation by default; transformers use layer norm or RMSNorm instead of batch norm, and residual connections, which keep the signal alive in models hundreds of layers deep.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Measure the scale of activations through depth
Push a random batch through a deep stack of linear + ReLU layers and print the standard deviation after each layer.
import torch, torch.nn as nn
def probe(init, depth=20, width=256, bn=False):
layers = []
for _ in range(depth):
lin = nn.Linear(width, width, bias=False)
{"small": lambda w: nn.init.normal_(w, std=0.01), "large": lambda w: nn.init.normal_(w, std=1.0),
"xavier": nn.init.xavier_normal_, "he": lambda w: nn.init.kaiming_normal_(w, nonlinearity="relu")}[init](lin.weight)
layers += [lin] + ([nn.BatchNorm1d(width)] if bn else []) + [nn.ReLU()]
net, x = nn.Sequential(*layers), torch.randn(512, width)
stds = []
with torch.no_grad():
for m in net:
x = m(x)
if isinstance(m, nn.ReLU): stds.append(x.std().item())
return [f"{s:.1e}" for s in stds[::5]]
for init in ["small", "xavier", "he", "large"]:
print(f"{init:6s}", probe(init))
print("small + BN", probe("small", bn=True))
Step 2 · Dropout in training and in evaluation
nn.Dropout zeroes a share p of the activations in training mode and scales the rest by 1/(1−p); in evaluation mode it does nothing. Forgetting model.eval() is a classic bug.
d = nn.Dropout(p=0.5); x = torch.ones(10)
d.train(); print(d(x)) # about half zeros, the rest 2.0
d.eval(); print(d(x)) # all ones
Step 3 · Batch norm vs layer norm
Batch norm normalises each feature over the batch, so it behaves differently at batch size 1 and keeps running statistics for inference. Transformers use layer norm (or RMSNorm), which normalises each example over its features and needs no batch statistics.
x = torch.randn(4, 8) * 3 + 1
print(nn.BatchNorm1d(8)(x).mean(0)) # ~0 per feature (column)
print(nn.LayerNorm(8)(x).mean(1)) # ~0 per example (row)
class RMSNorm(nn.Module):
def __init__(self, d, eps=1e-6): super().__init__(); self.w = nn.Parameter(torch.ones(d)); self.eps = eps
def forward(self, x): return self.w * x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)