# Initialisation, batch normalisation and dropout in a deep network

Why do deep networks need careful initialisation? Send a real batch of 128 examples through a stack of random layers and watch the scale of the signal, layer by layer, then fix it.

<div data-sim="normdrop"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

The top chart is the standard deviation of the activations after every layer, on a log scale; the histogram below is the distribution at the selected layer. Every number is computed on a real batch through real random weight matrices.

## Things to try

- Pick 'too small' init: the signal dies within a few layers.
- Pick 'too large': it explodes.
- Keep 'too small' and switch on batch normalisation: the problem disappears.
- Set dropout to 0.5 and toggle training mode: the share of exact zeros jumps.

## How it works in real systems

PyTorch layers use Kaiming-uniform initialisation by default; transformers use layer norm or RMSNorm instead of batch norm, and residual connections, which keep the signal alive in models hundreds of layers deep.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Measure the scale of activations through depth

Push a random batch through a deep stack of linear + ReLU layers and print the standard deviation after each layer.

```python
import torch, torch.nn as nn
def probe(init, depth=20, width=256, bn=False):
    layers = []
    for _ in range(depth):
        lin = nn.Linear(width, width, bias=False)
        {"small": lambda w: nn.init.normal_(w, std=0.01), "large": lambda w: nn.init.normal_(w, std=1.0),
         "xavier": nn.init.xavier_normal_, "he": lambda w: nn.init.kaiming_normal_(w, nonlinearity="relu")}[init](lin.weight)
        layers += [lin] + ([nn.BatchNorm1d(width)] if bn else []) + [nn.ReLU()]
    net, x = nn.Sequential(*layers), torch.randn(512, width)
    stds = []
    with torch.no_grad():
        for m in net:
            x = m(x)
            if isinstance(m, nn.ReLU): stds.append(x.std().item())
    return [f"{s:.1e}" for s in stds[::5]]
for init in ["small", "xavier", "he", "large"]:
    print(f"{init:6s}", probe(init))
print("small + BN", probe("small", bn=True))
```

### Step 2 · Dropout in training and in evaluation

`nn.Dropout` zeroes a share p of the activations in training mode and scales the rest by 1/(1−p); in evaluation mode it does nothing. Forgetting `model.eval()` is a classic bug.

```python
d = nn.Dropout(p=0.5); x = torch.ones(10)
d.train(); print(d(x))     # about half zeros, the rest 2.0
d.eval();  print(d(x))     # all ones
```

### Step 3 · Batch norm vs layer norm

Batch norm normalises each feature over the batch, so it behaves differently at batch size 1 and keeps running statistics for inference. Transformers use layer norm (or RMSNorm), which normalises each example over its features and needs no batch statistics.

```python
x = torch.randn(4, 8) * 3 + 1
print(nn.BatchNorm1d(8)(x).mean(0))    # ~0 per feature (column)
print(nn.LayerNorm(8)(x).mean(1))      # ~0 per example (row)
class RMSNorm(nn.Module):
    def __init__(self, d, eps=1e-6): super().__init__(); self.w = nn.Parameter(torch.ones(d)); self.eps = eps
    def forward(self, x): return self.w * x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
```

<div class="cl-end"></div>

## More simulations

[← PCA](/ai-simulations/pca) · [All AI Simulations](/ai-simulations) · [LR schedules →](/ai-simulations/learning-rate-schedules)
