Normalization layers
Normalization layers rescale activations inside neural networks to stabilise and speed up training. Learn batch norm, layer norm and RMSNorm and where each is used.
Overview
Normalisation layers are small components placed inside a neural network to keep the numbers flowing through it in a sensible range. As data passes through many layers, values can drift: some grow very large, others shrink towards zero, and their spread changes as the weights change during training. A normalisation layer measures the values it receives, rescales them to a standard centre and spread, and then applies a learned scale and shift so the network can still choose the range it prefers.
This simple step has a large effect. Networks with normalisation usually train faster, tolerate higher learning rates and are less sensitive to how their weights were initialised. Very deep networks, including ResNet models and large transformers, are hard to train well without it.
An everyday comparison is the volume levelling on a music streaming service. Songs are recorded at very different loudness. Without levelling you keep reaching for the volume control. With it, every track arrives at a similar loudness, and you can still turn the overall volume up or down. Normalisation layers do this for activations, and the learned scale and shift play the part of the volume knob.
What it is
Normalisation methods differ mainly in which values they group together when computing the average and spread.
| Method | Statistics computed over | Learned parameters | Typical use |
|---|---|---|---|
| Batch normalisation | Each channel, across the batch and spatial positions | Scale and shift per channel | CNNs for images |
| Layer normalisation | All features of a single example | Scale and shift per feature | Transformers, recurrent networks |
| RMSNorm | Root mean square of a single example's features, no centring | Scale per feature | Many recent large language models |
| Group normalisation | Groups of channels within one example | Scale and shift per channel | Vision with small batches, diffusion U-Nets |
| Instance normalisation | Each channel of each example separately | Scale and shift per channel | Style transfer, some image generation |
All of these share one pattern: subtract a mean (except RMSNorm), divide by a measure of spread plus a tiny constant to avoid division by zero, then multiply by a learned scale (often called gamma) and add a learned shift (beta).
The key practical difference is whether the statistics depend on other examples in the batch. Batch normalisation does, which makes it behave differently in training and inference. The others work on each example on its own, so they behave the same at any batch size.
Where it came from
Sergey Ioffe and Christian Szegedy of Google introduced batch normalisation in 2015. They argued it reduced "internal covariate shift", the changing distribution of each layer's inputs during training. It allowed much higher learning rates and quickly became standard in image models. Later research, notably a 2018 study by Shibani Santurkar and colleagues, suggested that its main benefit is making the optimisation landscape smoother, rather than reducing covariate shift; the precise explanation is still discussed.
Batch normalisation was awkward for recurrent networks and small batches, so alternatives followed. Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey Hinton proposed layer normalisation in 2016. Instance normalisation appeared the same year for style transfer. Yuxin Wu and Kaiming He introduced group normalisation in 2018 for vision tasks with small batches. Biao Zhang and Rico Sennrich proposed RMSNorm in 2019, showing that dropping the mean subtraction kept most of the benefit at lower cost.
The 2017 transformer used layer normalisation after each sub-layer, known as post-norm. Many later models moved it before each sub-layer, known as pre-norm, which tends to train more stably in deep stacks. Several widely used open language model families use RMSNorm in a pre-norm arrangement.
How it works
Step 1: choose the group
The layer decides which values belong together. For layer normalisation on a transformer token with 4,096 features, the group is those 4,096 numbers. For batch normalisation on an image feature map, the group is every value in one channel across the whole batch.
Step 2: compute statistics
The layer computes the mean of the group and its variance. RMSNorm instead computes the root mean square, the square root of the average of squared values, and skips the mean.
Step 3: normalise
Each value has the mean subtracted and is divided by the square root of the variance plus a small epsilon, such as 0.00001. After this step the group has a mean near zero and a spread near one.
Step 4: scale and shift
Each feature or channel is multiplied by its learned gamma and has its learned beta added. Because these are trained, the network can undo the normalisation if that helps, or pick any other range.
Batch normalisation at inference
During training, batch norm uses the current batch's statistics and also keeps a running average of them. At inference it switches to those running averages, so a single example's output does not depend on what else is in the batch. Frameworks control this switch with training and evaluation modes.
Where it sits
In CNNs, batch norm usually comes after a convolution and before the activation. In pre-norm transformers, a normalisation layer comes before the attention block and before the feed-forward block, with residual connections around both.
Key concepts and terms
| Term | Meaning |
|---|---|
| Activation | A value produced inside the network by a layer |
| Mean and variance | The centre and spread of a group of values |
| Epsilon | A tiny constant that prevents division by zero |
| Gamma | The learned scale applied after normalising |
| Beta | The learned shift applied after normalising |
| Running statistics | Averages of batch statistics kept for inference |
| Internal covariate shift | The changing distribution of a layer's inputs during training |
| Pre-norm | Normalising before each transformer sub-layer |
| Post-norm | Normalising after each sub-layer and residual addition |
| RMS | Root mean square, the square root of the mean of squares |
| Training and evaluation mode | Framework switch that changes batch norm and dropout behaviour |
A worked example
This small example uses easy numbers. One token in a transformer has just four features: 2, 4, 6 and 8.
Layer normalisation:
- Mean = (2 + 4 + 6 + 8) / 4 = 5.
- Differences from the mean: −3, −1, 1, 3. Squares: 9, 1, 1, 9. Variance = 20 / 4 = 5.
- Standard deviation ≈ 2.236 (ignoring epsilon).
- Normalised values ≈ −1.34, −0.45, 0.45, 1.34.
- With gamma = 1 and beta = 0 these are the outputs. If training had set gamma to 2 for the first feature, its output would become about −2.68.
RMSNorm on the same token:
- Mean of squares = (4 + 16 + 36 + 64) / 4 = 30.
- RMS = √30 ≈ 5.477.
- Outputs ≈ 0.37, 0.73, 1.10, 1.46.
The RMSNorm outputs keep their positive offset because the mean was not removed; they are simply rescaled to a standard size. That saves one reduction per token, which adds up across billions of tokens.
Strengths and limitations
Strengths:
- Faster, more stable training with higher learning rates.
- Less sensitivity to weight initialisation.
- Makes very deep networks practical when combined with residual connections.
- Batch norm can add a mild regularising effect from batch noise.
- Layer norm and RMSNorm work the same for any batch size.
Limitations:
- Batch norm depends on batch size and behaves differently in training and inference.
- Batch norm needs care in distributed training, where statistics may need syncing across devices.
- Adds some compute and memory, and an extra source of numerical issues in low precision.
- The theory of why normalisation helps is still not fully settled.
- Placement choices, such as pre-norm or post-norm, affect stability and must be chosen carefully.
How it compares with alternatives
| Approach | How it stabilises training | Notes |
|---|---|---|
| Batch normalisation | Normalises across the batch | Strong for CNNs with large batches |
| Layer normalisation | Normalises each example's features | Default in many transformers |
| RMSNorm | Rescales by root mean square only | Cheaper; common in large language models |
| Careful initialisation alone | Sets weights so signals keep their scale | Works for some architectures without normalisation |
| Weight normalisation | Reparameterises weights rather than activations | Independent of batch; less common today |
Practical guidance and best practices
- Use batch norm for CNNs trained with reasonably large batches.
- Use layer norm or RMSNorm for transformers and sequence models.
- Prefer group norm when vision batches are very small, such as high-resolution segmentation.
- Switch models to evaluation mode before validation and inference so batch norm uses running statistics.
- Use synchronised batch norm in multi-GPU training when per-device batches are small.
- Keep normalisation computations in a stable precision, or use fused kernels that do.
- Follow the placement used by a proven architecture rather than inventing a new one.
- When fine-tuning with tiny batches, consider freezing batch norm statistics.
Common mistakes
- Evaluating with the model still in training mode, so batch norm uses the test batch's statistics.
- Using batch norm with a batch size of one or two, where statistics are meaningless.
- Applying weight decay to normalisation scale and shift parameters without thinking; many recipes exclude them.
- Changing pre-norm to post-norm in a deep model and then wondering why training diverges.
- Forgetting that running statistics must be updated if the data distribution changes during fine-tuning.
Frequently asked questions
What is the difference between batch norm and layer norm?
Batch norm computes statistics for each channel across all examples in a batch. Layer norm computes statistics across all features of one example. Batch norm therefore depends on the batch; layer norm does not.
Why do transformers use layer norm instead of batch norm?
Sequences vary in length, batches are often small per device, and generation processes one token at a time. Layer norm and RMSNorm handle all of these naturally because they work on each token independently.
What does RMSNorm leave out?
It skips subtracting the mean and the learned shift, and only rescales by the root mean square with a learned scale. This is cheaper and, in practice, works about as well for many language models.
Does normalisation replace good initialisation?
It reduces sensitivity to initialisation but does not remove the need for sensible choices. Most successful recipes use both.
Why does my model behave differently at inference?
If it contains batch norm, training uses batch statistics while inference uses running averages. Forgetting to switch to evaluation mode, or having running statistics that do not match the deployment data, both cause differences. Dropout also changes behaviour between modes.
Is batch normalisation still used?
Yes, it remains common in convolutional image models. Transformer-based models usually use layer normalisation or RMSNorm instead.
Glossary
- Channel: one feature map in a convolutional layer.
- Feature: one number in an example's representation vector.
- Fused kernel: a single GPU operation that performs several steps at once.
- Residual connection: adding a block's input to its output.
- Standard deviation: the square root of the variance.
- Synchronised batch norm: batch norm whose statistics are computed across several devices.
- Variance: the average squared distance of values from their mean.
Summary
Normalisation layers rescale activations to a standard centre and spread, then apply learned scale and shift, which makes deep networks train faster and more reliably. Batch normalisation from 2015 transformed image models, while layer normalisation and the lighter RMSNorm became the standard in transformers because they work on each example independently. Choosing the right type and placement, and remembering the training and evaluation difference for batch norm, are the main practical concerns.
Related
It belongs to Building blocks in my stack, under Deep learning. Others in the same group: