# What is Normalization layers? Definition and meaning

Layers that rescale activations to a standard range, stabilising and accelerating training.

## Definition

**Normalization layers**: Layers that rescale activations to a standard range, stabilising and accelerating training.

In short: Keeping activations in a healthy range.

## When it is used

CNNs (batch norm) and transformers (layer norm, RMSNorm).

## Where it fits

Normalization layers belongs to **Building blocks** in the [Deep learning](/topics/deep-learning) topic. These are the other terms in the same group:

- **[Attention mechanism](/glossary/attention-mechanism)**: A mechanism that computes weighted combinations of inputs, with weights based on relevance between queries and keys.
- **[Backpropagation](/glossary/backpropagation)**: An algorithm that uses the chain rule to compute gradients of the loss with respect to every weight.
- **[Dropout](/glossary/dropout)**: A regularisation technique that zeroes random activations during training to prevent co-adaptation.
- **[Optimizers](/glossary/optimizers)**: Algorithms that update model parameters from gradients, such as SGD, momentum, Adam and AdamW.
- **[Word embeddings](/glossary/word-embeddings)**: Learned vector representations of words where similar words have similar vectors, as in Word2Vec and GloVe.
- **[U-Net](/glossary/u-net)**: A convolutional encoder-decoder with skip connections that preserves fine detail for pixel-level predictions.
- **[Knowledge distillation](/glossary/knowledge-distillation)**: Training a compact model to match the outputs of a larger model, transferring much of its capability.
- **[Pruning](/glossary/pruning)**: Removing weights, channels or layers that contribute little, to shrink and speed up a network.

## Learn more

The full guide to **[Normalization layers](/tools/normalization-layers)** explains how it works, when to use it and how it compares with the alternatives, with examples. The [glossary](/glossary) lists every term on the site from A to Z.

Previous term: [Neural ODE](/glossary/neural-ode) · Next term: [Normalizing flows](/glossary/normalizing-flows)
