# Dropout

Dropout randomly switches off neurons during training so a network cannot rely on any single one, reducing overfitting. Learn how it works and when to use it.

## Overview

Dropout is a regularisation technique for neural networks. During training, each time a batch passes through the network, it randomly sets a fraction of the activations in a layer to zero. A different random set is dropped on every step. At inference, nothing is dropped and the full network is used.

The purpose is to reduce overfitting, where a model memorises its training data instead of learning patterns that carry over to new data. Because any unit might vanish on a given step, the network cannot depend on a few specific units working together in fragile ways. It is pushed to spread useful information across many units, which tends to generalise better.

An everyday comparison is a sports team that trains with random players sitting out each session. No single star can carry every drill, so everyone learns to contribute and the team copes when someone is missing on match day. Dropout trains a network in the same way, and on match day, at inference, the whole team plays.

## What it is

Dropout is simple to describe, but a few related ideas matter in practice.

| Part | Meaning |
|---|---|
| Dropout rate p | The probability that each activation is set to zero during training |
| Keep probability | 1 − p, the chance an activation survives |
| Mask | The random pattern of zeros and ones applied on a given step |
| Inverted scaling | Multiplying surviving activations by 1 / (1 − p) during training |
| Training mode | Dropout is active |
| Evaluation mode | Dropout is turned off and the layer passes values unchanged |
| Variants | Spatial dropout, DropConnect, attention dropout, stochastic depth |
| Monte Carlo dropout | Keeping dropout on at inference to estimate uncertainty |

In code, dropout is usually a layer, such as `nn.Dropout(p=0.1)` in [PyTorch](/tools/pytorch) or `layers.Dropout(0.1)` in [Keras](/tools/keras). It has no learned parameters; it only needs the rate.

## Where it came from

Geoffrey Hinton and colleagues at the University of Toronto introduced dropout in a 2012 preprint titled "Improving neural networks by preventing co-adaptation of feature detectors". The same year, dropout was one of the techniques used in the AlexNet image classifier, which won the ImageNet competition by a wide margin and helped launch the deep learning boom.

The full study appeared in 2014 in the Journal of Machine Learning Research: "Dropout: A Simple Way to Prevent Neural Networks from Overfitting" by Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever and Ruslan Salakhutdinov. It showed gains across vision, speech, text and biology tasks, and explained dropout as a cheap way of training and averaging a huge number of thinned networks that share weights.

Variants followed. DropConnect (2013) dropped individual weights instead of activations. Spatial dropout dropped whole feature maps in [convolutional networks](/tools/cnn). In 2016, Yarin Gal and Zoubin Ghahramani showed that dropout can be read as approximate Bayesian inference, which made Monte Carlo dropout a popular uncertainty method. The same year, stochastic depth applied the idea to whole residual blocks. Transformers use dropout on attention weights and on residual paths, although many very large language models now train with little or no dropout because they see so much data that overfitting is less of a concern.

## How it works

### Step 1: draw a mask

For a layer's output, each value independently gets a random number. With probability p it is marked to drop; otherwise it is kept. With p = 0.2, roughly one in five values is dropped on each step.

### Step 2: zero and rescale

Dropped values become zero. Kept values are multiplied by 1 / (1 − p). This is called inverted dropout. It keeps the expected size of the layer's output the same as without dropout, so the next layer sees consistent magnitudes.

### Step 3: backpropagate through the mask

During [backpropagation](/tools/backpropagation), gradients flow only through the kept units, scaled the same way. Weights feeding dropped units get no update from this step. Each step therefore trains a different thinned sub-network.

### Step 4: switch off at inference

When the model is put into evaluation mode, dropout does nothing: every unit is used and no scaling is applied. Thanks to the inverted scaling during training, no adjustment is needed here.

### Why it helps

Two views explain the effect. First, it prevents co-adaptation: units cannot rely on particular partners always being present, so each must be useful on its own. Second, it acts like an ensemble: training many thinned networks that share weights, then using the full network at test time, approximates averaging their predictions.

### Monte Carlo dropout

If you leave dropout on at inference and run the same input many times, you get a spread of predictions. The spread gives a rough estimate of the model's uncertainty, which is useful in medicine, forecasting and [Bayesian deep learning](/tools/bayesian-deep-learning) more broadly.

## Key concepts and terms

| Term | Meaning |
|---|---|
| Overfitting | Fitting training data so closely that performance on new data suffers |
| Regularisation | Any technique that discourages overfitting |
| Dropout rate | Fraction of activations zeroed during training |
| Co-adaptation | Units relying on specific other units to be useful |
| Inverted dropout | Scaling kept values during training so inference needs no change |
| Thinned network | The sub-network left after a dropout mask is applied |
| Spatial dropout | Dropping whole channels of a feature map |
| DropConnect | Dropping individual weights rather than activations |
| Attention dropout | Dropout applied to attention weights in transformers |
| Stochastic depth | Randomly skipping whole residual blocks during training |
| Monte Carlo dropout | Repeated stochastic inference with dropout on to estimate uncertainty |

## A worked example

This illustrative example follows one dropout layer with p = 0.25 on a tiny vector of four activations: 2.0, 1.0, 3.0, 4.0.

1. **Draw a mask.** Suppose the random draw drops the second value: mask = 1, 0, 1, 1.
2. **Apply the mask.** The result is 2.0, 0, 3.0, 4.0.
3. **Rescale.** Multiply kept values by 1 / (1 − 0.25) ≈ 1.333. Output: 2.67, 0, 4.0, 5.33.
4. **Check the average size.** On average one value in four is dropped and the rest are scaled up by a third, so the expected output matches the original.
5. **Next step, new mask.** On the next batch the draw might drop the fourth value instead, training a different sub-network.
6. **At inference.** The layer simply outputs 2.0, 1.0, 3.0, 4.0.

Now a model-level scenario, also illustrative. A team trains a classifier on 5,000 labelled product reviews. Without dropout, training accuracy reaches 99 per cent while validation accuracy peaks at 84 per cent and then falls, a clear sign of overfitting. Adding dropout of 0.3 after each hidden layer slows the rise in training accuracy, but validation accuracy holds steady for longer. They pick the rate that gives the best validation score after trying 0.1, 0.3 and 0.5.

## Strengths and limitations

**Strengths:**

- Very simple to add, with one hyperparameter.
- No learned parameters and little computational cost.
- Often reduces overfitting on small and medium datasets.
- Works with most architectures.
- Can provide uncertainty estimates through Monte Carlo dropout.

**Limitations:**

- Slows convergence, so training may need more steps.
- Too high a rate causes underfitting.
- Interacts with batch normalisation in ways that can hurt if placed carelessly.
- Less useful when data is plentiful, as in many large language models.
- Monte Carlo dropout uncertainty is approximate and can be poorly calibrated.

## How it compares with alternatives

| Technique | How it fights overfitting | Notes |
|---|---|---|
| Dropout | Random zeroing of activations | Cheap and general |
| Weight decay | Penalises large weights | Built into [optimizers](/tools/optimizers) such as AdamW |
| Data augmentation | Creates varied training examples | Very effective for images and audio |
| Early stopping | Stops training when validation stops improving | Needs a validation set |
| [Normalisation layers](/tools/normalization-layers) | Stabilise training, with mild regularising effect | Primarily for optimisation, not regularisation |

## Practical guidance and best practices

1. Start with a modest rate, around 0.1 for transformers and 0.2 to 0.5 for dense hidden layers, and tune with validation data.
2. Add dropout where overfitting appears, not everywhere by default.
3. Use spatial dropout rather than standard dropout for convolutional feature maps.
4. Always switch to evaluation mode for validation and inference.
5. Combine dropout with weight decay and early stopping rather than relying on it alone.
6. When fine-tuning a pretrained model, keep its original dropout settings unless you have a reason to change them.
7. For uncertainty estimates, run 20 to 100 Monte Carlo passes and check calibration on held-out data.
8. Reduce or remove dropout when you have far more data than parameters can memorise.

## Common mistakes

- Leaving the model in training mode at inference, giving noisy, inconsistent predictions.
- Applying dropout to the output layer of a classifier.
- Setting the rate so high that the model cannot fit even the training data.
- Comparing training loss with and without dropout, when only validation metrics are meaningful.
- Placing dropout directly before batch normalisation without testing, which can shift statistics between training and inference.

## Frequently asked questions

### What dropout rate should I use?

There is no single best value. Rates of 0.1 are common in transformers, and 0.2 to 0.5 in fully connected layers of smaller networks. The right rate depends on model size and data size, so tune it with a validation set.

### Is dropout used at inference?

Normally no. Dropout is switched off at inference and the full network is used. The exception is Monte Carlo dropout, where it is kept on deliberately to estimate uncertainty.

### Why scale the kept activations?

If 20 per cent of values are zeroed, the layer's output shrinks by about 20 per cent on average. Scaling kept values by 1 / (1 − p) restores the expected size, so the model behaves consistently between training and inference.

### Do large language models use dropout?

Some do, at low rates, especially in fine-tuning. Many very large pretraining runs use little or none, because the models see so much data that overfitting is less of a problem than underfitting.

### How is dropout related to ensembles?

Each training step uses a different thinned network, and all of them share weights. Using the full network at test time approximates averaging the predictions of that very large family of networks, which is why dropout behaves somewhat like a cheap ensemble.

## Glossary

- Activation: the output of a unit or layer.
- Ensemble: a group of models whose predictions are combined.
- Generalisation: how well a model performs on data it has not seen.
- Hyperparameter: a setting chosen before training, such as the dropout rate.
- Mask: a pattern of zeros and ones that selects which values to keep.
- Underfitting: when a model is too constrained to fit even its training data.
- Validation set: data held out to measure generalisation during development.

## Summary

Dropout randomly zeroes a fraction of activations during training and scales the rest, forcing a network to spread knowledge across many units instead of relying on a fragile few. Introduced by Hinton and colleagues in 2012 and studied in depth in 2014, it became one of the standard tools against overfitting and later found a second use in uncertainty estimation. It is cheap and general, but needs tuning, must be turned off at inference and matters less when data is abundant.