# Backpropagation

Backpropagation computes how each weight in a neural network affects the error, using the chain rule, so gradient descent can improve the network.

## Overview

Backpropagation, short for "backward propagation of errors", is the method neural networks use to work out how to improve. After a network makes a prediction and we measure how wrong it was, backpropagation calculates, for every single weight, how much a small change in that weight would change the error. That collection of numbers is called the gradient. An [optimizer](/tools/optimizers) then nudges each weight in the direction that reduces the error.

Backpropagation does not decide how big the steps are or which direction to try first; that is the optimizer's job. What it does is make the gradient cheap to compute. A network with a billion weights needs a billion partial derivatives per step. Backpropagation gets all of them in roughly the cost of one extra pass through the network, by reusing intermediate results from the output back towards the input.

An everyday comparison is a team that misses a deadline. The manager looks at the final delay, then asks each stage before it how much it contributed, and each stage asks its own suppliers in turn. Blame flows backwards through the chain, and every person learns their share. Backpropagation passes blame for the error backwards through the layers in just this way, and the chain rule of calculus makes the sharing exact.

## What it is

Backpropagation brings together a few ideas from calculus and computing.

| Element | Role |
|---|---|
| Forward pass | Runs inputs through the network and stores intermediate values |
| Loss function | Measures how far predictions are from targets, as one number |
| Computational graph | The network viewed as a chain of simple operations |
| Local derivatives | How each operation's output changes with its inputs |
| Chain rule | Multiplies local derivatives to get the effect of an early value on the loss |
| Backward pass | Walks the graph from the loss to the inputs, accumulating gradients |
| Gradient | The vector of partial derivatives of the loss with respect to every weight |
| Weight update | The optimizer's step that uses the gradient |

In modern software, backpropagation is a special case of reverse-mode automatic differentiation. Frameworks such as [PyTorch](/tools/pytorch), [TensorFlow](/tools/tensorflow) and [JAX](/tools/jax) record the operations of the forward pass and run the backward pass automatically, so practitioners rarely write derivatives by hand.

## Where it came from

The mathematics behind backpropagation, the chain rule, is centuries old. The efficient algorithm for applying it in reverse through a long computation was described by Seppo Linnainmaa in 1970 as reverse-mode automatic differentiation. Paul Werbos proposed using this kind of method to train neural networks in his 1974 doctoral thesis.

The method became widely known through a 1986 paper in Nature by David Rumelhart, Geoffrey Hinton and Ronald Williams, "Learning representations by back-propagating errors". It showed that networks with hidden layers trained this way could learn useful internal representations, which revived interest in multi-layer networks.

Training deep networks remained difficult for years. In the early 1990s, Sepp Hochreiter analysed the vanishing gradient problem, in which gradients shrink as they pass back through many layers. Later advances made deep backpropagation practical: ReLU activations, careful weight initialisation, [normalisation layers](/tools/normalization-layers), residual connections as in [ResNet](/tools/resnet), and GPUs fast enough to run millions of passes.

## How it works

### Step 1: forward pass

Take one batch of inputs. Each layer computes its output from the previous layer's output and its own weights. For a simple layer this is a matrix multiply plus a bias, followed by an activation function. The framework stores the values it will need later.

### Step 2: compute the loss

At the end, the loss function compares predictions with true targets. For regression this might be mean squared error; for classification, cross-entropy. The result is a single number.

### Step 3: start at the end

The derivative of the loss with respect to itself is 1. From there, the algorithm computes the derivative of the loss with respect to the network's output, using the loss function's own derivative.

### Step 4: move backwards one layer at a time

For each layer, going backwards, two things are computed:

- The gradient with respect to the layer's weights: how the loss changes if those weights change. This is what the optimizer needs.
- The gradient with respect to the layer's input: how the loss changes if the previous layer's output changes. This is passed further back.

Each is the incoming gradient multiplied by the layer's local derivative. This is the chain rule applied step by step.

### Step 5: accumulate and update

When one value feeds several later operations, the gradients from all of them are added together. Once the backward pass reaches the start, every weight has its gradient, and the optimizer updates the weights. The cycle then repeats on the next batch.

### Why it is efficient

Computing each weight's effect separately would need one pass per weight. Going backwards reuses every intermediate gradient, so all the derivatives cost about the same as one forward pass, often quoted as two to three times the forward cost. The price is memory: activations from the forward pass must be kept until the backward pass uses them. [Gradient checkpointing](/tools/gradient-checkpointing) trades extra compute for less memory by recomputing some of them.

## Key concepts and terms

| Term | Meaning |
|---|---|
| Gradient | The vector of partial derivatives of the loss with respect to parameters |
| Chain rule | Rule for differentiating a composition of functions |
| Loss | A single number measuring prediction error |
| Activation | The output of a layer, or the non-linear function applied to it |
| Forward pass | Computing predictions from inputs |
| Backward pass | Computing gradients from the loss back to the inputs |
| Reverse-mode autodiff | General technique of which backpropagation is a case |
| Vanishing gradient | Gradients shrinking towards zero through many layers |
| Exploding gradient | Gradients growing very large through many layers |
| Gradient clipping | Limiting gradient size to keep training stable |
| Learning rate | How far the optimizer moves weights along the gradient |

## A worked example

This tiny example uses round numbers so the arithmetic is easy to follow. A network has one input x, one weight w, one bias b and no activation. It predicts y_hat = w × x + b. The loss is the squared error L = (y_hat − y)².

1. Start with w = 0.5, b = 0, input x = 2 and target y = 3.
2. **Forward pass:** y_hat = 0.5 × 2 + 0 = 1. Loss L = (1 − 3)² = 4.
3. **Derivative of the loss with respect to the prediction:** dL/dy_hat = 2 × (y_hat − y) = 2 × (−2) = −4.
4. **Chain rule for w:** dy_hat/dw = x = 2, so dL/dw = −4 × 2 = −8.
5. **Chain rule for b:** dy_hat/db = 1, so dL/db = −4.
6. **Update** with learning rate 0.05: w becomes 0.5 − 0.05 × (−8) = 0.9; b becomes 0 − 0.05 × (−4) = 0.2.
7. **Check:** the new prediction is 0.9 × 2 + 0.2 = 2.0, and the loss falls from 4 to 1.

A real network repeats exactly this pattern across millions of weights and many layers. Each layer multiplies the incoming gradient by its own local derivative, and the framework does the bookkeeping.

## Strengths and limitations

**Strengths:**

- Computes all gradients at roughly the cost of one extra pass.
- Works for any network built from differentiable operations.
- Fully automated in modern frameworks.
- Scales from tiny models to models with billions of parameters.

**Limitations:**

- Needs differentiable operations; discrete choices require workarounds.
- Stores forward activations, which takes a lot of memory.
- Gradients can vanish or explode in deep or recurrent networks.
- Gives only local information, so training can stall in flat regions.
- Unlike learning in the brain as currently understood, it needs a separate backward pass, which some researchers see as a limit on biological plausibility.

## How it compares with alternatives

| Method | Idea | Typical use |
|---|---|---|
| Backpropagation | Exact gradients by reverse-mode differentiation | Training almost all neural networks |
| Forward-mode differentiation | Propagates derivatives forwards with the computation | Few inputs, many outputs; some scientific code in [JAX](/tools/jax) |
| Numerical gradients | Nudge each weight and measure the change | Checking gradients in tests, not training |
| Evolution strategies | Estimate improvement from random perturbations | Non-differentiable objectives, some [reinforcement learning](/tools/reinforcement-learning) |
| Local learning rules | Updates from local signals, as in [spiking networks](/tools/spiking-neural-networks) | Neuromorphic hardware research |

## Practical guidance and best practices

1. Let the framework compute gradients; hand-written derivatives are a source of bugs.
2. Use activation functions such as ReLU or GELU that keep gradients flowing.
3. Initialise weights with schemes matched to your activations.
4. Add normalisation layers and residual connections to deep models.
5. Clip gradients by global norm when training transformers or recurrent networks.
6. Monitor gradient norms during training; sudden spikes or collapses point to trouble.
7. Use [mixed precision](/tools/mixed-precision) with loss scaling where needed to avoid underflowing gradients.
8. Use gradient checkpointing when memory, not compute, is the bottleneck.

## Common mistakes

- Forgetting to reset gradients between steps, so they accumulate unintentionally.
- Detaching tensors from the graph by accident, so some weights never update.
- Using a learning rate so high that gradients explode within a few steps.
- Blaming backpropagation for problems that come from data or the loss choice.
- Running evaluation with gradient tracking on, wasting memory.

## Frequently asked questions

### Is backpropagation the same as gradient descent?

No. Backpropagation computes the gradient. Gradient descent, or an optimizer such as Adam, uses that gradient to change the weights. They work together in every training step but do different jobs.

### Why is it called backpropagation?

Because the error signal travels backwards: from the loss at the output, through each layer in reverse order, to the earliest weights. Each layer receives the gradient from the layer after it and passes a new gradient to the layer before it.

### What is the vanishing gradient problem?

When many layers each multiply the gradient by a number smaller than one, the gradient reaching early layers becomes tiny, and those layers barely learn. Better activations, initialisation, normalisation and residual connections largely address it.

### Do I need to understand calculus to use backpropagation?

To train models with modern frameworks, no; they handle it for you. Understanding the chain rule helps, though, when you debug training problems, design custom layers or reason about memory and stability.

### Why does training use so much memory?

The backward pass needs values computed during the forward pass, so the framework keeps them. Larger batches, longer sequences and more layers all increase what must be stored. Gradient checkpointing and smaller batches reduce it.

## Glossary

- Autograd: PyTorch's automatic differentiation engine.
- Computational graph: a network of operations connecting inputs to outputs.
- Derivative: the rate at which one quantity changes as another changes.
- Epoch: one full pass over the training data.
- Partial derivative: a derivative with respect to one variable while others are held fixed.
- Residual connection: a shortcut that adds a layer's input to its output, helping gradients flow.
- Stochastic gradient descent: updating weights from gradients of small random batches.
- Weight: a learned number inside a network.

## Summary

Backpropagation is the algorithm that computes how every weight in a neural network affects the error, by applying the chain rule backwards from the loss. Rooted in reverse-mode automatic differentiation from 1970 and popularised for neural networks in 1986, it remains the engine of deep learning training. It is efficient and fully automated in modern frameworks, while its weak points, such as memory use and vanishing or exploding gradients, are managed with good architecture, initialisation, normalisation and careful optimisation.