Gradient descent

Overview

Gradient descent is the method that most machine learning models use to learn. A model has parameters, such as the weights of a neural network or the coefficients of a regression. A loss function scores how wrong the model is on the training data. Gradient descent changes the parameters a little at a time so that the loss goes down. It does this by computing the gradient, which points in the direction where the loss rises fastest, and stepping the other way.

The everyday picture is a walker on a foggy hillside who wants to reach the valley floor. They cannot see the valley, only the ground under their feet. So they feel which way the slope falls most steeply, take a step that way, and repeat. Long strides cover ground quickly but can overshoot the bottom and climb the far side. Tiny steps are safe but slow. That stride length is the learning rate, and choosing it well is most of the practical art.

This page explains the update rule, the three main variants (batch, stochastic and mini-batch), how the learning rate controls convergence, why local minima and saddle points matter, how momentum and Nesterov momentum speed things up, and how gradient descent connects to backpropagation and adaptive optimizers such as Adam.

What it is

Gradient descent is a first-order optimisation algorithm. "First-order" means it uses only the gradient (first derivatives), not curvature (second derivatives). Its parts are few.

The method is general. It works for any loss that is differentiable (or nearly so) with respect to the parameters. That is why the same idea trains linear regression, logistic regression, the base learners inside gradient boosting (which applies the idea in function space), and deep neural networks with billions of weights.

Where it came from

The method of steepest descent is usually credited to Augustin-Louis Cauchy, who described it in 1847 as a way to solve systems of equations by minimising a function. For more than a century it was a tool of numerical analysis.

In 1951 Herbert Robbins and Sutton Monro published "A Stochastic Approximation Method" in the Annals of Mathematical Statistics. It showed how to find a root of a function when you can only observe noisy measurements, provided the step sizes shrink in a suitable way. This is the mathematical ancestor of stochastic gradient descent.

In 1964 Boris Polyak introduced the "heavy ball" method, which adds momentum by letting the previous step carry over into the next. In 1983 Yurii Nesterov published an accelerated gradient method that, for smooth convex problems, provably converges faster than plain gradient descent.

The 1986 Nature paper by David Rumelhart, Geoffrey Hinton and Ronald Williams popularised backpropagation, the efficient way to compute gradients in multilayer networks, and made gradient descent the standard way to train them. In 2013 Ilya Sutskever and colleagues showed that well-tuned momentum and careful initialisation were enough to train deep networks that had been thought to need more complex methods. In 2014 Yann Dauphin and colleagues argued that in high-dimensional problems, saddle points rather than poor local minima are the main obstacle. The same year Diederik Kingma and Jimmy Ba proposed Adam, which adapts the step size for each parameter and became a default choice for deep learning.

How it works

The basic update

At each step, compute the gradient of the loss with respect to every parameter, then move each parameter a small amount against its gradient.

w ← w − η · ∇L(w)                                (plain gradient descent)

g = (1/B) · Σ_{i in batch} ∇ℓ(w; x_i, y_i)       (mini-batch gradient estimate)

v ← μ · v + g ;   w ← w − η · v                  (momentum, PyTorch form)

v ← μ · v + g ;   w ← w − η · (g + μ · v)        (Nesterov, PyTorch form)

Here w is the parameter vector, η (eta) is the learning rate, ∇L(w) is the gradient of the total loss, ℓ is the loss on one example (x_i, y_i), B is the batch size, g is the gradient estimated on the current batch, v is the velocity (a running sum of past gradients) and μ (mu) is the momentum coefficient, typically around 0.9.

Batch, stochastic and mini-batch

The three classic variants differ only in how many examples are used to estimate the gradient.

• Batch (full-batch) gradient descent uses the whole training set for every step. The gradient is exact, so the path is smooth, but each step is expensive and the whole dataset must be processed before any progress is made.
• Stochastic gradient descent (SGD) in its strict sense uses one example per step. Steps are cheap and noisy. The noise can help the model escape flat regions, but it also makes the loss curve jumpy.
• Mini-batch gradient descent uses a small group, often 32 to a few thousand examples. It is the practical middle ground and is what nearly all deep learning uses. It also matches how GPUs work, since a batch can be processed in parallel. In everyday speech, and in library names such as torch.optim.SGD, "SGD" usually means mini-batch gradient descent.

The noise in a mini-batch gradient shrinks roughly with the square root of the batch size. Doubling the batch does not halve the noise; you need four times as many examples to halve it. This is one reason very large batches give diminishing returns.

The learning rate and convergence

The learning rate is the most important setting. If it is too large, the loss oscillates or explodes, because each step overshoots the minimum. If it is too small, training crawls and may stall on a plateau. For a simple bowl-shaped (convex quadratic) loss there is a hard limit: the step must be smaller than two divided by the largest curvature, or the iterates diverge.

On convex problems with a suitable step size, gradient descent is guaranteed to approach the global minimum. With noisy gradients, convergence needs the step size to shrink over time, which is the idea behind Robbins and Monro's conditions and behind modern learning rate schedules. In practice, deep learning uses a schedule such as warm-up followed by cosine decay rather than a fixed rate.

Local minima, plateaus and saddle points

Neural network losses are not convex. They can have many local minima, flat plateaus and saddle points. A saddle point is like the middle of a horse's saddle: downhill in some directions and uphill in others, with a gradient of zero at the centre. In high dimensions there are vastly more saddle points than true local minima, and plain gradient descent can slow to a crawl near them because the gradient is tiny. Mini-batch noise and momentum both help to move away. Research and practice suggest that in large networks many local minima reach similar loss values, so finding a "good enough" minimum that generalises matters more than finding the global one.

Momentum and Nesterov

Momentum keeps a running average of recent gradients and steps along it. In a long narrow valley, plain gradient descent zig-zags across the steep walls and moves slowly along the floor. Momentum cancels the zig-zag, because alternating gradients average out, and builds speed along the consistent direction. Nesterov momentum looks ahead: it evaluates the correction as if the velocity step had already been taken, which often reduces overshooting. In PyTorch you enable it with nesterov=True alongside a non-zero momentum.

Where backpropagation fits

Gradient descent needs gradients; it does not say how to compute them. For a network with millions of parameters, backpropagation computes all of them in one backward pass using the chain rule. Frameworks such as PyTorch do this automatically. Backpropagation is the measuring tape; gradient descent is the walk.

From SGD to adaptive optimizers

Adaptive optimizers are gradient descent with extra bookkeeping. RMSProp divides each parameter's step by a running average of its squared gradients, so parameters with large gradients take smaller steps. Adam combines this with momentum and adds a bias correction for the early steps. AdamW decouples weight decay from the gradient update. All of them still follow the same loop: estimate a gradient on a batch, transform it, step against it. The wider field of optimisation also includes second-order methods such as L-BFGS, which use curvature information but are less common for large networks.

import torch

model = torch.nn.Linear(10, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.05,
                            momentum=0.9, nesterov=True)
loss_fn = torch.nn.MSELoss()

for x, y in loader:              # mini-batches from a DataLoader
    optimizer.zero_grad()        # clear old gradients
    loss = loss_fn(model(x), y)  # forward pass
    loss.backward()              # backpropagation fills .grad
    optimizer.step()             # w <- w - lr * update

Key concepts and terms

A worked example

Take a toy linear model ŷ = w · x with one weight and mean squared error. The data are three points: (1, 2), (2, 4) and (3, 6). The true answer is w = 2. Start at w = 0 with learning rate 0.05. This example is illustrative.

The loss is L(w) = (1/3) · Σ (w·x − y)² and its gradient is (2/3) · Σ x · (w·x − y).

1. Step 1. At w = 0, the errors are −2, −4 and −6. The gradient is (2/3) × (1×−2 + 2×−4 + 3×−6) = (2/3) × −28 ≈ −18.67. The update is w = 0 − 0.05 × (−18.67) ≈ 0.933.
2. Step 2. At w = 0.933, the errors are −1.067, −2.133 and −3.2. The gradient is (2/3) × (−1.067 − 4.267 − 9.6) ≈ −9.96. The new value is w ≈ 0.933 + 0.498 = 1.431.
3. Later steps. Each step closes roughly 47 per cent of the remaining gap, because the gradient here equals (28/3) × (w − 2) and 1 − 0.05 × 28/3 ≈ 0.53. After ten steps w is within about 0.004 of 2.

Now try learning rate 0.25. The factor becomes 1 − 0.25 × 28/3 ≈ −1.33. The gap flips sign and grows by a third every step: the walker overshoots the valley and climbs higher on the other side each time. The threshold for this problem is 2 ÷ (28/3) ≈ 0.214. Above it, training diverges. This is the curvature limit from the "How it works" section, made concrete.

With mini-batches of one point, each step would use only one of the three gradients. Steps would be cheaper, and the path would wander around 2 rather than glide straight to it, which is why the step size is usually reduced late in training.

Strengths and limitations

Strengths:

• Simple to state and to implement, with very little memory beyond the parameters and gradients.
• Scales to billions of parameters, because each step costs about as much as one forward and backward pass.
• Works with any differentiable model and loss.
• Mini-batch noise acts as a mild regulariser and helps escape flat regions and saddle points.
• Well understood theoretically on convex problems, with clear convergence results.

Limitations:

• Sensitive to the learning rate; a poor choice wastes compute or breaks training.
• Slow on badly conditioned problems, where it zig-zags across narrow valleys.
• Finds a local, not global, minimum on non-convex losses.
• Needs gradients, so it does not apply directly to non-differentiable objectives or discrete choices.
• Sensitive to feature scale; unscaled inputs stretch the loss surface and slow progress (see data preprocessing).

How it compares with alternatives

Practical guidance and best practices

1. Scale your inputs (standardise or normalise) before training; it makes the loss surface rounder and gradient descent faster.
2. Find a learning rate empirically. A short learning-rate range test, sweeping from very small to large, shows where the loss falls fastest.
3. Use mini-batches. Start with a batch size that fits comfortably in memory; if you scale the batch up a lot, scale the learning rate and add a warm-up.
4. Add momentum of about 0.9 when using plain SGD. Try Nesterov; it rarely hurts.
5. Use a schedule. Warm-up followed by cosine or step decay usually beats a constant rate.
6. Shuffle the data every epoch so batches are not correlated.
7. Monitor both training and validation loss. Falling training loss with rising validation loss means overfitting, not an optimiser problem.
8. Clip gradients by norm in recurrent networks and very deep models to stop rare huge steps.
9. When results are unstable, lower the learning rate before changing anything else.

Common mistakes

• Forgetting to reset gradients each step (in PyTorch, optimizer.zero_grad()), so they silently accumulate.
• Using one learning rate for every problem; good values depend on the model, the optimiser and the batch size.
• Changing the batch size by a large factor without revisiting the learning rate.
• Reading a noisy mini-batch loss curve step by step instead of smoothing it or watching per-epoch averages.
• Training on unscaled features with very different ranges.
• Blaming the optimiser for a bug in the loss, the labels or the data pipeline.

Frequently asked questions

Is SGD the same as mini-batch gradient descent?

Strictly, stochastic gradient descent uses one example per step and mini-batch gradient descent uses several. In practice the terms are used interchangeably, and torch.optim.SGD performs mini-batch updates on whatever batch you pass it. The word "stochastic" refers to the randomness of the gradient estimate, which is present in both.

Why not always use the full dataset for each step?

Because it is expensive and often unnecessary. A random mini-batch gives an unbiased estimate of the full gradient for a fraction of the cost. Many cheap, noisy steps usually make more progress per second than a few exact ones. The noise can also help the model settle in flatter regions that tend to generalise better.

Does gradient descent find the best possible model?

On convex problems such as linear and logistic regression, it can reach the global minimum. On neural networks it finds a local minimum or a flat region. In practice these are often good enough, and the quality of the data, the architecture and regularisation matter more than finding the exact global minimum.

What is the difference between gradient descent and backpropagation?

Backpropagation computes the gradient; gradient descent uses it. Backpropagation applies the chain rule backwards through the network to find how each weight affects the loss. Gradient descent then decides how far to move each weight.

Should I use SGD or Adam?

Adam and AdamW are easier to tune and are the usual default for Transformers and many tabular and language models. SGD with momentum remains common in computer vision, where it often generalises well when tuned. If you have a limited tuning budget, start with AdamW; if you are reproducing a vision paper that used SGD, use SGD.

How do I know the learning rate is too high?

The training loss jumps around, rises, or becomes NaN within the first few hundred steps. If lowering the learning rate by a factor of three to ten makes the curve smooth, the rate was too high.

Glossary

• Gradient: the direction and rate of steepest increase of a function.
• Learning rate: the multiplier that sets the step size of each update.
• Stochastic gradient descent: gradient descent using a random sample of data per step.
• Mini-batch: a small subset of training examples processed together.
• Momentum: a velocity term that accumulates past gradients.
• Nesterov momentum: momentum that evaluates the correction at a look-ahead point.
• Saddle point: a stationary point that is not a minimum or a maximum.
• Convex: having a single bowl-shaped minimum with no other local minima.
• Epoch: one full pass through the training set.
• Divergence: the loss growing instead of shrinking during training.

Summary

Gradient descent trains a model by repeatedly computing how the loss changes with each parameter and stepping a small amount downhill. Batch, stochastic and mini-batch variants differ only in how much data estimates each gradient, and mini-batches are the practical standard. The learning rate controls whether training converges smoothly, crawls or explodes, and schedules let it change over time. Non-convex losses bring local minima and, more importantly, saddle points, which momentum, Nesterov momentum and mini-batch noise help to cross. Backpropagation supplies the gradients, and adaptive optimizers such as Adam and AdamW are gradient descent with per-parameter step sizes. Understanding this one loop explains most of how modern machine learning learns.

Related

It belongs to Training fundamentals in my stack, under Machine learning. Others in the same group: