# Gradient descent and its variants: SGD, momentum, Nesterov, RMSProp, Adam

All optimisers use the gradient. They differ in how they turn it into a step. Race six of them across valleys, saddles and bowls and see why Adam became the default.

<div data-sim="optim"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

Contours are lines of equal loss; each coloured path is one optimiser starting from the same point. The chart on the right shows the loss after every step on a log scale. Click the surface to choose a new start.

## Things to try

- Elongated bowl: plain SGD zig-zags across the narrow direction; momentum and Adam glide along it.
- Rosenbrock valley: the curved floor is hard for everyone; watch who reaches the far end.
- Saddle point: start near the middle and see which optimisers escape.
- Two valleys: the start decides which minimum each optimiser finds.

## How it works in real systems

Large models are usually trained with AdamW (Adam with decoupled weight decay), a warmup then decay learning rate schedule and gradient clipping. SGD with momentum is still common in computer vision because it often generalises slightly better.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · A hard function to minimise

The Rosenbrock function has a long curved valley: easy to enter, slow to follow.

```python
import torch
def rosen(p): x, y = p; return (1 - x) ** 2 + 100 * (y - x ** 2) ** 2
```

### Step 2 · Race the optimisers

Same start, same number of steps, each with a learning rate that suits it.

```python
setups = {"SGD": (torch.optim.SGD, dict(lr=1e-3)), "Momentum": (torch.optim.SGD, dict(lr=1e-3, momentum=0.9)),
          "Nesterov": (torch.optim.SGD, dict(lr=1e-3, momentum=0.9, nesterov=True)), "Adagrad": (torch.optim.Adagrad, dict(lr=0.5)),
          "RMSprop": (torch.optim.RMSprop, dict(lr=1e-2)), "Adam": (torch.optim.Adam, dict(lr=5e-2)), "AdamW": (torch.optim.AdamW, dict(lr=5e-2, weight_decay=0.0))}
for name, (cls, kw) in setups.items():
    p = torch.tensor([-1.5, 2.0], requires_grad=True); opt = cls([p], **kw)
    for _ in range(2000):
        opt.zero_grad(); loss = rosen(p); loss.backward(); opt.step()
    print(f"{name:9s} loss {loss.item():.2e} at {p.detach().numpy().round(3)}")
```

### Step 3 · Adam, written out

Momentum of the gradient (m) and of its square (v), bias-corrected, then a per-parameter step size.

```python
def adam_step(p, g, m, v, t, lr=1e-3, b1=0.9, b2=0.999, eps=1e-8):
    m = b1 * m + (1 - b1) * g
    v = b2 * v + (1 - b2) * g * g
    m_hat, v_hat = m / (1 - b1 ** t), v / (1 - b2 ** t)
    return p - lr * m_hat / (v_hat ** 0.5 + eps), m, v
```

AdamW differs only in applying weight decay directly to the weights instead of adding it to the gradient.

<div class="cl-end"></div>

## More simulations

[← Backpropagation](/ai-simulations/backpropagation) · [All AI Simulations](/ai-simulations) · [Convolution →](/ai-simulations/convolution)
