Gradient descent and its variants: SGD, momentum, Nesterov, RMSProp, Adam
All optimisers use the gradient. They differ in how they turn it into a step. Race six of them across valleys, saddles and bowls and see why Adam became the default.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
Contours are lines of equal loss; each coloured path is one optimiser starting from the same point. The chart on the right shows the loss after every step on a log scale. Click the surface to choose a new start.
Things to try
- Elongated bowl: plain SGD zig-zags across the narrow direction; momentum and Adam glide along it.
- Rosenbrock valley: the curved floor is hard for everyone; watch who reaches the far end.
- Saddle point: start near the middle and see which optimisers escape.
- Two valleys: the start decides which minimum each optimiser finds.
How it works in real systems
Large models are usually trained with AdamW (Adam with decoupled weight decay), a warmup then decay learning rate schedule and gradient clipping. SGD with momentum is still common in computer vision because it often generalises slightly better.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · A hard function to minimise
The Rosenbrock function has a long curved valley: easy to enter, slow to follow.
import torch
def rosen(p): x, y = p; return (1 - x) ** 2 + 100 * (y - x ** 2) ** 2
Step 2 · Race the optimisers
Same start, same number of steps, each with a learning rate that suits it.
setups = {"SGD": (torch.optim.SGD, dict(lr=1e-3)), "Momentum": (torch.optim.SGD, dict(lr=1e-3, momentum=0.9)),
"Nesterov": (torch.optim.SGD, dict(lr=1e-3, momentum=0.9, nesterov=True)), "Adagrad": (torch.optim.Adagrad, dict(lr=0.5)),
"RMSprop": (torch.optim.RMSprop, dict(lr=1e-2)), "Adam": (torch.optim.Adam, dict(lr=5e-2)), "AdamW": (torch.optim.AdamW, dict(lr=5e-2, weight_decay=0.0))}
for name, (cls, kw) in setups.items():
p = torch.tensor([-1.5, 2.0], requires_grad=True); opt = cls([p], **kw)
for _ in range(2000):
opt.zero_grad(); loss = rosen(p); loss.backward(); opt.step()
print(f"{name:9s} loss {loss.item():.2e} at {p.detach().numpy().round(3)}")
Step 3 · Adam, written out
Momentum of the gradient (m) and of its square (v), bias-corrected, then a per-parameter step size.
def adam_step(p, g, m, v, t, lr=1e-3, b1=0.9, b2=0.999, eps=1e-8):
m = b1 * m + (1 - b1) * g
v = b2 * v + (1 - b2) * g * g
m_hat, v_hat = m / (1 - b1 ** t), v / (1 - b2 ** t)
return p - lr * m_hat / (v_hat ** 0.5 + eps), m, v
AdamW differs only in applying weight decay directly to the weights instead of adding it to the gradient.