Learning-rate schedules: warmup, cosine, step and one-cycle
The learning rate is the most important hyperparameter, and it rarely stays constant. Pick schedules, set the peak and the warmup, and watch what each does to the same noisy training problem.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The upper chart is the learning rate at every step for each selected schedule. The lower chart is the training loss of SGD on an ill-conditioned quadratic with noisy gradients, run with that schedule and the same random seed, so the curves differ only by the schedule.
Things to try
- Raise the peak learning rate until the constant schedule diverges; see which schedules survive.
- Set warmup to 0 and to 20%.
- Turn the gradient noise up: decaying schedules pull ahead of constant ones.
How it works in real systems
Most large models are trained with linear warmup followed by cosine or linear decay (Hugging Face get_cosine_schedule_with_warmup); one-cycle is popular for fast training of vision models; the original transformer used inverse square root.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · The schedules as functions
import math
def warmup_cosine(step, total, peak, warmup, floor=0.0):
if step < warmup: return peak * (step + 1) / warmup
p = (step - warmup) / max(1, total - warmup)
return floor + (peak - floor) * 0.5 * (1 + math.cos(math.pi * p))
def inverse_sqrt(step, peak, warmup): # "Attention Is All You Need"
return peak * min((step + 1) / warmup, math.sqrt(warmup / (step + 1)))
print([round(warmup_cosine(s, 1000, 3e-4, 50), 6) for s in (0, 49, 500, 999)])
Step 2 · Built-in PyTorch and Hugging Face schedulers
import torch
from transformers import get_cosine_schedule_with_warmup
model = torch.nn.Linear(10, 1)
opt = torch.optim.AdamW(model.parameters(), lr=3e-4)
sched = get_cosine_schedule_with_warmup(opt, num_warmup_steps=100, num_training_steps=10_000)
# or: torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=3e-3, total_steps=10_000)
for step in range(5):
opt.step(); sched.step() # call once per optimiser step, after opt.step()
print(sched.get_last_lr())
Step 3 · Find a good peak learning rate
The learning-rate range test: raise the learning rate exponentially over a few hundred steps and watch the loss. Pick a peak somewhat below where the loss starts to blow up.
def lr_range_test(model, loss_fn, batches, lo=1e-6, hi=1, steps=200):
opt = torch.optim.AdamW(model.parameters(), lr=lo); out = []
for i, (xb, yb) in zip(range(steps), batches):
lr = lo * (hi / lo) ** (i / (steps - 1))
for g in opt.param_groups: g["lr"] = lr
loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
out.append((lr, loss.item()))
if loss.item() > 4 * min(l for _, l in out): break
return out
More simulations
← Init, BatchNorm, dropout · All AI Simulations · Positional encoding →