# Learning-rate schedules: warmup, cosine, step and one-cycle

The learning rate is the most important hyperparameter, and it rarely stays constant. Pick schedules, set the peak and the warmup, and watch what each does to the same noisy training problem.

<div data-sim="lrsched"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

The upper chart is the learning rate at every step for each selected schedule. The lower chart is the training loss of SGD on an ill-conditioned quadratic with noisy gradients, run with that schedule and the same random seed, so the curves differ only by the schedule.

## Things to try

- Raise the peak learning rate until the constant schedule diverges; see which schedules survive.
- Set warmup to 0 and to 20%.
- Turn the gradient noise up: decaying schedules pull ahead of constant ones.

## How it works in real systems

Most large models are trained with linear warmup followed by cosine or linear decay (Hugging Face get_cosine_schedule_with_warmup); one-cycle is popular for fast training of vision models; the original transformer used inverse square root.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · The schedules as functions

```python
import math
def warmup_cosine(step, total, peak, warmup, floor=0.0):
    if step < warmup: return peak * (step + 1) / warmup
    p = (step - warmup) / max(1, total - warmup)
    return floor + (peak - floor) * 0.5 * (1 + math.cos(math.pi * p))
def inverse_sqrt(step, peak, warmup):          # "Attention Is All You Need"
    return peak * min((step + 1) / warmup, math.sqrt(warmup / (step + 1)))
print([round(warmup_cosine(s, 1000, 3e-4, 50), 6) for s in (0, 49, 500, 999)])
```

### Step 2 · Built-in PyTorch and Hugging Face schedulers

```python
import torch
from transformers import get_cosine_schedule_with_warmup
model = torch.nn.Linear(10, 1)
opt = torch.optim.AdamW(model.parameters(), lr=3e-4)
sched = get_cosine_schedule_with_warmup(opt, num_warmup_steps=100, num_training_steps=10_000)
# or: torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=3e-3, total_steps=10_000)
for step in range(5):
    opt.step(); sched.step()                    # call once per optimiser step, after opt.step()
    print(sched.get_last_lr())
```

### Step 3 · Find a good peak learning rate

The learning-rate range test: raise the learning rate exponentially over a few hundred steps and watch the loss. Pick a peak somewhat below where the loss starts to blow up.

```python
def lr_range_test(model, loss_fn, batches, lo=1e-6, hi=1, steps=200):
    opt = torch.optim.AdamW(model.parameters(), lr=lo); out = []
    for i, (xb, yb) in zip(range(steps), batches):
        lr = lo * (hi / lo) ** (i / (steps - 1))
        for g in opt.param_groups: g["lr"] = lr
        loss = loss_fn(model(xb), yb); opt.zero_grad(); loss.backward(); opt.step()
        out.append((lr, loss.item()))
        if loss.item() > 4 * min(l for _, l in out): break
    return out
```

<div class="cl-end"></div>

## More simulations

[← Init, BatchNorm, dropout](/ai-simulations/normalisation-and-dropout) · [All AI Simulations](/ai-simulations) · [Positional encoding →](/ai-simulations/positional-encoding)
