A training run, live: warmup, gradient clipping, loss spikes and checkpoints
Long training runs fail in predictable ways: a bad batch, a learning rate that is too high, a numeric blow-up. This simulation trains a real small network live and gives you the tools teams use to keep a run alive.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The charts show the smoothed training loss, the validation loss, the gradient norm before clipping and the learning rate schedule. The square shows the decision regions the network has learned; the log records checkpoints, bad batches and rollbacks.
Things to try
- Train once with the defaults: warmup, then a cosine decay.
- Inject a bad batch with clipping on, then off: compare the gradient norm and the damage.
- Raise the peak learning rate to 1 and remove warmup: watch the early loss.
- Let the automatic rollback catch a spike, or restore the last checkpoint by hand.
How it works in real systems
Production runs log loss, gradient norm and learning rate every step, save checkpoints every few hundred or thousand steps, skip or replay batches around a spike, and restart from the last good checkpoint after a hardware failure. Gradient clipping at a norm of about 1.0 is standard for transformers.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · A training loop that survives
import math, torch
def lr_at(step, peak=3e-4, warmup=2000, total=100_000, floor=0.1):
if step < warmup: return peak * (step + 1) / warmup
p = (step - warmup) / (total - warmup)
return peak * (floor + (1 - floor) * 0.5 * (1 + math.cos(math.pi * p)))
best, ckpt_every = float("inf"), 1000
for step, batch in enumerate(loader):
for g in opt.param_groups: g["lr"] = lr_at(step)
loss = model(**batch).loss
loss.backward()
gnorm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
if not torch.isfinite(gnorm) or loss.item() > 3 * best: # spike: skip this batch
opt.zero_grad(); continue
opt.step(); opt.zero_grad(); best = min(best, loss.item())
if step % ckpt_every == 0:
torch.save({"model": model.state_dict(), "opt": opt.state_dict(), "step": step}, f"ckpt/{step}.pt")
Step 2 · Resume from the last checkpoint
import glob, os
last = max(glob.glob("ckpt/*.pt"), key=os.path.getmtime)
state = torch.load(last, map_location="cpu")
model.load_state_dict(state["model"]); opt.load_state_dict(state["opt"])
start = state["step"] + 1 # also restore the data loader position and RNG states in a real run