# A real training run, live: loss curves, checkpoints and a confusion matrix

This is what training looks like from the inside. A real network learns to read handwritten digits while the dashboard shows the same signals an engineer watches on a GPU cluster: loss curves, learning rate schedule, gradient norm, checkpoints and the mistakes it still makes.

<div data-sim="run"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

The data is the classic 8×8 handwritten digits set: 1,437 images for training and 360 the network never trains on. Each step takes a mini-batch, runs the forward pass, measures cross-entropy, backpropagates and updates the weights with AdamW. After every epoch the model is scored on the validation images, the best version is saved as a checkpoint, and training stops early if validation stops improving.

## Things to try

- Start with the defaults and watch validation accuracy pass 95% within a few epochs.
- Set the peak learning rate to 0.1: the loss jumps and the gradient norm spikes. That is why real runs use warmup and clipping.
- Turn on 20% wrong labels: training accuracy stalls below 100% while validation stays high for a while, then the model starts memorising noise.
- Use the small model and dropout 0.5: training is slower but the gap between train and validation shrinks.
- Look at the red samples and the confusion matrix: 3 versus 8 and 4 versus 9 are the classic confusions.

## How it works in real systems

Large models are trained with exactly this loop, only bigger. A language model run logs the same curves to tools such as Weights & Biases or TensorBoard, uses a warmup then cosine or linear decay schedule, clips gradients, saves checkpoints every few thousand steps and keeps the best one by validation loss. The differences are scale (billions of parameters, thousands of GPUs, data parallelism) and the loss (cross-entropy over the next token instead of over ten digits).

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Set up and load the data

You need Python 3.10+, PyTorch and scikit-learn. The 8×8 digits set ships with scikit-learn, so nothing is downloaded.

```bash
pip install torch scikit-learn numpy
```

```python
import numpy as np, torch, torch.nn as nn
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True)          # 1,797 images, 64 pixels each (values 0–16)
X = (X / 16.0).astype("float32")              # scale to [0, 1]
X_tr, X_va, y_tr, y_va = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
X_tr, X_va = torch.tensor(X_tr), torch.tensor(X_va)
y_tr, y_va = torch.tensor(y_tr), torch.tensor(y_va)
print(X_tr.shape, X_va.shape)                 # torch.Size([1437, 64]) torch.Size([360, 64])
```

A stratified split keeps the share of every digit equal in training and validation, so the validation accuracy is comparable to the training accuracy.

### Step 2 · Define the model

The same network as the simulation: 64 inputs, one hidden layer, 10 outputs (one logit per digit).

```python
torch.manual_seed(0)
model = nn.Sequential(
    nn.Linear(64, 64), nn.ReLU(),
    nn.Linear(64, 10),
)
print(sum(p.numel() for p in model.parameters()), "parameters")   # 4,810
loss_fn = nn.CrossEntropyLoss()     # softmax + negative log-likelihood in one, numerically stable
```

`CrossEntropyLoss` expects raw logits, not probabilities: do not add a softmax layer at the end.

### Step 3 · Optimiser and learning-rate schedule

AdamW with weight decay, a short linear warmup, then cosine decay. The schedule is a function of the step; `LambdaLR` multiplies the base learning rate by it.

```python
import math
EPOCHS, BATCH = 40, 64
steps_per_epoch = math.ceil(len(X_tr) / BATCH)
total, warmup = EPOCHS * steps_per_epoch, 2 * steps_per_epoch

opt = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=1e-4)
def lr_factor(step):
    if step < warmup:
        return (step + 1) / warmup
    progress = (step - warmup) / max(1, total - warmup)
    return 0.05 + 0.95 * 0.5 * (1 + math.cos(math.pi * progress))   # floor at 5% of the peak
sched = torch.optim.lr_scheduler.LambdaLR(opt, lr_factor)
```

### Step 4 · The training loop: clipping, checkpoints, early stopping

Every epoch: shuffle, run mini-batches, clip the gradient norm, step the optimiser and the schedule, then evaluate on the validation set. The best model is saved; training stops when validation loss has not improved for `PATIENCE` epochs.

```python
PATIENCE, best, bad = 6, float("inf"), 0
for epoch in range(EPOCHS):
    model.train()
    perm = torch.randperm(len(X_tr))
    for i in range(0, len(X_tr), BATCH):
        idx = perm[i:i + BATCH]
        loss = loss_fn(model(X_tr[idx]), y_tr[idx])
        opt.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
        opt.step(); sched.step()

    model.eval()
    with torch.no_grad():
        va_logits = model(X_va)
        va_loss = loss_fn(va_logits, y_va).item()
        va_acc = (va_logits.argmax(1) == y_va).float().mean().item()
    print(f"epoch {epoch:2d}  lr {sched.get_last_lr()[0]:.5f}  val loss {va_loss:.4f}  val acc {va_acc:.3f}")

    if va_loss < best - 1e-4:
        best, bad = va_loss, 0
        torch.save({"model": model.state_dict(), "epoch": epoch, "val_loss": va_loss}, "best.pt")
    else:
        bad += 1
        if bad >= PATIENCE:
            print("early stopping"); break
```

Typical result: about 97% validation accuracy within 20 epochs.

### Step 5 · Restore the best checkpoint and read the confusion matrix

Training may have continued past the best epoch, so reload the checkpoint before evaluating. The confusion matrix shows which digits are mistaken for which.

```python
from sklearn.metrics import confusion_matrix, classification_report
ckpt = torch.load("best.pt")
model.load_state_dict(ckpt["model"]); model.eval()
with torch.no_grad():
    pred = model(X_va).argmax(1).numpy()
print("best epoch", ckpt["epoch"])
print(confusion_matrix(y_va.numpy(), pred))
print(classification_report(y_va.numpy(), pred, digits=3))
```

Look at the off-diagonal cells: on this data set 8s and 1s, and 9s and 3s, are the usual confusions, the same ones the simulation shows.

<div class="cl-end"></div>

## More simulations

[← Evaluation](/ai-simulations/train-evaluation) · [All AI Simulations](/ai-simulations) · [Network playground →](/ai-simulations/neural-network-playground)
