Loss functions: MSE, cross-entropy, Huber, focal and hinge
Training minimises one number, the loss. How that number punishes small and large mistakes decides what the model learns. Move the prediction and feel the difference.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The blue curve is the loss for the error or predicted probability you choose; the dashed amber curve is its gradient, the first signal backpropagation sends into the network. For categorical cross-entropy, move the three scores and watch softmax and the loss react.
Things to try
- Compare MSE and MAE for a large error: MSE grows with the square, so outliers dominate.
- In binary cross-entropy, make the prediction confidently wrong: the loss shoots up.
- Raise γ in focal loss: easy examples stop counting, hard ones keep their weight.
- In categorical cross-entropy, the gradient on each score is simply probability minus target.
How it works in real systems
Regression models usually train on MSE or Huber; classifiers on cross-entropy; detectors on focal loss; language models on cross-entropy over the next token, summed over trillions of tokens.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Regression losses
import torch, torch.nn as nn
pred = torch.tensor([2.5, 0.0, 2.0, 8.0]); target = torch.tensor([3.0, -0.5, 2.0, 7.0])
print("MSE ", nn.MSELoss()(pred, target).item())
print("MAE ", nn.L1Loss()(pred, target).item())
print("Huber", nn.HuberLoss(delta=1.0)(pred, target).item())
Step 2 · Classification losses
Cross-entropy takes raw logits and class indices. Binary cross-entropy with logits is the stable two-class version.
logits = torch.tensor([[2.0, 0.5, -1.0], [0.1, 0.2, 3.0]]); labels = torch.tensor([0, 2])
print("cross-entropy", nn.CrossEntropyLoss()(logits, labels).item())
print("with label smoothing", nn.CrossEntropyLoss(label_smoothing=0.1)(logits, labels).item())
z = torch.tensor([1.5, -0.3]); yb = torch.tensor([1.0, 0.0])
print("BCE", nn.BCEWithLogitsLoss()(z, yb).item())
Step 3 · Focal and hinge losses
Focal loss down-weights easy examples, for heavy class imbalance; hinge loss is the SVM loss.
def focal(logits, y, gamma=2.0):
p = torch.sigmoid(logits); pt = torch.where(y == 1, p, 1 - p)
return (-(1 - pt) ** gamma * torch.log(pt.clamp_min(1e-9))).mean()
def hinge(scores, y): # y in {-1, +1}
return torch.clamp(1 - y * scores, min=0).mean()
print(focal(z, yb).item(), hinge(z, torch.tensor([1.0, -1.0])).item())