# Loss functions: MSE, cross-entropy, Huber, focal and hinge

Training minimises one number, the loss. How that number punishes small and large mistakes decides what the model learns. Move the prediction and feel the difference.

<div data-sim="loss"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

The blue curve is the loss for the error or predicted probability you choose; the dashed amber curve is its gradient, the first signal backpropagation sends into the network. For categorical cross-entropy, move the three scores and watch softmax and the loss react.

## Things to try

- Compare MSE and MAE for a large error: MSE grows with the square, so outliers dominate.
- In binary cross-entropy, make the prediction confidently wrong: the loss shoots up.
- Raise γ in focal loss: easy examples stop counting, hard ones keep their weight.
- In categorical cross-entropy, the gradient on each score is simply probability minus target.

## How it works in real systems

Regression models usually train on MSE or Huber; classifiers on cross-entropy; detectors on focal loss; language models on cross-entropy over the next token, summed over trillions of tokens.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Regression losses

```python
import torch, torch.nn as nn
pred = torch.tensor([2.5, 0.0, 2.0, 8.0]); target = torch.tensor([3.0, -0.5, 2.0, 7.0])
print("MSE  ", nn.MSELoss()(pred, target).item())
print("MAE  ", nn.L1Loss()(pred, target).item())
print("Huber", nn.HuberLoss(delta=1.0)(pred, target).item())
```

### Step 2 · Classification losses

Cross-entropy takes raw logits and class indices. Binary cross-entropy with logits is the stable two-class version.

```python
logits = torch.tensor([[2.0, 0.5, -1.0], [0.1, 0.2, 3.0]]); labels = torch.tensor([0, 2])
print("cross-entropy", nn.CrossEntropyLoss()(logits, labels).item())
print("with label smoothing", nn.CrossEntropyLoss(label_smoothing=0.1)(logits, labels).item())
z = torch.tensor([1.5, -0.3]); yb = torch.tensor([1.0, 0.0])
print("BCE", nn.BCEWithLogitsLoss()(z, yb).item())
```

### Step 3 · Focal and hinge losses

Focal loss down-weights easy examples, for heavy class imbalance; hinge loss is the SVM loss.

```python
def focal(logits, y, gamma=2.0):
    p = torch.sigmoid(logits); pt = torch.where(y == 1, p, 1 - p)
    return (-(1 - pt) ** gamma * torch.log(pt.clamp_min(1e-9))).mean()
def hinge(scores, y):                     # y in {-1, +1}
    return torch.clamp(1 - y * scores, min=0).mean()
print(focal(z, yb).item(), hinge(z, torch.tensor([1.0, -1.0])).item())
```

<div class="cl-end"></div>

## More simulations

[← Activations](/ai-simulations/activation-functions) · [All AI Simulations](/ai-simulations) · [Backpropagation →](/ai-simulations/backpropagation)
