Özer Cem KELAHMETKELAHMET
AI Simulations · Neural network basics

Loss functions: MSE, cross-entropy, Huber, focal and hinge

Training minimises one number, the loss. How that number punishes small and large mistakes decides what the model learns. Move the prediction and feel the difference.

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

The blue curve is the loss for the error or predicted probability you choose; the dashed amber curve is its gradient, the first signal backpropagation sends into the network. For categorical cross-entropy, move the three scores and watch softmax and the loss react.

Things to try

  • Compare MSE and MAE for a large error: MSE grows with the square, so outliers dominate.
  • In binary cross-entropy, make the prediction confidently wrong: the loss shoots up.
  • Raise γ in focal loss: easy examples stop counting, hard ones keep their weight.
  • In categorical cross-entropy, the gradient on each score is simply probability minus target.

How it works in real systems

Regression models usually train on MSE or Huber; classifiers on cross-entropy; detectors on focal loss; language models on cross-entropy over the next token, summed over trillions of tokens.

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · Regression losses

import torch, torch.nn as nn
pred = torch.tensor([2.5, 0.0, 2.0, 8.0]); target = torch.tensor([3.0, -0.5, 2.0, 7.0])
print("MSE  ", nn.MSELoss()(pred, target).item())
print("MAE  ", nn.L1Loss()(pred, target).item())
print("Huber", nn.HuberLoss(delta=1.0)(pred, target).item())

Step 2 · Classification losses

Cross-entropy takes raw logits and class indices. Binary cross-entropy with logits is the stable two-class version.

logits = torch.tensor([[2.0, 0.5, -1.0], [0.1, 0.2, 3.0]]); labels = torch.tensor([0, 2])
print("cross-entropy", nn.CrossEntropyLoss()(logits, labels).item())
print("with label smoothing", nn.CrossEntropyLoss(label_smoothing=0.1)(logits, labels).item())
z = torch.tensor([1.5, -0.3]); yb = torch.tensor([1.0, 0.0])
print("BCE", nn.BCEWithLogitsLoss()(z, yb).item())

Step 3 · Focal and hinge losses

Focal loss down-weights easy examples, for heavy class imbalance; hinge loss is the SVM loss.

def focal(logits, y, gamma=2.0):
    p = torch.sigmoid(logits); pt = torch.where(y == 1, p, 1 - p)
    return (-(1 - pt) ** gamma * torch.log(pt.clamp_min(1e-9))).mean()
def hinge(scores, y):                     # y in {-1, +1}
    return torch.clamp(1 - y * scores, min=0).mean()
print(focal(z, yb).item(), hinge(z, torch.tensor([1.0, -1.0])).item())

More simulations

← Activations · All AI Simulations · Backpropagation →

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open