Özer Cem KELAHMETKELAHMET
Topics · Deep learning

How neural networks learn: loss, backpropagation, optimisers and learning rates

Every neural network learns with the same loop: predict, measure the error, compute how each weight affected it, and nudge the weights. This page explains each piece and how to tune it.

Overview

Training is repeated small corrections. The network predicts, a loss function scores the mistake, backpropagation computes the gradient (how the loss changes as each weight changes), and an optimiser moves every weight a little in the direction that lowers the loss. Millions of such steps turn random weights into a useful model.

The pieces

Piece Role Read more Try it
Loss function One number for how wrong the predictions are Loss functions Simulation
Backpropagation Gradient of the loss for every weight, by the chain rule Backpropagation Simulation
Optimiser How the gradient becomes a weight update Optimizers Simulation
Learning rate schedule Step size over the run Schedules Simulation
Initialisation Starting weights that keep signals stable Weight initialization
Activation function Non-linearity that lets layers bend Activation functions Simulation

Choosing the loss

Regression uses mean squared or absolute error; classification uses cross-entropy, which punishes confident mistakes heavily; language models use cross-entropy on the next token. The loss defines what "better" means, so a wrong loss trains the wrong behaviour.

Optimisers in practice

Plain gradient descent uses one learning rate for every weight. Momentum smooths steps across batches. Adam and AdamW keep a running estimate of each weight's gradient size and scale its step, which makes them robust defaults; AdamW adds proper weight decay. For large models, AdamW with warmup and cosine decay is the standard recipe.

The learning rate is the most important setting

Too high and the loss jumps or diverges; too low and training crawls or stalls. Warmup starts small so early, noisy gradients cannot throw the model off, then a decay lowers the rate as the model settles. Gradient clipping caps unusually large steps; see the training run simulation.

Batch size

Larger batches give smoother gradients and better hardware use but need a larger learning rate and can generalise slightly worse. Gradient accumulation simulates a large batch on small hardware.

Learn it hands-on

Follow one neuron → backpropagation → optimisers → a real training run, or read train a neural network from scratch.

Frequently asked questions

Why is my loss NaN?

Usually a learning rate that is too high, missing gradient clipping, or numeric overflow in low precision. Lower the rate, add warmup and clipping, and check inputs for invalid values.

Adam or SGD?

AdamW is the default for transformers and most new models; SGD with momentum is still competitive for some vision models.

Summary

Loss defines the goal, backpropagation finds the direction, the optimiser takes the step, and the learning rate schedule decides how big each step is. Get the learning rate right first.

Back to the topic

This page is part of Deep learning, which maps every concept, tool, simulation and article on the subject.

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open