Özer Cem KELAHMETKELAHMET
Articles · Concepts

Training a neural network from scratch: forward pass, backpropagation and gradient descent

A neural network is a stack of simple functions with adjustable numbers. Training is a loop that measures how wrong those numbers are and nudges each one in the direction that makes the error smaller. This guide builds that loop by hand, and lets you watch a real network learn in the page.

At a glance
Reading time
10 min
Sections
9
Code examples
4
Simulations
1
Sources
4
Published
17 Nov 2026

Frameworks such as PyTorch hide training behind one line, loss.backward(). That line is worth understanding, because almost every training problem, from a loss that will not fall to a model that memorises its data, is easier to fix once you know what happens inside it.

This guide trains a small network to separate two classes of points on a plane. It is the same mechanism that trains large language models, at a scale you can watch.

The simulation: train it yourself

Press Train. The background shows what the network currently predicts at every point: blue for class 0, teal for class 1, stronger colour for more confidence, and the white line is the decision boundary where the network is exactly undecided. Filled dots are the training data; rings are test data the network never trains on. The diagram shows the weights (teal positive, blue negative, thicker is larger) with signals flowing through the strongest ones, and the chart shows training and test loss falling.

Interactive neural network training. It needs JavaScript; every step is also explained below.

Things to try:

  • Circle with one hidden layer of two neurons. It cannot draw a circle: two neurons can only combine two straight lines. Raise the neurons to six and it can.
  • Spiral. Needs two hidden layers and patience. With a learning rate of 1 the loss jumps around; at 0.03 it falls smoothly.
  • Noise at 0.5. The training loss keeps falling while the test loss stops falling and rises: that is overfitting, live.
  • ReLU vs tanh. ReLU boundaries are made of straight pieces; tanh boundaries are smooth curves.

Everything is computed in your browser with the same equations as below: no library, no server.

What a neuron computes

A neuron takes inputs, multiplies each by a weight, adds a bias and passes the sum through an activation function:

$$z = w_1 x_1 + w_2 x_2 + b, \qquad a = \tanh(z)$$

The weights decide how much each input matters; the bias shifts the threshold. Without the activation, stacking layers would still give a straight line, because a sum of linear functions is linear. The activation bends it. Common choices:

  • tanh squeezes to between −1 and 1 and is smooth.
  • ReLU returns \(\max(0, z)\); it is cheap and the default in deep networks.
  • sigmoid squeezes to between 0 and 1; used at the output to give a probability.

The forward pass

A layer is many neurons side by side, which is one matrix multiplication. For a network with one hidden layer:

$$h = \tanh(W_1 x + b_1), \qquad p = \sigma(W_2 h + b_2)$$

Here \(x\) is the input point, \(h\) the hidden layer's output and \(p\) the predicted probability of class 1. Computing \(p\) from \(x\) is the forward pass.

import numpy as np

rng = np.random.default_rng(0)
def init(n_in, n_hidden):
    return {"W1": rng.normal(0, 1 / np.sqrt(n_in), (n_hidden, n_in)), "b1": np.zeros(n_hidden),
            "W2": rng.normal(0, 1 / np.sqrt(n_hidden), (1, n_hidden)), "b2": np.zeros(1)}

def forward(p, X):                       # X: (n, 2)
    h = np.tanh(X @ p["W1"].T + p["b1"])  # (n, hidden)
    out = 1 / (1 + np.exp(-(h @ p["W2"].T + p["b2"])))
    return h, out[:, 0]                   # probabilities, shape (n,)

Initialisation matters more than it looks. Weights that are all zero make every neuron identical forever; weights that are too large saturate tanh so gradients vanish. Scaling by \(1/\sqrt{n_{in}}\) keeps the signal at a sensible size.

The loss: one number for "how wrong"

For two classes the standard loss is binary cross-entropy:

$$L = -\frac{1}{n}\sum_i \big[y_i \log p_i + (1 - y_i)\log(1 - p_i)\big]$$

If the true class is 1 and the network says 0.99, the loss is tiny; if it says 0.01, the loss is large. Cross-entropy punishes confident mistakes much more than hesitant ones, which is what you want from a classifier.

Backpropagation: who is to blame, and by how much

Training needs, for every weight, the answer to one question: if I increase this weight a little, does the loss go up or down, and how fast? That number is the gradient \(\partial L / \partial w\).

Backpropagation computes all of them in one backward sweep using the chain rule. Think of a relay of blame: the output layer works out how wrong it was, passes a share of that error back to each hidden neuron in proportion to the weight that connected them, and each hidden neuron scales it by how sensitive its activation was.

With sigmoid output and cross-entropy loss, the error at the output simplifies to the prediction minus the target:

$$\delta_2 = p - y$$

Going back one layer, the error is passed through the weights and multiplied by the slope of tanh, which is \(1 - h^2\):

$$\delta_1 = (\delta_2 W_2) \odot (1 - h^2)$$

Each weight's gradient is then the error of the neuron it feeds times the input it carries:

$$\frac{\partial L}{\partial W_2} = \frac{1}{n}\delta_2^\top h, \qquad \frac{\partial L}{\partial W_1} = \frac{1}{n}\delta_1^\top x$$

def backward(p, X, y, h, prob):
    n = len(y)
    d2 = (prob - y)[:, None]                 # (n, 1) error at the output
    gW2 = d2.T @ h / n; gb2 = d2.mean(0)
    d1 = (d2 @ p["W2"]) * (1 - h ** 2)       # (n, hidden) error at the hidden layer
    gW1 = d1.T @ X / n; gb1 = d1.mean(0)
    return {"W1": gW1, "b1": gb1, "W2": gW2, "b2": gb2}

You can check a backward pass numerically: change one weight by a tiny amount, recompute the loss, and compare the slope with the gradient. If they disagree, the backward code has a bug. This "gradient check" is how every framework's autograd was tested.

Gradient descent: take a small step downhill

With the gradients known, move every parameter a small step against its gradient:

$$w \leftarrow w - \eta \, \frac{\partial L}{\partial w}$$

The step size \(\eta\) is the learning rate. Picture walking down a valley in fog: you feel the slope under your feet and step downhill. Steps too small and you take all day; too large and you jump over the valley floor and land higher on the other side. The simulation's learning-rate slider shows both.

def train(X, y, hidden=8, lr=0.3, epochs=2000, batch=32):
    p = init(X.shape[1], hidden)
    for epoch in range(epochs):
        idx = rng.permutation(len(y))
        for s in range(0, len(y), batch):                 # mini-batch gradient descent
            b = idx[s:s + batch]
            h, prob = forward(p, X[b])
            g = backward(p, X[b], y[b], h, prob)
            for k in p:
                p[k] -= lr * g[k]
        if epoch % 200 == 0:
            _, prob = forward(p, X)
            loss = -np.mean(y * np.log(prob + 1e-9) + (1 - y) * np.log(1 - prob + 1e-9))
            print(epoch, round(loss, 4), ((prob > 0.5) == y).mean())
    return p

# two rings: class 1 inside, class 0 outside
a = rng.uniform(0, 2 * np.pi, 400); r = np.where(np.arange(400) % 2, 0.4, 0.85) + rng.normal(0, 0.05, 400)
X = np.c_[r * np.cos(a), r * np.sin(a)]; y = (np.arange(400) % 2).astype(float)
params = train(X, y)

Mini-batches, small random groups of examples, make each step cheap and add a little noise that helps escape flat regions. One pass over all the data is an epoch.

The training loop, in one paragraph

Take a batch. Run the forward pass to get predictions. Compute the loss. Run the backward pass to get a gradient for every parameter. Step every parameter against its gradient. Repeat. Every modern optimiser (momentum, Adam, AdamW) changes only the step: it remembers past gradients to move more steadily and adapts the step size per parameter.

How you know it is learning, and not memorising

Keep data the network never trains on and watch its loss too. While both fall, the network is learning a pattern. When training loss keeps falling and test loss rises, it is memorising the noise in the training set. Standard remedies:

  • More data or less noisy data.
  • A smaller network, fewer neurons or layers.
  • Regularisation, such as weight decay (penalising large weights) or dropout (randomly silencing neurons during training).
  • Early stopping: keep the weights from the epoch with the lowest test loss.

From this to a large model

A large language model uses the same four steps: forward pass, loss (cross-entropy over the next token), backpropagation and an optimiser step. The differences are scale and structure: billions of parameters instead of dozens, attention layers instead of a single hidden layer, AdamW instead of plain gradient descent, and thousands of GPUs computing the gradients of different batches in parallel. The equations in this article are the core of all of it.

Frequently asked questions

Why do we need activation functions?

Without them a deep network collapses into a single linear function, which can only draw straight boundaries. The non-linearity is what lets layers build curves and, at scale, complex features.

What is a good learning rate?

There is no universal value. Start around 0.01 to 0.1 for plain gradient descent on small problems, or 1e-3 for Adam, then try a few values ten times apart and keep the one whose loss falls fastest without jumping.

Why does my loss not go down?

The usual causes are a learning rate that is too high or too low, inputs that are not scaled, a bug in the gradients (run a gradient check), or a network too small for the pattern.

What is the difference between backpropagation and gradient descent?

Backpropagation computes the gradients. Gradient descent uses them to update the weights. Training needs both.

Is the simulation a real neural network?

Yes. It runs the forward pass, binary cross-entropy, backpropagation and mini-batch gradient descent in JavaScript in your browser, on freshly generated data, with the equations shown in this article.

Summary

Training a neural network is a loop of four steps: predict with a forward pass, measure the error with a loss, find each parameter's share of the blame with backpropagation, and move each parameter a small step downhill. Watch training and test loss together, choose the learning rate by experiment, and add capacity only until the test loss stops improving.

Sources

Continue learning

Cite this article

Kelahmet, Ö. C. (2026, November 17). Training a neural network from scratch: forward pass, backpropagation and gradient descent. Özer Cem KELAHMET. https://www.ozercemkelahmet.com/articles/train-a-neural-network-from-scratch-backpropagation-simulation

BibTeX
@online{kelahmet2026train,
  author = {Kelahmet, {\"O}zer Cem},
  title  = {{Training a neural network from scratch: forward pass, backpropagation and gradient descent}},
  year   = {2026},
  month  = {nov},
  url    = {https://www.ozercemkelahmet.com/articles/train-a-neural-network-from-scratch-backpropagation-simulation}
}

How this article connects

2 pages link to or from it. Drag to turn, click a dot to open it.

Version historyfirst version
  1. Published

Factual corrections are listed here with what changed. Wording and formatting edits are not listed.

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open