Özer Cem KELAHMETKELAHMET
AI Simulations · Neural network basics

Backpropagation, one number at a time

Backpropagation is how a network finds out which weights to blame. Follow one training example through a 2-2-1 network and see every number the algorithm computes.

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

Inputs on the left, two tanh neurons, a sigmoid output and the loss on the right. Each press of Next runs one stage: the forward pass fills in the values, the loss compares them with the target, the backward pass writes a gradient on every weight, and the update moves each weight against its gradient.

Things to try

  • Press Next four times and check one gradient by hand: error at the output times the input it carries.
  • Run 50 updates and watch the loss curve drop toward zero.
  • Notice that the hidden errors are multiplied by 1 − h²: that is the slope of tanh.

How it works in real systems

Frameworks such as PyTorch build this graph automatically for millions of operations and call the backward pass with one line, loss.backward(). The rules are exactly the ones on screen: the chain rule, applied from the loss back to every weight.

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · A tiny autograd engine

Each value remembers how it was made, and how to send gradient back to its inputs.

import math
class V:
    def __init__(self, data, parents=(), back=lambda: None):
        self.data, self.grad, self._parents, self._back = data, 0.0, parents, back
    def __add__(self, o):
        o = o if isinstance(o, V) else V(o); out = V(self.data + o.data, (self, o))
        def back(): self.grad += out.grad; o.grad += out.grad
        out._back = back; return out
    def __mul__(self, o):
        o = o if isinstance(o, V) else V(o); out = V(self.data * o.data, (self, o))
        def back(): self.grad += o.data * out.grad; o.grad += self.data * out.grad
        out._back = back; return out
    def tanh(self):
        t = math.tanh(self.data); out = V(t, (self,))
        def back(): self.grad += (1 - t * t) * out.grad
        out._back = back; return out
    def backward(self):
        order, seen = [], set()
        def visit(v):
            if v not in seen:
                seen.add(v); [visit(p) for p in v._parents]; order.append(v)
        visit(self); self.grad = 1.0
        for v in reversed(order): v._back()

Step 2 · Run it on one neuron and check the numbers

x1, x2, w1, w2, b = V(2.0), V(0.0), V(-3.0), V(1.0), V(6.88)
out = (x1 * w1 + x2 * w2 + b).tanh()
out.backward()
print(out.data, w1.grad, w2.grad, b.grad)

Step 3 · Confirm with PyTorch

import torch
x1, x2 = torch.tensor(2.0), torch.tensor(0.0)
w1, w2, b = (torch.tensor(v, requires_grad=True) for v in (-3.0, 1.0, 6.88))
o = torch.tanh(x1 * w1 + x2 * w2 + b); o.backward()
print(o.item(), w1.grad.item(), w2.grad.item(), b.grad.item())   # same numbers

A finite-difference check, (f(w+ε) − f(w−ε)) / 2ε, is the classic way to test a hand-written backward pass.

More simulations

← Loss functions · All AI Simulations · Optimisers →

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open