Backpropagation, one number at a time
Backpropagation is how a network finds out which weights to blame. Follow one training example through a 2-2-1 network and see every number the algorithm computes.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
Inputs on the left, two tanh neurons, a sigmoid output and the loss on the right. Each press of Next runs one stage: the forward pass fills in the values, the loss compares them with the target, the backward pass writes a gradient on every weight, and the update moves each weight against its gradient.
Things to try
- Press Next four times and check one gradient by hand: error at the output times the input it carries.
- Run 50 updates and watch the loss curve drop toward zero.
- Notice that the hidden errors are multiplied by 1 − h²: that is the slope of tanh.
How it works in real systems
Frameworks such as PyTorch build this graph automatically for millions of operations and call the backward pass with one line, loss.backward(). The rules are exactly the ones on screen: the chain rule, applied from the loss back to every weight.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · A tiny autograd engine
Each value remembers how it was made, and how to send gradient back to its inputs.
import math
class V:
def __init__(self, data, parents=(), back=lambda: None):
self.data, self.grad, self._parents, self._back = data, 0.0, parents, back
def __add__(self, o):
o = o if isinstance(o, V) else V(o); out = V(self.data + o.data, (self, o))
def back(): self.grad += out.grad; o.grad += out.grad
out._back = back; return out
def __mul__(self, o):
o = o if isinstance(o, V) else V(o); out = V(self.data * o.data, (self, o))
def back(): self.grad += o.data * out.grad; o.grad += self.data * out.grad
out._back = back; return out
def tanh(self):
t = math.tanh(self.data); out = V(t, (self,))
def back(): self.grad += (1 - t * t) * out.grad
out._back = back; return out
def backward(self):
order, seen = [], set()
def visit(v):
if v not in seen:
seen.add(v); [visit(p) for p in v._parents]; order.append(v)
visit(self); self.grad = 1.0
for v in reversed(order): v._back()
Step 2 · Run it on one neuron and check the numbers
x1, x2, w1, w2, b = V(2.0), V(0.0), V(-3.0), V(1.0), V(6.88)
out = (x1 * w1 + x2 * w2 + b).tanh()
out.backward()
print(out.data, w1.grad, w2.grad, b.grad)
Step 3 · Confirm with PyTorch
import torch
x1, x2 = torch.tensor(2.0), torch.tensor(0.0)
w1, w2, b = (torch.tensor(v, requires_grad=True) for v in (-3.0, 1.0, 6.88))
o = torch.tanh(x1 * w1 + x2 * w2 + b); o.backward()
print(o.item(), w1.grad.item(), w2.grad.item(), b.grad.item()) # same numbers
A finite-difference check, (f(w+ε) − f(w−ε)) / 2ε, is the classic way to test a hand-written backward pass.