Neural network playground: watch every neuron learn
Pick a dataset, set the layers and neurons, and press play. Every neuron tile shows the pattern it has learned over the input plane, so you can see straight lines in the first layer combine into curves and spirals in the next.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The network has two inputs (the x and y of a point), hidden layers of tanh, ReLU or sigmoid neurons, and one sigmoid output: the probability of class 1. Training runs mini-batch gradient descent on binary cross-entropy. The lines between tiles are the weights: amber positive, blue negative, thicker is larger.
Things to try
- Spiral with one hidden layer of two neurons: it cannot be solved. Add layers and neurons until it can.
- Switch to Linear activation: whatever the depth, the boundary stays a straight line.
- Raise the noise and lower the training share: the test loss rises while the training loss keeps falling.
- Add L2 regularisation and watch the weights get thinner and the boundary smoother.
How it works in real systems
Real networks have millions of these neurons and learn from images, text or tables instead of two numbers, but each neuron still computes a weighted sum and an activation, and training still nudges every weight downhill on the loss.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Generate the two-moons data
Everything here is plain NumPy, so you can see every line of a neural network.
import numpy as np
from sklearn.datasets import make_moons
X, y = make_moons(n_samples=400, noise=0.15, random_state=0) # X: (400, 2), y: 0 or 1
y = y.reshape(-1, 1).astype(float)
Step 2 · Initialise a 2-8-8-1 network
He initialisation for the ReLU layers, small random weights, zero biases.
rng = np.random.default_rng(0)
sizes = [2, 8, 8, 1]
W = [rng.normal(0, np.sqrt(2 / m), (m, n)) for m, n in zip(sizes[:-1], sizes[1:])]
b = [np.zeros((1, n)) for n in sizes[1:]]
relu = lambda z: np.maximum(0, z)
sigmoid = lambda z: 1 / (1 + np.exp(-z))
Step 3 · Forward and backward pass
The forward pass stores every layer's output; the backward pass applies the chain rule from the loss back to each weight. With a sigmoid output and binary cross-entropy, the gradient at the output is simply p − y.
def forward(X):
acts, zs = [X], []
for i, (w, bb) in enumerate(zip(W, b)):
z = acts[-1] @ w + bb
zs.append(z)
acts.append(sigmoid(z) if i == len(W) - 1 else relu(z))
return acts, zs
def backward(acts, zs, y):
n = len(y)
delta = (acts[-1] - y) / n # dLoss/dz at the output
gW, gb = [], []
for i in reversed(range(len(W))):
gW.insert(0, acts[i].T @ delta)
gb.insert(0, delta.sum(0, keepdims=True))
if i:
delta = (delta @ W[i].T) * (zs[i - 1] > 0) # ReLU derivative is 0 or 1
return gW, gb
Step 4 · Train it and draw what each neuron learned
Plain gradient descent. Afterwards, evaluate a hidden neuron on a grid: that picture is the tile the simulation shows for the neuron.
lr = 0.5
for step in range(3001):
acts, zs = forward(X)
p = acts[-1]
loss = -np.mean(y * np.log(p + 1e-9) + (1 - y) * np.log(1 - p + 1e-9))
gW, gb = backward(acts, zs, y)
for i in range(len(W)):
W[i] -= lr * gW[i]; b[i] -= lr * gb[i]
if step % 500 == 0:
print(step, round(loss, 4), "acc", ((p > 0.5) == y).mean())
import matplotlib.pyplot as plt
xx, yy = np.meshgrid(np.linspace(-1.5, 2.5, 200), np.linspace(-1, 1.5, 200))
grid = np.c_[xx.ravel(), yy.ravel()]
acts, _ = forward(grid)
plt.contourf(xx, yy, acts[2][:, 0].reshape(xx.shape), 30, cmap="Blues") # neuron 0 of hidden layer 2
plt.scatter(X[:, 0], X[:, 1], c=y.ravel(), s=8, cmap="gray"); plt.show()