Neural networks

Overview

A neural network is a model built from many small, simple units called neurons, arranged in layers. Each neuron takes some numbers in, multiplies each by a weight, adds them up with a bias, and passes the result through a non-linear activation function. The outputs of one layer become the inputs of the next. The weights start out random. Training adjusts them, a little at a time, so that the network's outputs match the examples it is shown.

An everyday picture is a large office that processes applications. The front desk notes basic facts. The next team combines those facts into simple judgements, such as "income looks stable". A later team combines those judgements into bigger ones, such as "low risk". The final manager makes the decision. Nobody writes the rules down. Instead, every time a decision turns out wrong, each team slightly changes how much attention it pays to each input from the team before. Over thousands of cases, the office gets good at the job. That feedback process is training.

This page goes from the single perceptron to the multilayer perceptron, explains neurons, weights, biases and layers, walks through the forward pass and the full training loop, puts the universal approximation theorem into plain words, and gives an overview of the main families: MLPs, CNNs, recurrent networks, Transformers and graph neural networks.

What it is

A neural network is a function with many adjustable numbers. Its building blocks are few.

A network with more than one hidden layer is usually called "deep", and training such networks is the subject of deep learning. The number of weights is the parameter count. A small tabular MLP might have tens of thousands of parameters; large language models have billions.

Where it came from

In 1943 Warren McCulloch and Walter Pitts proposed a mathematical model of a neuron as a threshold unit. In 1958 Frank Rosenblatt introduced the perceptron, a single layer of such units with a rule for learning its weights from examples. In 1969 Marvin Minsky and Seymour Papert's book "Perceptrons" set out the limits of single-layer perceptrons, for example that they cannot compute XOR, and interest in the field fell.

Multilayer networks needed a way to train hidden layers. The 1986 Nature paper by David Rumelhart, Geoffrey Hinton and Ronald Williams popularised backpropagation for this. In 1989 George Cybenko proved that a network with one hidden layer of sigmoid units can approximate any continuous function on a bounded region, and Kurt Hornik extended such results in 1991. Also in 1989 Yann LeCun and colleagues applied backpropagation to convolutional networks for handwritten digits, work that led to LeNet-5 in 1998. Jeffrey Elman's simple recurrent network appeared in 1990, and Sepp Hochreiter and Jürgen Schmidhuber published the LSTM in 1997.

In 2006 Hinton, Simon Osindero and Yee-Whye Teh showed that deep belief networks could be trained layer by layer, reviving interest in depth. In 2012 AlexNet, by Alex Krizhevsky, Ilya Sutskever and Hinton, won the ImageNet challenge by a wide margin using GPUs and ReLU, and deep learning took off. Graph neural networks were formalised by Franco Scarselli and colleagues in 2009, with graph convolutional networks by Thomas Kipf and Max Welling following in 2016. In 2017 Ashish Vaswani and colleagues introduced the Transformer in "Attention Is All You Need", the architecture behind most current language models.

How it works

One neuron and one layer

neuron:   a = f( w₁x₁ + w₂x₂ + … + wₙxₙ + b ) = f( w · x + b )
layer:    h = f( W x + b )
network:  ŷ = f₃( W₃ f₂( W₂ f₁( W₁ x + b₁ ) + b₂ ) + b₃ )
update:   W ← W − η · ∂L/∂W

Here x is the input vector with components x₁ … xₙ, w the neuron's weight vector, b its bias, f the activation function and a the neuron's output. In the layer form, W is a matrix with one row of weights per neuron and h is the vector of outputs. The network line shows three layers applied in turn, producing the prediction ŷ. In the update, L is the loss, ∂L/∂W its gradient with respect to the weights and η the learning rate.

From perceptron to multilayer perceptron

A single perceptron draws one straight line (a hyperplane) through the input space and labels one side 1 and the other 0. That is enough for linearly separable problems but not for XOR, where the classes sit on opposite corners of a square. Add one hidden layer with two units and the problem becomes easy: each hidden unit draws its own line, and the output unit combines them. That is the multilayer perceptron (MLP), also called a fully connected or dense feed-forward network. Each extra layer can build features out of the previous layer's features.

The forward pass

To make a prediction, data flow forwards. The input vector is multiplied by the first weight matrix, the bias is added, and the activation is applied. The result goes into the next layer, and so on, until the output layer. For classification the final scores are usually turned into probabilities by a sigmoid or softmax. In practice, a whole batch of examples is processed at once by stacking them as rows of a matrix, which is what makes GPUs so effective.

The training loop

Training repeats the same small cycle many thousands of times.

1. Shuffle and batch. Split the training data into mini-batches, often 32 to 1,024 examples.
2. Forward pass. Run the batch through the network to get predictions.
3. Loss. Compare predictions with targets using a loss function, giving one number for the batch.
4. Backward pass. Use backpropagation to compute the gradient of the loss with respect to every weight and bias.
5. Update. Let the optimiser move each weight a small step against its gradient, as in gradient descent.
6. Repeat. Move to the next batch. One full pass over the data is an epoch; training usually runs for several to hundreds of epochs.
7. Validate. After each epoch, measure the loss and metrics on held-out validation data, and stop when they stop improving.

The starting weights matter: they must be random so neurons differ, and scaled so signals neither explode nor vanish (see weight initialization). The learning rate usually changes during training according to a learning rate schedule.

import torch
from torch import nn

model = nn.Sequential(
    nn.Linear(20, 64), nn.ReLU(),
    nn.Linear(64, 64), nn.ReLU(),
    nn.Linear(64, 3),              # 3 class logits
)
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.AdamW(model.parameters(), lr=1e-3)

for epoch in range(20):
    model.train()
    for x, y in train_loader:
        opt.zero_grad()
        loss = loss_fn(model(x), y)   # forward pass + loss
        loss.backward()               # backward pass
        opt.step()                    # update

Universal approximation in plain words

The universal approximation theorem says that a network with a single hidden layer, enough neurons and a suitable non-linear activation can get as close as you like to any continuous function on a bounded range of inputs. In plain words: there is no smooth shape a neural network is fundamentally unable to draw. It does not say how many neurons you need (possibly a huge number), that training will find the right weights, or that the result will generalise to new data. Depth helps in practice because many functions can be built far more efficiently by composing simple pieces layer on layer than by one enormous flat layer.

Why they generalise, and when they do not

A network with more parameters than training examples can memorise its data. It generalises when the data are plentiful and representative, and when training is guided by regularization such as weight decay, dropout, data augmentation and early stopping. The bias-variance trade-off still applies, though very large networks often behave better than classical theory would predict.

Key concepts and terms

A worked example

An illustrative network learns XOR: output 1 when exactly one of two binary inputs is 1. Use two hidden ReLU units and one linear output. The following weights solve it.

• Hidden unit 1: h₁ = ReLU(x₁ + x₂).
• Hidden unit 2: h₂ = ReLU(x₁ + x₂ − 1).
• Output: ŷ = h₁ − 2·h₂.

Check the four cases.

No single linear unit can do this, but two hidden units and one combination can. In training, the network would not be given these weights. It would start from small random values, compute outputs for the four examples, measure squared error, backpropagate, and adjust. After a few hundred updates with a reasonable learning rate it typically lands on weights that behave like these, possibly scaled or with the hidden units swapped. Occasionally a run gets stuck (for example if a ReLU unit dies), which is one reason practitioners use several random seeds and slightly wider layers than the minimum.

Scale this up and the same mechanics train a tabular model with 20 input columns, two hidden layers of 64 units and three output classes, as in the code above. It has (20×64 + 64) + (64×64 + 64) + (64×3 + 3) = 1,344 + 4,160 + 195 = 5,699 parameters.

Strengths and limitations

Strengths:

• Learn features automatically from raw data such as pixels, audio and text.
• Scale with data and compute; larger models trained on more data often keep improving.
• Flexible: the same training loop works for classification, regression, generation and embeddings.
• Specialised architectures encode useful structure (locality, order, relationships).
• Pre-trained networks transfer to new tasks with little labelled data through transfer learning.

Limitations:

• Data-hungry; small datasets often favour tree ensembles or linear models.
• Compute- and energy-intensive to train at scale.
• Many hyperparameters; results can be sensitive to tuning and random seeds.
• Hard to interpret; individual weights rarely have clear meaning.
• Can be confidently wrong on inputs unlike the training data.

How it compares with alternatives

Within neural networks, the main families differ in their built-in assumptions.

Outside neural networks, tree ensembles such as gradient boosting remain very strong on many tabular problems, while neural tabular foundation models are narrowing that gap.

Practical guidance and best practices

1. Start with a simple baseline (a linear model or small MLP) and only add complexity if it clearly helps.
2. Standardise numeric inputs and encode categorical ones before training.
3. Use well-tested defaults: ReLU or GELU activations, He or framework-default initialisation, AdamW with a modest learning rate.
4. Hold out validation data and track it every epoch; use early stopping.
5. Regularise with weight decay and dropout before making the network smaller.
6. Overfit a tiny subset first (a few batches). If the network cannot drive training loss near zero there, something is broken.
7. Choose the architecture family that matches the structure of your data.
8. Fix random seeds and log hyperparameters so runs can be compared and reproduced.
9. Tune the learning rate first; it matters more than most other settings.

Common mistakes

• Feeding unscaled features with very different ranges, which slows or destabilises training.
• Leaking validation or test data into training through preprocessing done on the whole dataset.
• Forgetting to switch between model.train() and model.eval(), which changes dropout and batch-norm behaviour.
• Making the network deeper or wider when the real problem is the data or the learning rate.
• Judging a model on training loss alone.
• Assuming the universal approximation theorem means any network will learn any task.

Frequently asked questions

How is a neural network different from the brain?

The name comes from a loose analogy with biological neurons, which receive signals and fire. Artificial neurons are just weighted sums with a simple non-linear function, trained by calculus rather than by biological processes. The analogy helped early thinking, but modern networks are best understood as mathematical functions.

How many layers and neurons should I use?

There is no formula. For tabular data, two to four hidden layers of 64 to 512 units is a common starting point. For images, text and audio, start from an established architecture. Then adjust using validation results, ideally with systematic hyperparameter tuning.

What is the difference between an epoch, a batch and an iteration?

A batch is the group of examples used for one update. An iteration is one update. An epoch is one pass over the whole training set. With 10,000 examples and a batch size of 100, one epoch is 100 iterations.

Do neural networks always beat other models?

No. On large unstructured data (images, audio, text) they are usually best. On small or medium tabular datasets, gradient-boosted trees are often as good or better and much cheaper to tune. The right choice depends on data size, structure and the cost of errors.

What does the universal approximation theorem really guarantee?

Only that a suitable network exists for any continuous target on a bounded domain. It says nothing about how large it must be, whether training will find it, or whether it will generalise. It explains why networks are flexible, not why they work well in practice.

Why are GPUs used for neural networks?

Because the core operations are large matrix multiplications that can be split into thousands of independent multiply-add operations. GPUs are designed to run such operations in parallel, often making training many times faster than on a CPU.

Glossary

• Neuron: a unit computing a weighted sum plus bias, followed by an activation.
• Weight: a learned multiplier on one input.
• Bias: a learned constant added before the activation.
• Layer: a group of neurons reading the same inputs.
• Perceptron: an early single-layer neural classifier with a step activation.
• Multilayer perceptron: a feed-forward network with one or more hidden dense layers.
• Forward pass: computing outputs from inputs through the layers.
• Backward pass: computing gradients from the loss to every parameter.
• Epoch: one full pass over the training data.
• Universal approximation: the result that wide enough networks can approximate any continuous function on a bounded region.

Summary

A neural network is a stack of layers of simple units, each computing a weighted sum plus a bias and a non-linear activation. The single perceptron can only draw straight boundaries; adding hidden layers creates the multilayer perceptron, which can represent far richer functions, as the universal approximation theorem guarantees in principle. Training repeats a simple loop: batch the data, run the forward pass, compute a loss, backpropagate gradients and update the weights, over many epochs while watching validation results. Specialised families build in assumptions about the data: CNNs for local patterns, recurrent networks for sequences, Transformers for flexible relationships through attention, and GNNs for graphs. Understanding the basic unit and the training loop explains all of them.

Related

It belongs to Neural network basics in my stack, under Deep learning. Others in the same group: