Özer Cem KELAHMETKELAHMET
Topics · Deep learning

Generalisation: overfitting, regularisation, normalisation and honest evaluation

A model is only useful on data it has never seen. This page covers why networks overfit and the techniques that keep them general.

Overview

Large networks can memorise their training data, noise included. Generalisation is the gap between training performance and performance on new data, and most of the craft of deep learning is keeping that gap small without making the model too weak.

Signs of overfitting

Training loss keeps falling while validation loss rises; the model is very confident on wrong answers; performance collapses on data from a slightly different source. The overfitting simulation shows the gap opening as model capacity grows. See also the bias-variance trade-off.

The techniques

Technique How it helps Read more
More and more varied data The best regulariser Data preprocessing
Data augmentation Creates realistic variations Synthetic data
Weight decay Penalises large weights Regularization
Dropout Randomly drops units so none dominates Dropout
Normalisation layers Stabilise training, smooth the loss surface Normalization layers
Early stopping Stop when validation stops improving
Transfer learning Start from a model trained on more data Transfer learning
Ensembles Average several models' errors Ensemble learning

Normalisation

Batch normalisation rescales activations across a batch; layer normalisation and RMSNorm rescale within each example and are standard in transformers. They let deeper networks train with higher learning rates. Try the normalisation and dropout simulation.

Honest evaluation

Split data into training, validation and test sets before any modelling, and split the way the model will be used: by time for forecasting, by user or patient for personal data. Use the test set once. Cross-validation gives steadier estimates on small data. For large models, check benchmarks for contamination; the evaluation simulation shows how leaked test items inflate scores.

Large models generalise differently

Very large models trained on enormous data often keep improving with more parameters, even past the point where classical theory expects overfitting. Their failures move from memorising the training set to gaps in what the data covered, which is why data quality and fresh evaluation sets matter so much.

Frequently asked questions

How much dropout should I use?

Values between 0.1 and 0.3 are common for medium models; very large pre-trained language models often use little or none.

Is a bigger model always worse at generalising?

No. With enough data and regularisation, bigger models usually generalise better. Overfitting is about capacity relative to data.

Summary

Use more varied data, regularise with weight decay, dropout and normalisation, stop early, and evaluate on splits that match real use. The deep learning essentials path puts these together.

Back to the topic

This page is part of Deep learning, which maps every concept, tool, simulation and article on the subject.

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open