Overfitting and regularisation: when a model learns the noise
A model that is too simple misses the pattern; one that is too complex memorises the noise. Move the complexity slider and watch the two errors part ways, the most important picture in machine learning.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
Blue points are the training data, amber rings are fresh test points from the same noisy curve, and the dashed line is the true function. The white curve is a polynomial fitted by least squares; λ adds an L2 penalty on the coefficients. The chart shows training and test error for every degree.
Things to try
- Go from degree 1 to 15 and find where test error is lowest.
- At degree 12, raise λ to 0.01: the wild curve calms down.
- At degree 12, switch to 40 training points: more data also tames the curve.
- Draw new samples: a complex model changes a lot from sample to sample; that variance is overfitting.
How it works in real systems
Every real training run fights this. Teams hold out a validation set, stop training when validation loss stops improving (early stopping), use weight decay, dropout and data augmentation, and judge models only on data they never trained on.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Make a small noisy data set
Twenty points from a smooth curve plus noise: few enough that a flexible model can memorise them.
import numpy as np
rng = np.random.default_rng(1)
x = np.sort(rng.uniform(-1, 1, 20)); y = np.sin(2.5 * x) + rng.normal(0, 0.2, 20)
x_val = np.sort(rng.uniform(-1, 1, 200)); y_val = np.sin(2.5 * x_val) + rng.normal(0, 0.2, 200)
Step 2 · Fit polynomials of growing degree
PolynomialFeatures creates x, x², …, xᵈ; a linear model on top of them is a polynomial.
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.metrics import mean_squared_error
for d in [1, 3, 5, 9, 15]:
m = make_pipeline(PolynomialFeatures(d), StandardScaler(), LinearRegression()).fit(x[:, None], y)
tr = mean_squared_error(y, m.predict(x[:, None]))
va = mean_squared_error(y_val, m.predict(x_val[:, None]))
print(f"degree {d:2d} train {tr:.3f} validation {va:.3f}")
Training error keeps falling with the degree; validation error falls, then rises. The gap is overfitting.
Step 3 · Regularise with ridge (L2)
Ridge adds α·‖w‖² to the loss, which keeps the coefficients small and the curve smooth. Pick α on the validation set, never on the training set.
for alpha in [1e-6, 1e-3, 1e-2, 1e-1, 1]:
m = make_pipeline(PolynomialFeatures(15), StandardScaler(), Ridge(alpha=alpha)).fit(x[:, None], y)
print(f"alpha {alpha:<6} validation {mean_squared_error(y_val, m.predict(x_val[:, None])):.3f}")
Step 4 · Do it properly with cross-validation
With only 20 points, a single split is noisy. GridSearchCV averages over folds.
from sklearn.model_selection import GridSearchCV
pipe = make_pipeline(PolynomialFeatures(), StandardScaler(), Ridge())
grid = GridSearchCV(pipe, {"polynomialfeatures__degree": [1, 3, 5, 9, 15], "ridge__alpha": [1e-4, 1e-2, 1]},
cv=5, scoring="neg_mean_squared_error").fit(x[:, None], y)
print(grid.best_params_, -grid.best_score_)
In neural networks the same ideas appear as weight decay, dropout, data augmentation and early stopping.