# Overfitting and regularisation: when a model learns the noise

A model that is too simple misses the pattern; one that is too complex memorises the noise. Move the complexity slider and watch the two errors part ways, the most important picture in machine learning.

<div data-sim="overfit"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

Blue points are the training data, amber rings are fresh test points from the same noisy curve, and the dashed line is the true function. The white curve is a polynomial fitted by least squares; λ adds an L2 penalty on the coefficients. The chart shows training and test error for every degree.

## Things to try

- Go from degree 1 to 15 and find where test error is lowest.
- At degree 12, raise λ to 0.01: the wild curve calms down.
- At degree 12, switch to 40 training points: more data also tames the curve.
- Draw new samples: a complex model changes a lot from sample to sample; that variance is overfitting.

## How it works in real systems

Every real training run fights this. Teams hold out a validation set, stop training when validation loss stops improving (early stopping), use weight decay, dropout and data augmentation, and judge models only on data they never trained on.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Make a small noisy data set

Twenty points from a smooth curve plus noise: few enough that a flexible model can memorise them.

```python
import numpy as np
rng = np.random.default_rng(1)
x = np.sort(rng.uniform(-1, 1, 20)); y = np.sin(2.5 * x) + rng.normal(0, 0.2, 20)
x_val = np.sort(rng.uniform(-1, 1, 200)); y_val = np.sin(2.5 * x_val) + rng.normal(0, 0.2, 200)
```

### Step 2 · Fit polynomials of growing degree

`PolynomialFeatures` creates x, x², …, xᵈ; a linear model on top of them is a polynomial.

```python
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.metrics import mean_squared_error

for d in [1, 3, 5, 9, 15]:
    m = make_pipeline(PolynomialFeatures(d), StandardScaler(), LinearRegression()).fit(x[:, None], y)
    tr = mean_squared_error(y, m.predict(x[:, None]))
    va = mean_squared_error(y_val, m.predict(x_val[:, None]))
    print(f"degree {d:2d}  train {tr:.3f}  validation {va:.3f}")
```

Training error keeps falling with the degree; validation error falls, then rises. The gap is overfitting.

### Step 3 · Regularise with ridge (L2)

Ridge adds α·‖w‖² to the loss, which keeps the coefficients small and the curve smooth. Pick α on the validation set, never on the training set.

```python
for alpha in [1e-6, 1e-3, 1e-2, 1e-1, 1]:
    m = make_pipeline(PolynomialFeatures(15), StandardScaler(), Ridge(alpha=alpha)).fit(x[:, None], y)
    print(f"alpha {alpha:<6}  validation {mean_squared_error(y_val, m.predict(x_val[:, None])):.3f}")
```

### Step 4 · Do it properly with cross-validation

With only 20 points, a single split is noisy. `GridSearchCV` averages over folds.

```python
from sklearn.model_selection import GridSearchCV
pipe = make_pipeline(PolynomialFeatures(), StandardScaler(), Ridge())
grid = GridSearchCV(pipe, {"polynomialfeatures__degree": [1, 3, 5, 9, 15], "ridge__alpha": [1e-4, 1e-2, 1]},
                    cv=5, scoring="neg_mean_squared_error").fit(x[:, None], y)
print(grid.best_params_, -grid.best_score_)
```

In neural networks the same ideas appear as weight decay, dropout, data augmentation and early stopping.

<div class="cl-end"></div>

## More simulations

[← Network playground](/ai-simulations/neural-network-playground) · [All AI Simulations](/ai-simulations) · [k-means →](/ai-simulations/k-means-clustering)
