Bias-variance trade-off
The bias-variance trade-off explains why models underfit or overfit, how expected error splits into parts, and how learning curves help you diagnose and fix it.
Overview
Every predictive model makes mistakes on new data, and those mistakes have different causes. Some models are too simple to capture the real pattern: they make the same kind of error again and again, whatever data you train them on. That is bias. Other models are so flexible that they chase the particular sample they were trained on: train them on a slightly different sample and their predictions change a lot. That is variance. The bias-variance trade-off is the observation that making a model more flexible usually lowers bias but raises variance, and the best model on new data sits somewhere in between.
This idea is the backbone of practical machine learning. It explains why a perfect training score is a warning sign, why more data helps some models more than others, why regularisation works, and why averaging many models in an ensemble can beat any single one. It also gives a diagnostic routine: compare training and validation error, look at how they change with data size and model complexity, and you can usually tell which side of the trade-off you are on.
An everyday analogy is archery. A beginner whose arrows cluster tightly but far from the bullseye has high bias and low variance: consistent, but consistently wrong, perhaps because the sight is misaligned. A beginner whose arrows land all over the target, centred on the bullseye on average, has low bias and high variance. A good archer has both low: a tight cluster in the middle. Training a model is like adjusting the bow, and you can only judge the adjustment by shooting at a fresh target.
What it is
| Concept | What it means | Typical symptom |
|---|---|---|
| Bias | Systematic error from wrong or too-simple assumptions | High training error and high validation error, close together |
| Variance | Sensitivity of the fitted model to the particular training sample | Low training error, much higher validation error |
| Irreducible error | Noise in the target that no model can predict | A floor that error cannot go below |
| Underfitting | The model is too simple for the pattern | Both errors high |
| Overfitting | The model has learnt noise as if it were signal | Large gap between training and validation error |
| Model capacity | How rich a set of functions a model can represent | More parameters, deeper trees, higher-degree polynomials |
| Generalisation gap | Validation error minus training error | Grows as variance grows |
The trade-off is about expected error on data drawn from the same distribution as the training data. It does not cover distribution shift, label errors in the test set or leakage, which need separate treatment. It is also a statement about averages: any single training run is one draw from a distribution of possible fitted models.
For squared error the split into bias, variance and noise is exact. For classification losses such as 0-1 loss there are several competing decompositions, and they do not all add up as neatly. The intuition carries over, but the clean formula belongs to regression with squared error.
Where it came from
The decomposition of mean squared error into a squared bias and a variance is a classical result of statistical estimation theory, long used to compare estimators. Its wide use as a lens on learning machines comes from Stuart Geman, Elie Bienenstock and René Doursat, whose 1992 paper "Neural networks and the bias/variance dilemma" in Neural Computation explained that flexible, non-parametric learners pay for low bias with high variance and need large samples or constraints. Ridge regression's 1970 argument by Hoerl and Kennard, that a little bias can buy a large reduction in variance, is the same idea applied to linear models.
Throughout the 1990s and 2000s the trade-off shaped how models were built: choose capacity by validation, regularise, and use ensembles such as bagging and random forests to cut variance. Then deep learning complicated the picture. Huge networks with far more parameters than training examples could fit random labels perfectly and still generalise well on real ones.
In 2019 Mikhail Belkin, Daniel Hsu, Siyuan Ma and Soumik Mandal published "Reconciling modern machine-learning practice and the classical bias–variance trade-off" in PNAS. They described a "double descent" curve: test error first follows the classic U shape, peaks near the point where the model can just fit the training data exactly, and then falls again as capacity grows further. Preetum Nakkiran and colleagues at OpenAI and Harvard extended this in "Deep double descent: where bigger models and more data hurt" (arXiv 2019, ICLR 2020), showing the effect across model size, training time and even dataset size in modern deep networks.
How it works
The decomposition in plain words
Imagine retraining the same kind of model many times, each time on a fresh training set of the same size drawn from the same source. For a fixed input x you would get a spread of predictions. Their average tells you about bias: how far the typical prediction is from the truth. Their spread tells you about variance: how much the prediction depends on which sample you happened to get. The target itself also has noise. For squared error these three parts add up exactly.
E[(y − f̂(x))²] = (E[f̂(x)] − f(x))² + E[(f̂(x) − E[f̂(x)])²] + σ²
\_____ bias² _____/ \_____ variance _____/ noise
Here y is the observed target at input x, f(x) is the true underlying function, f̂(x) is the model's prediction after training on a random training set, E[·] is the average over all possible training sets (and over the noise for the left side), and σ² is the variance of the noise around f(x), which no model can remove. The left side is the expected squared error on new data.
How capacity moves the parts
As you increase capacity, for example by raising the degree of a polynomial, deepening a decision tree or reducing k in k-nearest neighbours, the average prediction gets closer to the truth, so bias falls. At the same time the model starts responding to the individual noise of each sample, so variance rises. Their sum, plus the constant noise, traces a U shape: high on the left from bias, high on the right from variance, lowest in the middle. Regularisation slides you along this curve without changing the model family: a stronger penalty acts like lower capacity.
Some knobs have an obvious direction. In k-nearest neighbours, small k means low bias and high variance; large k means the reverse. In ridge regression, a large penalty means high bias and low variance. In boosting, more trees lower bias and eventually raise variance, which is why early stopping matters. Averaging many high-variance models, as in bagging, lowers variance while leaving bias roughly unchanged.
Learning curves
A learning curve plots training and validation error against the number of training examples. With few examples, training error is low (it is easy to fit a handful of points) and validation error is high. As data grows, training error rises and validation error falls.
- High bias: both curves flatten early at a high error, close together. More data will not help much; you need a more flexible model or better features.
- High variance: a wide gap remains between low training error and higher validation error, and validation error is still falling at the right edge. More data, stronger regularisation or a simpler model should help.
scikit-learn computes these with learning_curve and plots them with LearningCurveDisplay in sklearn.model_selection.
Validation curves
A validation curve plots training and validation score against one hyperparameter, such as tree depth or the penalty strength. On the low-capacity side both scores are poor; on the high-capacity side the training score keeps improving while the validation score drops. The peak of the validation curve is the setting to choose. scikit-learn offers validation_curve and ValidationCurveDisplay. Repeating this over several hyperparameters at once is hyperparameter tuning.
Double descent in plain words
The classic U curve assumes that once a model can fit the training data perfectly, adding capacity only adds variance. Double descent shows this is not the whole story. Near the "interpolation threshold", where a model has just enough capacity to fit every training point, it is forced into a contorted, high-variance solution, and test error peaks. Beyond that point, there are many ways to fit the data exactly, and training methods such as gradient descent tend to pick smooth, small-norm solutions among them. Test error can then fall again, sometimes below the bottom of the first U.
Practical consequences are modest but real. A model at the threshold can look much worse than a smaller or a much larger one. More data can briefly hurt when it pushes a model across the threshold. And regularisation, early stopping or simply going larger usually smooths the peak away. The trade-off is not wrong; it describes what happens below the threshold, and implicit regularisation explains what happens beyond it.
Diagnosis and fixes
| Symptom | Likely cause | Things to try |
|---|---|---|
| Training and validation error both high | High bias | More capacity, better features, less regularisation, train longer |
| Training error low, validation error much higher | High variance | More data, stronger regularisation, augmentation, simpler model, ensembling |
| Validation error far above training error from the start, unstable across folds | Small or noisy validation set | Use cross-validation, repeat with several seeds |
| Validation error good, test error bad | Leakage or tuning to the validation set | Check data preprocessing for leakage, use nested cross-validation |
| Both errors near a floor that does not move | Irreducible noise | Better labels or new information, not a new model |
Key concepts and terms
| Term | Meaning |
|---|---|
| Bias | Average error of the model's predictions relative to the truth |
| Variance | How much predictions change with a different training sample |
| Irreducible error | Noise in the target that sets a lower bound on error |
| Capacity | The richness of the functions a model can represent |
| Underfitting | Too little capacity or too much regularisation |
| Overfitting | Fitting noise; low training error with poor generalisation |
| Generalisation gap | Difference between training and validation error |
| Learning curve | Errors plotted against training-set size |
| Validation curve | Errors plotted against one hyperparameter |
| Interpolation threshold | The capacity at which a model can just fit the training data exactly |
| Double descent | Test error that falls, rises near the threshold, then falls again |
| Implicit regularisation | The tendency of training methods to prefer simple solutions without an explicit penalty |
A worked example
This is an illustrative scenario. An energy analyst wants to predict a building's hourly electricity use from outdoor temperature. There are 200 hours of training data and 200 for validation. The true relationship is U-shaped: heating at low temperatures, cooling at high ones.
Degree 1 (a straight line). Training RMSE 9.8 kWh, validation RMSE 10.1 kWh. The two numbers are close and both high. A straight line cannot bend, so it misses the U. This is high bias.
Degree 2 (a parabola). Training RMSE 4.1, validation RMSE 4.3. Both drop sharply and stay close. This matches the true shape.
Degree 12. Training RMSE 3.2, validation RMSE 7.9. The curve wiggles to pass near every training point, including noisy ones, and swings wildly at the extremes of temperature. This is high variance.
Learning curve for degree 12. With 50 training points the gap is enormous; with 2,000 points it shrinks to 4.0 versus 4.6. More data tames variance, but the simpler degree-2 model got there with a tenth of the data.
Fix with regularisation. Keeping degree 12 but adding a ridge penalty chosen by five-fold cross-validation gives training RMSE 4.0 and validation RMSE 4.4. The penalty has pulled the flexible model back towards the simpler shape. The analyst reports the degree-2 model for its clarity and notes that the noise floor, about 3.9 kWh, comes from occupancy that temperature cannot explain.
Strengths and limitations
Strengths:
- Gives a simple, general language for why models fail on new data.
- Turns into concrete diagnostics: learning curves and validation curves.
- Guides decisions about data collection, capacity, regularisation and ensembling.
- Exact and quantitative for squared-error regression.
- Explains why bagging and averaging work.
Limitations:
- The exact decomposition holds for squared error; classification versions are less tidy.
- Bias and variance cannot be measured directly from one dataset; they are estimated by resampling.
- The classic U shape does not describe heavily over-parameterised models, where double descent appears.
- It assumes training and test data come from the same distribution; it says nothing about drift.
- It treats noise as fixed, while in practice better data collection can reduce it.
How it compares with alternatives
| Lens | What it explains | How it relates |
|---|---|---|
| Bias-variance trade-off | Error from simplicity versus sensitivity | The everyday working model |
| Statistical learning theory (VC dimension, Rademacher complexity) | Worst-case bounds on the generalisation gap | Formalises capacity; bounds are often loose in practice |
| Double descent and implicit regularisation | Why huge models still generalise | Extends the trade-off beyond the interpolation threshold |
| Approximation, estimation and optimisation error | Splits error by cause in deep learning | Adds the error from imperfect training to the picture |
| Bayesian view | Priors as regularisers, posterior uncertainty | Bias comes from the prior, variance from limited data; see probability |
Practical guidance and best practices
- Always compare training and validation scores side by side; the gap tells you more than either number alone.
- Plot a learning curve before collecting more data, to check whether more data would actually help.
- Plot a validation curve for the one or two hyperparameters that control capacity most.
- Start with a simple baseline, such as linear regression or logistic regression, so you know the bias floor of a simple model.
- When variance is the problem, try regularisation and data augmentation before shrinking the model.
- When bias is the problem, improve features or switch to a more flexible family such as gradient boosting.
- Use cross-validation with several folds or seeds so that variance in your measurement is not mistaken for variance in the model.
- Keep a final test set untouched until the end, so the last estimate is honest.
Common mistakes
- Treating a near-perfect training score as success instead of as a warning.
- Collecting more data for a model that is limited by bias, where more data will not help.
- Diagnosing from a single train-validation split that is too small to trust.
- Tuning hyperparameters on the test set, then reporting that same set as the final score.
- Assuming a bigger network always overfits more, without checking for double-descent behaviour or the effect of regularisation.
- Confusing distribution shift with variance: a model that fails on next year's data may be fine on today's.
Frequently asked questions
Can a model have both high bias and high variance?
Yes. A model can be too simple in the wrong way and still unstable, for example a shallow tree trained on very little data with an unlucky choice of split feature. In practice you see both errors high and also large differences between folds. The fixes then combine better features or a more suitable model with more data or averaging.
Does more data always reduce error?
More data reduces variance and almost never hurts a well-regularised model. It does little for bias: a straight line stays a straight line however many points you give it. Learning curves show which case you are in. Near the interpolation threshold, adding a little data can briefly raise test error, which is one of the effects described as deep double descent.
How is this related to regularisation?
Regularisation is the main tool for moving along the trade-off without changing the model family. A stronger penalty, a lower dropout keep-rate or a shallower tree raises bias and lowers variance. The best strength is the one that minimises validation error, which is the bottom of the U for that knob.
Why do ensembles help?
Averaging the predictions of several models that make partly independent errors reduces variance, while the average bias stays about the same. Bagging and random forests rely on this. Boosting works differently: it mainly reduces bias by adding models that correct earlier mistakes, and needs shrinkage and early stopping to keep variance in check. See ensemble learning.
Does double descent mean overfitting is not real?
No. Overfitting is very real for models near the interpolation threshold and for models trained without any implicit or explicit regularisation. Double descent shows that very large models trained with methods such as gradient descent can generalise well despite fitting the training data, because the training method prefers smooth solutions. You still need validation to know where your model sits.
How do I estimate bias and variance in practice?
Use resampling. Train the model on many bootstrap samples or cross-validation folds, predict on a fixed evaluation set, and compute the average prediction and the spread of predictions for each point. The spread estimates variance. The gap between the average prediction and the target, minus an estimate of noise, estimates bias. Most practitioners rely on learning and validation curves instead, because they answer the practical question directly.
Glossary
- Bias: systematic error from a model that is too simple or wrongly specified.
- Variance: the sensitivity of a fitted model to the specific training sample.
- Noise: random variation in the target that no model can predict.
- Capacity: how complex a set of functions a model can represent.
- Learning curve: a plot of training and validation error against training-set size.
- Validation curve: a plot of training and validation error against a hyperparameter.
- Interpolation: fitting every training example exactly.
- Double descent: a second fall in test error after the interpolation threshold.
- Bagging: training models on bootstrap samples and averaging them to reduce variance.
Summary
The bias-variance trade-off says that expected error on new data has three parts: bias from a model that is too simple, variance from a model that is too sensitive to its particular training sample, and irreducible noise. For squared error the three add up exactly. Increasing capacity usually trades bias for variance, giving the familiar U-shaped test error, while regularisation, more data and ensembling reduce variance. Learning curves and validation curves turn the idea into a routine: compare training and validation error, see how they change with data and with capacity, and pick the fix that matches the symptom. Very large models add a twist called double descent, where test error falls again beyond the point of perfect fit, but the core lesson remains: judge a model by data it has not seen.
Related
It belongs to Training fundamentals in my stack, under Machine learning. Others in the same group: