What is Bias-variance trade-off? Definition and meaning
The idea that a model's error on new data comes from being too simple (bias), too sensitive to its training sample (variance), and unavoidable noise.
Definition
Bias-variance trade-off: The idea that a model's error on new data comes from being too simple (bias), too sensitive to its training sample (variance), and unavoidable noise.
In short: Why models miss or memorise.
When it is used
Diagnosing underfitting and overfitting and deciding whether to add data, capacity or regularisation.
Where it fits
Bias-variance trade-off belongs to Training fundamentals in the Machine learning topic. These are the other terms in the same group:
- Gradient descent: Gradient descent is an iterative method that lowers a loss by repeatedly moving parameters in the direction of steepest decrease.
- Loss functions: A loss function is a formula that scores how far a model's predictions are from the targets, giving training something to minimise.
- Regularization: Any technique that limits or guides what a model learns so that it performs better on unseen data, not just on its training set.
- Evaluation metrics: Numbers that summarise how well a model's predictions match the truth, chosen to reflect what matters for the task.
- Hyperparameter tuning: The search for the training settings, such as learning rate or tree depth, that give a model its best performance on unseen data.
- Feature engineering: The craft of creating, transforming and selecting input variables so that a model can learn the pattern more easily and more reliably.
- Data preprocessing: The steps that turn raw, messy data into a clean, consistent numeric form that a model can be trained and evaluated on without leakage.
- Synthetic data: Data produced by a model, program or simulator rather than recorded from real events, used to train, test or share machine learning systems.
Learn more
The full guide to Bias-variance trade-off explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.