Ensemble learning

Overview

Ensemble learning means training several models and combining their predictions instead of trusting a single one. The combined prediction is usually more accurate and more stable than any member on its own. The idea is old and simple, but it sits behind some of the strongest tools in practical machine learning, including random forests and gradient boosting libraries such as XGBoost, LightGBM and CatBoost.

Ensembles work because different models make different mistakes. If the mistakes are not perfectly correlated, averaging them cancels part of the noise while the shared signal remains. Some ensembles mainly reduce variance (the jumpiness of a model that changes a lot when the training data changes). Others mainly reduce bias (systematic under-fitting). A few do both.

An everyday picture: ask one friend to guess the number of sweets in a jar and the answer may be far off. Ask fifty people and take the average, and the result is often surprisingly close. Each person is wrong in their own way, and the errors partly cancel. That only works if people guess independently. If they all copy the loudest person, fifty guesses are no better than one. Diversity is the whole point.

What it is

Two design questions define every ensemble. First, how do you make the members different? You can change the data each one sees (bagging), the features (random subspaces), the algorithm (stacking, voting), the random seed (deep ensembles), or the target each one aims at (boosting). Second, how do you combine them? Options run from a plain average to learned weights and a full second-level model.

Ensembles are a strategy, not a single algorithm. That is why they appear across classical supervised learning, deep learning and even anomaly detection, where methods such as Isolation Forest are ensembles of random trees.

Where it came from

The statistical intuition goes back to the "wisdom of crowds" and to classical results on averaging independent estimates. In machine learning, the modern story starts around 1990. Lars Kai Hansen and Peter Salamon showed in 1990 that ensembles of neural networks generalise better than single networks. In the same year Robert Schapire's paper "The Strength of Weak Learnability" proved that a learner only slightly better than chance can be "boosted" into an accurate one, answering a question raised by Michael Kearns and Leslie Valiant.

David Wolpert introduced stacked generalisation in 1992. Leo Breiman published bagging ("bootstrap aggregating") in 1996, and also a paper on stacked regressions the same year. Yoav Freund and Robert Schapire introduced AdaBoost in 1995, with the journal version in 1997; the work later earned them the Gödel Prize. Tin Kam Ho described random decision forests in 1995, and Breiman's 2001 paper "Random Forests" combined bagging with random feature selection into the form used today.

Jerome Friedman's "Greedy Function Approximation: A Gradient Boosting Machine" (2001) recast boosting as gradient descent in function space, which opened the door to any differentiable loss. Fast, regularised implementations followed: XGBoost (Chen and Guestrin, 2016), LightGBM (Ke and colleagues, 2017) and CatBoost (Prokhorenkova and colleagues, 2018). The Netflix Prize, won in 2009, was famously decided by a blend of hundreds of models.

For neural networks, Huang and colleagues proposed snapshot ensembles in 2017 (collecting several models from one training run), and Lakshminarayanan, Pritzel and Blundell showed the same year that simple deep ensembles give strong uncertainty estimates.

How it works

Why averaging helps

Suppose you average M models whose errors each have variance σ² and an average pairwise correlation ρ. The variance of the average is:

Var(average) = ρ · σ² + ((1 − ρ) / M) · σ²

Bagging / averaging prediction:   ŷ(x) = (1/M) · Σ_m f_m(x)
Boosting prediction:              F_M(x) = F_0(x) + Σ_m ν · h_m(x)
Gradient boosting target:         r_i = − ∂L(y_i, F(x_i)) / ∂F(x_i)
Ensemble ambiguity (squared loss): error(ensemble) = mean error(members) − mean spread(members)

Here σ² is the error variance of one model, ρ the average correlation between members' errors, and M the number of members. In the prediction formulas, f_m is the m-th member, F_0 is a starting guess (such as the mean), h_m is the m-th weak learner, ν is the learning rate (shrinkage), L is the loss function, and r_i are the pseudo-residuals the next learner tries to predict. The last line, the ambiguity decomposition of Krogh and Vedelsby, says an averaging ensemble is better than its average member by exactly how much the members disagree. The first line shows the key lesson: as M grows, the second term shrinks to zero, but the first term ρ·σ² stays. More members cannot remove correlated error. Only diversity can.

Bagging

Bagging draws M bootstrap samples (random samples of the training set, with replacement, the same size as the original). Each sample leaves out roughly a third of the rows on average. A model is trained on each, and predictions are averaged for regression or voted for classification. The left-out rows give a free "out-of-bag" estimate of error. Bagging helps most with unstable, high-variance learners such as deep decision trees. It helps little with stable models such as linear regression.

Random forests

A random forest is bagging with one more source of randomness: at each split, a tree may only choose among a random subset of the features. This stops every tree from splitting on the same dominant feature, so the trees become less correlated, which lowers the ρ term above. Forests are robust, need little tuning and rarely overfit badly as more trees are added.

Boosting

Boosting builds models one after another. AdaBoost increases the weight of misclassified examples so the next weak learner focuses on them, then gives each learner a vote proportional to its accuracy. Gradient boosting generalises this: each new tree is fitted to the negative gradient of the loss (for squared error, simply the residuals), and a small learning rate scales its contribution. Modern libraries add regularisation, second-order gradient information, histogram-based split finding and careful handling of categorical features and missing values. Boosting is usually the strongest out-of-the-box choice for tabular data, but it can overfit if run for too many rounds, so early stopping on a validation set matters.

Stacking and blending

Stacking trains several different base models, then trains a meta-model on their predictions. The crucial detail is that the meta-model must see out-of-fold predictions: each base model predicts rows it was not trained on, using cross-validation. Otherwise the meta-model learns to trust whichever base model memorised the training set. Blending is a simpler variant that uses one hold-out split instead of full cross-validation. It is faster but wastes data and is noisier.

Voting

Voting combines independently trained models without a learned meta-model. Hard voting takes the majority label. Soft voting averages predicted probabilities, which usually works better when the members are reasonably calibrated.

Deep and snapshot ensembles

A deep ensemble trains several copies of a neural network from different random initialisations and averages their outputs. Different starting points lead to different solutions, which gives genuine diversity. Snapshot ensembles save checkpoints at several low points of a cyclic learning rate schedule, giving several models for the price of one run. Related cheap tricks include averaging weights along a training run and Monte Carlo dropout, which treats dropout at test time as sampling from many sub-networks.

Uncertainty from ensembles

When members agree, the ensemble is confident. When they disagree, the spread is a signal of uncertainty. For regression, the variance across members estimates how unsure the model is about a point. For classification, disagreement can be measured with the entropy of the averaged probabilities or with mutual information between the prediction and the member identity (see information theory). Disagreement grows on inputs unlike the training data, which makes ensembles a practical tool for spotting out-of-distribution cases. They are one of the simplest routes to the goals of Bayesian deep learning.

Key concepts and terms

A worked example

Imagine an illustrative loan-default dataset with 50,000 rows and 30 features, and a target of whether a loan defaulted within a year. You hold out 20 per cent as a test set and use five-fold cross-validation on the rest.

1. A single decision tree, depth 12, reaches a test ROC AUC of about 0.71. It looks much better on training data, a sign of high variance.
2. A random forest of 500 trees, each splitting on a random subset of features, lifts AUC to about 0.80. The out-of-bag estimate agrees closely with cross-validation.
3. A gradient-boosted model with learning rate 0.05, depth 6 and early stopping (it stops at 820 rounds) reaches about 0.83.
4. You also train a logistic regression on scaled features (about 0.78) and a small neural network (about 0.80).
5. For stacking, you collect five-fold out-of-fold probabilities from the forest, the boosted model, the logistic regression and the network, and train a logistic regression meta-model on those four columns. Test AUC reaches about 0.84.

The gain from step 3 to step 5 is small, because the strongest member already captures most of the signal. That is typical. The bigger benefits are often stability across data refreshes and better calibrated probabilities. The numbers here are illustrative, not results from a real dataset.

A minimal scikit-learn version of steps 2 to 5 looks like this:

from sklearn.ensemble import (RandomForestClassifier,
                              HistGradientBoostingClassifier,
                              StackingClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

base = [
    ("rf", RandomForestClassifier(n_estimators=500, n_jobs=-1, random_state=0)),
    ("gb", HistGradientBoostingClassifier(learning_rate=0.05, early_stopping=True)),
    ("lr", make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))),
]
stack = StackingClassifier(estimators=base,
                           final_estimator=LogisticRegression(),
                           stack_method="predict_proba", cv=5)
stack.fit(X_train, y_train)
proba = stack.predict_proba(X_test)[:, 1]

StackingClassifier builds the out-of-fold predictions internally using the cv folds, which avoids the most common stacking leak.

Strengths and limitations

Strengths:

• Usually more accurate than any single member, especially for high-variance learners.
• More stable: small changes in data move the prediction less.
• Tree ensembles handle mixed feature types, non-linearities and interactions with little feature engineering.
• Disagreement between members gives a cheap, useful uncertainty signal.
• Bagging and random forests are easy to parallelise and hard to break.

Limitations:

• Training and inference cost grow with the number of members.
• Harder to explain than one small model; tools such as SHAP help but add work.
• Gains vanish when members are highly correlated.
• Stacking adds pipeline complexity and is easy to leak if out-of-fold predictions are not used.
• Boosting can overfit noisy labels if run too long without early stopping.
• Large ensembles are heavy to deploy and monitor; teams often distil them into a smaller model with knowledge distillation.

How it compares with alternatives

A single large model trained well can sometimes match an ensemble. The question is whether the extra accuracy and stability are worth the extra cost and complexity in your setting.

Practical guidance and best practices

1. Start with one strong baseline, usually a gradient-boosted model with early stopping, before building anything bigger.
2. Use a random forest when you want a robust model with almost no tuning, or as a sanity check.
3. Measure diversity: if two members' predictions correlate above about 0.95, adding the second rarely helps.
4. Prefer different model families (trees, linear, neural) over many copies of the same one when stacking.
5. Always build stacking features from out-of-fold predictions, never from in-sample predictions.
6. Keep the meta-model simple, such as a regularised logistic or linear regression; complex meta-models overfit.
7. Tune members with proper hyperparameter tuning on validation folds, then retrain on the full training set.
8. Check calibration of the final probabilities and recalibrate if needed.
9. Report the gain over the best single member; if it is tiny, ship the single model.
10. Budget inference latency and memory before committing to a large ensemble in production.

Common mistakes

• Training the stacking meta-model on predictions the base models made on their own training rows.
• Adding many near-identical models and expecting a gain that only diversity can give.
• Tuning the ensemble on the test set, which turns the test score into an optimistic guess.
• Running boosting for a fixed large number of rounds with no early stopping.
• Treating ensemble disagreement as calibrated probability without checking it.
• Forgetting that every member needs the same preprocessing at inference time.

Frequently asked questions

Do more trees in a random forest ever hurt?

Adding trees to a random forest does not cause overfitting in the usual sense; the error curve flattens rather than rising. Extra trees only cost time and memory. Past a few hundred trees, gains are usually tiny, so pick a number where the out-of-bag error has levelled off.

Is boosting always better than bagging?

No. Boosting often wins on clean tabular data because it reduces bias as well as variance. Bagging and random forests are more forgiving of noisy labels and need less tuning. On a new problem it is worth trying both, which takes little effort with modern libraries.

What is the difference between stacking and blending?

Both train a meta-model on base model predictions. Stacking produces those predictions with cross-validation, so every training row gets an out-of-fold prediction. Blending uses one hold-out split. Stacking uses data more fully; blending is quicker and simpler.

Can I ensemble neural networks cheaply?

Yes. Snapshot ensembles collect several checkpoints from one run with a cyclic learning rate. Monte Carlo dropout samples many sub-networks at test time. Weight averaging along the training path gives a single model that behaves like a small ensemble. Full deep ensembles cost more but are usually the most reliable.

How do ensembles give uncertainty?

Each member makes its own prediction. The spread between them tells you how much the members disagree, which rises in regions with little training data. You can report that spread directly or turn it into intervals, but check that the intervals are well calibrated on held-out data.

Are gradient-boosted trees still state of the art for tables?

They remain among the strongest and most practical choices, especially on larger tables. Recent benchmarks also show that tuned deep models, cross-model ensembles and tabular foundation models can match or beat them in some settings, particularly smaller datasets.

Glossary

• Bagging: training models on bootstrap samples and averaging them.
• Boosting: training models in sequence, each correcting the previous ones.
• Stacking: combining different models with a learned meta-model.
• Voting: combining models by majority label or averaged probability.
• Random subspace: giving each model or split a random subset of features.
• Pseudo-residual: the negative gradient of the loss that a boosting step fits.
• Snapshot ensemble: several checkpoints from one training run used as members.
• Deep ensemble: several independently initialised neural networks averaged together.
• Ambiguity: the average disagreement of members around the ensemble prediction.
• Out-of-fold: predicted by a model that did not train on that row.

Summary

Ensemble learning combines several models so that their individual mistakes partly cancel. Bagging and random forests reduce variance by averaging models trained on resampled data and features. Boosting reduces bias by adding models that fix the remaining errors, and modern gradient-boosting libraries make it the default for tabular work. Stacking, blending and voting combine different model families, and deep and snapshot ensembles bring the same benefits to neural networks, along with a practical measure of uncertainty. The central rule is diversity: more members help only when they err differently, and every gain should be weighed against the extra cost of training, serving and explaining the result. For the trade-off behind all of this, see the bias-variance trade-off.

Related

It belongs to Classic models in my stack, under Machine learning. Others in the same group: