Random forest

Overview

A random forest is an ensemble of decision trees. Each tree is trained on a random sample of the rows, and at every split it considers only a random subset of the features. The forest predicts by averaging the trees for regression, or by taking a majority vote for classification. Because the trees are different from each other, their individual mistakes tend to cancel out, which gives a model that is accurate, stable and hard to break.

Random forests are one of the most dependable general-purpose methods for tabular data. They need little tuning, work with numeric and encoded categorical features, tolerate noise and outliers, and give useful by-products such as out-of-bag error estimates and feature importance. They are usually a bit less accurate than a tuned gradient-boosting model such as XGBoost, LightGBM or CatBoost, but they are often the safest first baseline. Tabular foundation models such as TabPFN and TabICL are routinely compared with them. See /model/benchmarks for how such comparisons are framed.

What it is

A random forest is a bagging method specialized for decision trees, with an extra layer of randomness in the choice of split features.

The core ingredients

• Decision trees as the base learner, usually grown deep with little or no pruning.
• Bagging (bootstrap aggregating): each tree is trained on a bootstrap sample, which is a random sample of rows drawn with replacement.
• Random feature subsets: at each split, only a random subset of features is considered as candidates.
• Aggregation: predictions are averaged (regression) or voted (classification).

What it is used for

• Classification and regression on structured data.
• Feature importance estimates.
• Missing value imputation through proximity or chained approaches.
• Outlier and anomaly scoring through related tree methods such as Isolation forest.
• Quick baselines before more complex models.

Where you meet it

• In scikit-learn as RandomForestClassifier and RandomForestRegressor.
• In R packages and in many distributed machine learning frameworks.
• As the rf mode of some boosting libraries.
• In AutoML tools such as PyCaret, where it is a standard candidate.

The kind of data it suits

Tables of moderate size with a mix of useful and useless features, where robustness matters more than the last fraction of accuracy.

Where it came from

Decision trees

Decision trees were developed through the 1960s to 1980s as readable rule-based models. They were popular for being easy to explain but known to be unstable: a small change in the data could produce a very different tree.

Bagging

The paper "Bagging Predictors" (1996) showed that averaging models trained on bootstrap samples reduces variance, especially for unstable learners like trees. This was the first big step: instead of trying to fix a single tree, average many.

Random subspaces and random split selection

Through the 1990s, several researchers explored adding randomness to tree building, including choosing split candidates at random and training trees on random subsets of features. These ideas aimed to make trees less similar to each other.

Random forests

The paper "Random Forests" (2001) combined bagging with random feature selection at each split, proved a bound on generalization error in terms of tree strength and tree correlation, and introduced out-of-bag estimates and variable importance as practical tools. The name and the method have been standard ever since.

Variants and neighbors

• Extremely randomized trees (extra trees) pick split thresholds at random and often train faster.
• Quantile regression forests keep leaf distributions to estimate prediction intervals.
• Isolation forests repurpose random trees for unsupervised outlier scoring.
• Gradient boosting (2001 and later) took a different route, adding trees in sequence rather than averaging independent ones. See XGBoost.

Role in benchmarks

For two decades, random forests have been a default baseline in studies of tabular learning. Work on TabNet, SAINT and FT-Transformer, and later on pre-trained models, reports results against tree ensembles. Background on that research is in /model/research.

How it works

The analogy: a panel of independent experts

Imagine asking a hundred experts to estimate the weight of an ox at a fair. Each expert has seen a slightly different set of examples and pays attention to different clues. Individually they are noisy. Averaged together, their estimate is surprisingly good. For this to work, the experts must not all make the same mistake. A random forest works hard to keep its experts different from one another.

Step by step: training

1. Choose the number of trees, B.
2. For each tree, draw a bootstrap sample. Sample N rows from the N training rows with replacement. About 63 percent of unique rows appear, and the rest (about 37 percent) are left out. Those left-out rows are called out-of-bag rows for that tree.
3. Grow a deep tree on the sample. At each node, pick a random subset of m features (the max_features setting).
4. Choose the best split among those m features by an impurity measure (Gini impurity or entropy for classification, variance reduction for regression).
5. Split and recurse until a stopping rule is met, such as a minimum leaf size or pure leaves.
6. Store the tree and move to the next one. Trees are independent, so training is easily parallel.

Step by step: prediction

1. Pass the row through every tree to reach a leaf in each.
2. For regression, average the leaf values.
3. For classification, average the class probabilities (or take a majority vote).
4. Return the result.

Why randomness helps

The error of an averaged ensemble depends on two things: how accurate each tree is, and how correlated the trees are. If every tree were identical, averaging would change nothing. By training on different samples and restricting features at each split, the forest lowers the correlation between trees. Each tree is a little weaker than a tree trained on everything, but the lower correlation more than pays for it.

Bias and variance in plain words

• A deep single tree has low bias (it can fit nearly anything) but high variance (it changes a lot with the data).
• Averaging many such trees reduces variance while keeping bias low.
• That is why random forests use deep trees, unlike boosting, which uses shallow ones.

Out-of-bag (OOB) estimate

Each row is left out of about 37 percent of the trees. For each row, average the predictions of only those trees that did not see it. The resulting error over all rows is a nearly free, nearly unbiased estimate of test error. It acts like cross-validation without extra training.

Impurity measures

Feature importance

Two common measures:

• Impurity-based (mean decrease in impurity): sum how much each feature's splits reduced impurity, averaged across trees. It is quick, but it favors features with many distinct values and can mislead.
• Permutation importance: shuffle one feature's values and see how much the score drops. It is more reliable and works with any metric, though correlated features can share credit.

Attribution tools such as SHAP and LIME provide per-prediction explanations.

Missing values and categories

• Many implementations require complete numeric input, so missing values must be imputed or flagged beforehand. Some implementations now support missing values natively.
• Categorical columns usually need encoding (one-hot, ordinal or target encoding done within folds), unless the implementation supports categories directly.

Extra trees

Extremely randomized trees draw split thresholds at random for each candidate feature and choose the best of those random splits. They often train faster and add even more diversity, sometimes at a small cost in bias.

Key concepts and terms

A worked example

A small numeric picture of averaging

Suppose we predict a house price and three trees give 200, 230 and 260 for the same house. The forest predicts the mean, 230. If the true price is 240:

• The individual errors are 40, 10 and 20.
• The forest error is 10.

Averaging does not always beat the best single tree, but it beats the typical one, and it removes the need to know in advance which tree is best.

Why the correlation matters, with numbers

If each tree has prediction variance 100, and the average correlation between trees is rho, the variance of the average of B trees is roughly rho * 100 + (1 - rho) * 100 / B.

Lower correlation lowers the floor on variance. Adding more trees alone cannot get below rho * 100. This is why random feature selection at each split is the heart of the method.

A realistic Python example

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import make_pipeline
from sklearn.compose import make_column_transformer
from sklearn.preprocessing import OrdinalEncoder
from sklearn.impute import SimpleImputer

df = pd.read_csv("patients.csv")
y = df.pop("readmitted")

cat_cols = list(df.select_dtypes("object").columns)
num_cols = [c for c in df.columns if c not in cat_cols]

prep = make_column_transformer(
    (OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1), cat_cols),
    (SimpleImputer(strategy="median", add_indicator=True), num_cols),
)

rf = RandomForestClassifier(
    n_estimators=500,
    max_features="sqrt",
    min_samples_leaf=3,
    oob_score=True,
    n_jobs=-1,
    class_weight="balanced_subsample",
    random_state=0,
)

X_train, X_test, y_train, y_test = train_test_split(
    df, y, test_size=0.2, stratify=y, random_state=0
)

model = make_pipeline(prep, rf)
model.fit(X_train, y_train)

print("OOB score:", rf.oob_score_)
print("Test AUC:", roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]))

Reading the result

• The OOB score is a built-in estimate of accuracy, close to what cross-validation would say.
• Missing numbers are filled with the median, and an indicator column keeps the fact that they were missing.
• class_weight="balanced_subsample" helps with rare classes.
• Five hundred trees is usually enough that more trees make little difference.

Permutation importance

from sklearn.inspection import permutation_importance

result = permutation_importance(
    model, X_test, y_test, scoring="roc_auc", n_repeats=10, random_state=0, n_jobs=-1
)
for name, score in sorted(zip(X_test.columns, result.importances_mean),
                          key=lambda t: -t[1])[:10]:
    print(f"{name:25s} {score:.4f}")

Regression with prediction intervals from the trees

import numpy as np
from sklearn.ensemble import RandomForestRegressor

reg = RandomForestRegressor(n_estimators=500, min_samples_leaf=5, n_jobs=-1, random_state=0)
reg.fit(X_train_num, y_train_reg)

all_preds = np.stack([t.predict(X_test_num) for t in reg.estimators_])
mean = all_preds.mean(axis=0)
lo, hi = np.percentile(all_preds, [5, 95], axis=0)

The spread of tree predictions gives a rough sense of uncertainty. For proper intervals, use a quantile-aware method such as quantile regression forests or conformal techniques.

Tuning with a search tool

import optuna
from sklearn.model_selection import cross_val_score

def objective(trial):
    clf = RandomForestClassifier(
        n_estimators=300,
        max_features=trial.suggest_float("max_features", 0.1, 1.0),
        min_samples_leaf=trial.suggest_int("min_samples_leaf", 1, 20),
        max_depth=trial.suggest_int("max_depth", 4, 40),
        n_jobs=-1, random_state=0,
    )
    return cross_val_score(clf, X_enc, y_train, cv=5, scoring="roc_auc").mean()

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=40)

Optuna and Bayesian optimization both fit this job, although random forests are forgiving enough that a small grid often suffices.

Strengths and limitations

Strengths

• Strong out of the box. Defaults often give a competitive score.
• Robust to noise and outliers. Averaging dampens individual errors.
• Low overfitting risk as trees increase. Adding trees does not make the forest overfit more, it just stabilizes.
• No feature scaling needed. Splits depend on order, not magnitude.
• Captures nonlinearities and interactions without manual feature design.
• Built-in validation. The OOB estimate saves a validation split.
• Parallel training. Trees are independent.
• Useful by-products. Feature importance, proximities and leaf-based explanations.
• Works on small data better than many neural methods.

Limitations

• Often less accurate than tuned boosting on structured data.
• Large models. Hundreds of deep trees can use a lot of memory and slow predictions.
• No extrapolation. Predictions stay within the range of training targets.
• Biased importance. Impurity importance favors high-cardinality features.
• Weak on very sparse, high-dimensional data like raw text counts.
• Less interpretable than a single tree.
• Unscaled probabilities. Output probabilities can be poorly calibrated.
• Per-table training. There is no knowledge transfer between datasets.
• Imbalanced data needs care. Class weights or balanced sampling are usually required.

How it compares with alternatives

A common practical order is: run a random forest for a fast robust baseline, then a boosting model for accuracy, and a foundation model as a second opinion on small tables.

Role in tabular foundation models

The robust baseline

A tabular foundation model is pre-trained on many tables so that it can predict on a new table, often by passing labeled rows as context and making one forward pass. A random forest is the opposite: it is fit from scratch to each new table. It is the simplest strong per-table learner, so it often serves as the sanity check. If a foundation model cannot beat a random forest on a table, something is wrong with the model, the data format or the evaluation. For how comparisons are set up, see /model/benchmarks and /model/research.

Raw-table properties side by side

A shared idea: ensembles and averaging

Tabular foundation models borrow the spirit of random forests in practice. Many of them average predictions over several random permutations of columns, feature subsets or context samples, which reduces variance in the same way that averaging trees does. In-context models often run multiple "views" of the same table, for example with different column orders or different subsets of training rows as context, and then combine the outputs. The rationale is the same: diverse, correlated-but-not-identical predictors average to something more stable. Read about design choices in /model/architecture.

Tree structure as a pre-training prior

Models such as TabPFN are pre-trained on synthetic tables drawn from random generative processes. Those processes include step-like, threshold-like and interaction patterns that trees capture naturally. A random forest has to rediscover those patterns on every table. A foundation model has seen them across millions of generated tables, so it brings them as a prior. For small tables, that prior is a large advantage.

In-context learning and nearest-neighbor-like behavior

A random forest can be viewed as an adaptive nearest-neighbor method: two rows are "close" if they often land in the same leaves. In-context learning with attention has a similar flavor: a query row attends strongly to training rows that resemble it and borrows their labels. The difference is that the forest learns its notion of similarity from one table, while the foundation model learns a general notion of similarity from many.

Imputation

Random forests are a long-standing tool for imputing missing values. One method fits a forest for each column with gaps, using the other columns as inputs, and iterates until values settle. Another uses leaf proximities to borrow values from similar rows. Foundation models can impute directly, because they are pre-trained to reconstruct masked cells in numeric and categorical columns alike. See Imputation networks and Masked cell modeling. A fair evaluation masks known cells, imputes with each method and compares errors.

Anomaly detection

A plain random forest is supervised, but its relatives are not. Isolation forest builds random trees to isolate points, and rows that are isolated in few splits are flagged as anomalies. Proximity-based outlier scores from a supervised forest are another option. Foundation models offer a third route, scoring how surprising a row or cell is given the rest of the table. See Anomaly detection.

Semantic and sector classification

For classifying a record or a column into a semantic group such as an industry sector, a forest needs features, for example token counts of names, value statistics or hashed text features. It does this reliably when labeled examples are plentiful. Foundation models that embed column names and values together can use meaning directly and need fewer labels. See Semantic sector heads.

Continual learning

A random forest is not designed for incremental updates. You can add new trees trained on new data, or retrain the whole forest, and some online variants exist. Adding trees keeps old trees unchanged, so older behavior persists, but the forest can become stale or large. Neural foundation models use methods like EWC and experience replay to learn new tasks without forgetting old ones.

A practical pairing

• Use a random forest as the first, quick, robust baseline.
• Use the OOB score as a free estimate while you build features.
• Compare with boosting and with a foundation model on the same splits.
• Consider averaging forest and foundation-model predictions, which often reduces error.

Practical guidance and best practices

A solid workflow

1. Define the metric and a split that matches production use.
2. Encode categories and handle missing values.
3. Fit a forest with 300 to 1,000 trees and the OOB score on.
4. Check permutation importance for leaks and useless features.
5. Tune max_features, min_samples_leaf and max_depth modestly.
6. Calibrate probabilities if you need them as real probabilities.
7. Evaluate once on a held-out test set.
8. Record settings and metrics with a tracker such as MLflow.

Parameter guide

Imbalanced classes

• Use class weights, or balanced bootstrap sampling.
• Watch AUC-PR and recall at fixed precision, not accuracy.
• Choose the decision threshold on validation data.

Model size and speed

• Raise min_samples_leaf to shrink trees.
• Limit max_depth for faster prediction.
• Use fewer trees if latency matters, and check the score curve.
• Convert to a compact format for serving when available.

Handling many categories

• Use ordinal or target encoding inside cross-validation folds for high-cardinality columns.
• Avoid one-hot encoding of very high-cardinality columns, because it dilutes max_features.

Probability calibration

Forest probabilities are averages of leaf frequencies, so they tend to be pulled away from 0 and 1. Use isotonic or sigmoid calibration on a held-out set when probabilities will be used for decisions.

Diagnosing problems

Common mistakes

• Trusting impurity importance alone. It favors features with many distinct values. Cross-check with permutation importance or SHAP.
• Using too few trees. Results can be noisy below roughly a hundred trees.
• Tuning for too many hours. Forests respond little to extensive tuning, so spend time on features and splits instead.
• Leaking the target. Features built from the label or from the future inflate scores.
• Random splits on time series. Use forward-looking splits.
• Ignoring imbalance. Accuracy can look great while the minority class is missed.
• Reading raw probabilities as calibrated. Calibrate if probabilities matter.
• Expecting extrapolation. Forests cannot predict beyond the range of training targets.
• One-hot encoding high-cardinality columns. This wastes feature draws and slows training.
• Using OOB estimates when rows are grouped. If rows from one entity appear in both bootstrap and OOB sets, the estimate is optimistic.
• Comparing a tuned boosting model with an untuned forest, or the reverse. Give each a fair tuning budget.
• Skipping a simple baseline. A linear model shows whether trees add real value.

Frequently asked questions

What is a random forest in simple terms?

A random forest is a group of decision trees that each make a prediction, with the final answer taken as the average or the majority vote. Every tree sees a slightly different random sample of the data and a random subset of features at each split, so the trees disagree in useful ways and their errors cancel out.

How is a random forest different from gradient boosting?

A random forest trains many deep trees independently and averages them, which mainly reduces variance. Gradient boosting trains shallow trees one after another, each correcting the previous errors, which reduces bias as well. Boosting is usually more accurate when tuned, and a forest is usually easier and more forgiving.

How many trees do I need?

A few hundred is usually enough. Adding more trees does not cause overfitting, it only stabilizes the result, but gains flatten out. Plot the out-of-bag score against the number of trees and stop where the curve levels off.

What is out-of-bag error?

Each tree is trained on a bootstrap sample that leaves out roughly a third of the rows. For every row, you average the predictions of only the trees that did not see it, and the error across all rows becomes the out-of-bag estimate. It works like cross-validation at almost no extra cost.

Does a random forest need feature scaling?

No. Splits depend on the ordering of values, so scaling or monotonic transformations do not change the trees. You still need to encode categories and, in many implementations, handle missing values.

Can random forests overfit?

Adding trees does not increase overfitting, but each deep tree can fit noise, and the forest as a whole can still overfit if leaves are tiny and features are noisy. Raising the minimum leaf size, lowering the number of features per split and limiting depth are the main controls.

How reliable is random forest feature importance?

The default impurity-based importance is quick but biased toward features with many distinct values and can mislead with correlated features. Permutation importance on held-out data is more trustworthy, and attribution methods such as SHAP add per-prediction detail.

Is a random forest better than a neural network for tabular data?

On many tabular problems, a random forest or boosted trees match or beat standard neural networks, particularly on small to medium data. Pre-trained tabular models are closing the gap and can win on small tables. The best choice depends on data size, feature types and tuning effort.

Can random forests handle missing values and categorical features?

Some implementations handle them natively, but many require you to impute missing values and encode categories first. A good practice is to add a missing indicator along with imputation, and to use ordinal or target encoding inside cross-validation for high-cardinality categories.

How does a random forest relate to tabular foundation models?

A random forest is a per-table learner and a dependable baseline, while a tabular foundation model is pre-trained across many tables and predicts in context. Foundation models are compared with forests to check they add real value, and they borrow the idea of averaging over diverse views to reduce variance.

Glossary

• Ensemble: a combination of several models whose outputs are merged.
• Bagging: training models on bootstrap samples and averaging them.
• Bootstrap sample: a random sample of rows drawn with replacement.
• Out-of-bag error: an error estimate computed from rows each tree did not see.
• Decision tree: a model that splits data by feature thresholds into groups.
• Impurity: a measure of how mixed the labels in a node are.
• Gini impurity: a common impurity measure for classification.
• Variance reduction: a split quality measure for regression.
• Permutation importance: the drop in score after shuffling a feature.
• Extra trees: trees that choose split thresholds at random.
• Calibration: agreement between predicted probabilities and observed frequencies.
• Proximity: how often two rows share leaves across trees.
• Overfitting: fitting noise so that new data is predicted poorly.
• In-context learning: predicting from labeled examples placed in the model input.
• Tabular foundation model: a network pre-trained on many tables.

Summary

A random forest averages many decorrelated decision trees, each trained on a bootstrap sample of rows and a random subset of features at every split. This design lowers variance while keeping bias low, giving a model that is accurate, robust, easy to use and equipped with a free validation estimate through out-of-bag rows. Its limits are lower peak accuracy than tuned boosting, large model size, no extrapolation and uncalibrated probabilities.

In the world of tabular foundation models, the random forest is the dependable baseline: fit per table, little tuning, hard to break. Pre-trained models such as TabPFN and TabICL are judged against it, and they echo its core idea by averaging over diverse views to reduce variance. Use a forest early for a quick answer, compare it with XGBoost, LightGBM and CatBoost for accuracy, and bring in a foundation model when data is small or setup time is short. Related reading: /model/architecture and /model/benchmarks.

Related

It belongs to Classical ML in my stack, under Foundation models and applied ML. Others in the same group: