XGBoost

Overview

XGBoost is a gradient-boosting library that builds a strong predictor from many small decision trees, adding one tree at a time so that each new tree corrects the mistakes of the ones before it. It adds explicit regularization, second-order gradient information, sparsity-aware split finding and a fast, parallel implementation. For years it has been the default first model to try on spreadsheet-style data, and it remains a hard baseline to beat.

On tabular problems such as churn, credit risk, demand forecasting or fraud scoring, XGBoost often matches or beats far more complex neural networks, and it trains in seconds to minutes on a laptop. It handles missing values natively, copes with mixed numeric features, needs little preprocessing, and exposes many knobs for control. Newer tabular foundation models, such as TabPFN and TabICL, are now measured against tuned XGBoost, which makes it the reference point for the whole field. See /model/benchmarks for how such comparisons are framed.

What it is

XGBoost stands for eXtreme Gradient Boosting. It is an open-source library, not a single algorithm. Under the hood it implements a particular flavor of gradient-boosted decision trees (GBDT).

The one-sentence definition

A GBDT model is a sum of many shallow decision trees, trained in sequence, where each tree fits the gradient of the loss with respect to the current predictions.

What the library gives you

• A core training engine written in C++ with bindings for Python, R, Java, Scala, and other languages.
• A scikit-learn style Python wrapper (XGBClassifier, XGBRegressor, XGBRanker) and a lower-level native API (xgboost.train with DMatrix).
• Support for classification, regression, ranking, survival analysis and custom objectives.
• Multi-core CPU training and GPU training.
• Distributed training on Spark, Dask and other frameworks.
• Built-in handling of missing values, sparse inputs and early stopping.
• Model inspection tools: feature importance, tree dumps and compatibility with SHAP.

What kind of data it suits

XGBoost is designed for structured, tabular data: rows are examples, columns are features. Features can be numbers, flags, or categories that have been encoded. It is not designed for raw images, audio or free text, though it can work on features extracted from them.

Where it came from

The story has several layers.

Decision trees and boosting

• Decision trees were used for decades as simple, readable models, but a single tree is unstable and overfits.
• Boosting arrived in the 1990s. The AdaBoost algorithm (1997) showed that many weak learners, each focused on the examples the previous ones got wrong, could form a very strong learner.
• The paper "Greedy Function Approximation: A Gradient Boosting Machine" (2001) recast boosting as gradient descent in function space. That gave boosting a general recipe: pick any differentiable loss, compute its gradient, fit a tree to it, repeat.

The library

• XGBoost began as a research project in the early 2010s and was released as open source around 2014.
• It became famous in data science competitions, where it won or placed highly in a large share of structured-data contests.
• The paper "XGBoost: A Scalable Tree Boosting System" (2016) described its core ideas: a regularized objective, a sparsity-aware split algorithm, a weighted quantile sketch for approximate splits, and cache-aware, out-of-core system design.

What came after

• LightGBM (2017) and CatBoost (2018) introduced their own speed and categorical-feature ideas. Over time XGBoost adopted some of these ideas, such as histogram-based training and leaf-wise growth options.
• Benchmarks of deep learning on tabular data, including work on FT-Transformer, SAINT and TabNet, repeatedly used tuned GBDT models as the bar to clear.
• Pre-trained tabular foundation models appeared in the 2020s, and they are again compared against XGBoost. Background on that line of work is in /model/research.

How it works

The analogy: a team of editors

Imagine a first draft of an essay. A first editor makes the biggest corrections. A second editor reads the corrected version and fixes what is still wrong. A third editor polishes what remains. No single editor is brilliant, but each one only has to fix a small remaining problem. After a hundred editors, the essay is very good.

In XGBoost, each "editor" is a small decision tree, and the "essay" is the prediction for every row.

Step by step

1. Start with a constant. The model begins with a base prediction, such as the average target for regression or the log-odds of the positive class for classification.
2. Measure the error. For every training row, compute how wrong the current prediction is, using a loss function (squared error, log loss, and so on).
3. Compute gradients and hessians. For each row, calculate the first derivative (gradient) and second derivative (hessian) of the loss with respect to the current prediction. The gradient says which way and how hard to push. The hessian says how confident the curvature is.
4. Grow one tree. Build a decision tree that uses the features to group rows with similar gradients. Each split is chosen to reduce the regularized objective as much as possible.
5. Compute leaf values. Every leaf gets a weight that is the best correction for the rows in it, shrunk by regularization.
6. Scale and add. Multiply the tree output by a learning rate (also called eta) and add it to the running prediction.
7. Repeat. Go back to step 2 with the updated predictions until the number of trees is reached or early stopping fires.

The regularized objective

XGBoost minimizes a sum of the data loss and a penalty on model complexity.

• The loss measures how well the model fits the labels.
• The penalty grows with the number of leaves in each tree (gamma) and with the size of the leaf weights (lambda for L2, alpha for L1).
• Smaller, gentler trees are therefore favored unless they truly reduce the loss.

Think of this like a budget for complexity. Each extra leaf has a price, and a split is only made if the improvement is worth the price.

Split finding

For every candidate split, XGBoost computes a gain score from the summed gradients and hessians on each side of the split. Conceptually:

• Gain is high when the left and right groups are each internally consistent in what correction they need.
• Gain is reduced by the complexity penalty.
• The split with the highest gain wins. If no split has positive gain, the node stays a leaf.

Because the score uses second-order information, leaf values and gains have closed-form expressions, which makes tree building fast and well-behaved.

Split algorithms

Histogramming is like sorting students into height groups before looking for the best cut-off, instead of comparing every student's exact height.

Sparsity-aware missing value handling

Missing values are not imputed by default. At every split, XGBoost learns a default direction for rows where the feature is missing. During training it tries sending the missing rows left and right and keeps whichever choice gives the higher gain. At prediction time, missing rows follow that learned direction.

This means "missing" can carry information. If customers with no recorded income behave differently, the tree can learn that.

Tree growth policies

• Depth-wise (the classic policy): grow level by level until max_depth is reached. Balanced, predictable.
• Loss-guided (leaf-wise): always split the leaf with the best gain, up to max_leaves. Can fit faster and deeper in some branches, with a higher overfitting risk. This is the default style of LightGBM.

Shrinkage, subsampling and early stopping

• Learning rate (eta). A small value such as 0.03 to 0.1 makes each tree contribute only a little, which reduces overfitting but needs more trees.
• Row subsampling (subsample). Each tree sees a random fraction of rows, similar to the idea in a random forest.
• Column subsampling (colsample_bytree, colsample_bylevel, colsample_bynode). Each tree, level or node sees only some features.
• Early stopping. Training stops when the validation metric has not improved for a set number of rounds. This replaces guesswork about how many trees to use.

Key concepts and terms

A worked example

A tiny numeric walk-through

Suppose we predict a house price in thousands. Four houses have true prices 100, 120, 200 and 220. With squared error loss, the gradient for each row is the prediction minus the target, and the hessian is 1.

1. Base prediction: the mean, 160. Residuals (target minus prediction): -60, -40, +40, +60.
2. Build a tree on feature "size". Small houses go left, large houses go right. The left leaf holds residuals -60 and -40, the right leaf holds +40 and +60.
3. With no regularization the leaf weights would be the means, -50 and +50. With lambda = 1, a leaf weight is roughly the sum of residuals divided by (count + lambda). Left becomes about -100 / 3, close to -33, and right about +33. Regularization pulls the values toward zero.
4. With eta = 0.5, the update is half of that: about -17 and +17.
5. New predictions: about 143 for the small houses and 177 for the large ones. The residuals shrink. The next tree works on those smaller residuals.

After many rounds, predictions converge toward the true values, with regularization preventing the trees from memorizing noise.

A realistic Python example

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
from xgboost import XGBClassifier

# df has numeric features, a few categorical columns, and a binary target
df = pd.read_csv("customers.csv")
y = df.pop("churned")

# Mark text columns as category so XGBoost can split on them directly
for col in df.select_dtypes("object").columns:
    df[col] = df[col].astype("category")

X_train, X_valid, y_train, y_valid = train_test_split(
    df, y, test_size=0.2, stratify=y, random_state=0
)

model = XGBClassifier(
    n_estimators=2000,
    learning_rate=0.03,
    max_depth=6,
    min_child_weight=3,
    subsample=0.8,
    colsample_bytree=0.8,
    reg_lambda=1.0,
    tree_method="hist",
    enable_categorical=True,
    eval_metric="auc",
    early_stopping_rounds=100,
    random_state=0,
)

model.fit(X_train, y_train, eval_set=[(X_valid, y_valid)], verbose=100)

pred = model.predict_proba(X_valid)[:, 1]
print("Validation AUC:", roc_auc_score(y_valid, pred))
print("Best iteration:", model.best_iteration)

Reading the result

• The large n_estimators is only an upper bound. Early stopping picks the best number of trees.
• Missing values in the frame need no imputation.
• Category columns are handled with native categorical support when enable_categorical=True.

Explaining predictions

import shap

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_valid)
shap.summary_plot(shap_values, X_valid)

SHAP values break each prediction into per-feature contributions. Tree models allow fast, exact computation, which is a major practical benefit of boosted trees.

Tuning with a search library

import optuna

def objective(trial):
    params = {
        "learning_rate": trial.suggest_float("learning_rate", 0.01, 0.2, log=True),
        "max_depth": trial.suggest_int("max_depth", 3, 10),
        "min_child_weight": trial.suggest_float("min_child_weight", 1, 20, log=True),
        "subsample": trial.suggest_float("subsample", 0.5, 1.0),
        "colsample_bytree": trial.suggest_float("colsample_bytree", 0.4, 1.0),
        "reg_lambda": trial.suggest_float("reg_lambda", 1e-3, 50, log=True),
        "reg_alpha": trial.suggest_float("reg_alpha", 1e-3, 10, log=True),
    }
    m = XGBClassifier(n_estimators=3000, tree_method="hist",
                      early_stopping_rounds=100, eval_metric="auc",
                      enable_categorical=True, **params)
    m.fit(X_train, y_train, eval_set=[(X_valid, y_valid)], verbose=False)
    return m.best_score

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=50)

Optuna and Bayesian optimization are common ways to search these settings efficiently. Ray Tune is an alternative for distributed search.

Strengths and limitations

Strengths

• Excellent accuracy on tabular data. With sensible tuning it is very hard to beat on small and medium tables.
• Little preprocessing. No scaling needed. Monotonic transformations of features do not change the splits.
• Native missing value handling. Learned default directions for absent values.
• Handles feature interactions automatically. Tree depth lets it model combinations of features.
• Robust to outliers in features. Splits depend on order, not magnitude.
• Fast training and prediction. Especially with histogram and GPU modes.
• Flexible objectives. Regression, classification, ranking, count and survival models, plus custom losses.
• Interpretability tools. Feature importance, partial dependence and exact tree SHAP.
• Mature ecosystem. Wrappers, model serving formats, distributed backends and monitoring integrations.

Limitations

• Needs tuning. Defaults are decent, but the best results need search over depth, learning rate and regularization.
• Overfits small noisy data if trees are too deep or too many.
• Weak extrapolation. Trees predict constants in regions, so they cannot extend a trend beyond the training range.
• Not smooth. Predictions are piecewise constant, which may not suit physical systems with smooth responses.
• Categorical features with very high cardinality are harder than with specialized methods such as CatBoost.
• Per-dataset training. Each new table needs a new model. There is no transfer of knowledge across tables.
• Sequential trees. Training is parallel inside each tree but not across trees.
• Limited with unstructured data such as images and text without feature extraction.
• Probability calibration may need a post-hoc step for well-calibrated probabilities.

How it compares with alternatives

A short rule of thumb: start with a gradient-boosted tree model as a baseline, add a random forest as a sanity check, and only move to heavier models when the baseline clearly leaves value on the table.

Role in tabular foundation models

Why XGBoost matters to the foundation-model story

A tabular foundation model is a single network pre-trained on a very large number of tables so that it can make predictions on a new table, often without gradient updates on that table. XGBoost sits on the other side of that picture. It is a per-table learner: it starts from nothing for each dataset. That makes it the natural reference for three questions:

1. Does the foundation model reach the accuracy of a tuned boosted tree?
2. How much faster or easier is it?
3. Where does the foundation model fail, and where does boosting still win?

You can read how this plays out in /model/benchmarks and the broader context in /model/research and /model/architecture.

The same raw-table realities, two different answers

Raw tables have properties that every method must handle.

Pre-training on synthetic tables, and the tree connection

Some tabular foundation models, such as TabPFN, are pre-trained on large numbers of synthetic tables drawn from random causal models. Tree-like structure is one of the patterns such priors cover, because many real tables behave like thresholds and interactions that trees capture well. A foundation model that learns from many such tables effectively learns an approximate, amortized version of what a tuned boosted ensemble would do on each one.

In-context learning versus boosting

• Boosting fits parameters (tree splits and leaf weights) to the training rows by gradient steps on that dataset.
• In-context learning passes the labeled training rows as context to a frozen network, which predicts new rows in a single forward pass.

For small tables, in-context learning can beat boosting without any search. For large tables, boosting scales more gracefully, because attention cost grows with the number of rows. Models such as TabICL address scale. Text-aware models such as TabSTAR address columns that contain words.

Imputation

Boosted trees are used for iterative imputation: each column with gaps is predicted from the others, in a loop. This is the idea behind many chained-equation imputers, and boosted trees are a frequent choice for the per-column learner. A tabular foundation model approaches the same task differently, by treating missing cells as the targets of a masked-cell objective during pre-training, so imputation is a native ability rather than a separate pipeline. See Masked cell modeling and Imputation networks.

Anomaly detection

Boosted trees are supervised, so they need labels. For anomaly detection without labels, tree-based Isolation forest and density-style methods are common. Foundation models can score how surprising a row or cell is given the rest of the table, which gives a label-free signal. XGBoost is still useful here as a supervised detector once some labeled fraud or fault examples exist. See Anomaly detection.

Semantic and sector classification

When columns have meaningful names or values, models that read semantics (for example, whether a column holds currencies, dates or sector codes) can classify a table or its columns. XGBoost can do the job after hand-built features such as text statistics are supplied. Foundation models that embed names and values jointly can do it more directly. See Semantic sector heads.

Continual learning

A boosted ensemble can be extended by adding trees to an existing model with the xgb_model argument. That is a simple form of continued training, but it can drift away from earlier behavior, and it does not protect old knowledge in a principled way. In neural foundation models, continual learning uses methods such as elastic weight consolidation and experience replay to add new capabilities without forgetting old ones.

A practical hybrid

• Use XGBoost as a strong, cheap baseline and a safety check.
• Use a tabular foundation model as a second opinion, especially on small tables.
• Compare on the same split, with the same metric.
• Consider blending the two, because their errors are often different.

Practical guidance and best practices

A sensible workflow

1. Split your data into train, validation and test sets. For time-dependent data, split by time.
2. Fit a baseline with default settings and early stopping. Record the validation metric.
3. Inspect feature importance and SHAP for leaks (a feature that is too good to be true).
4. Tune the main parameters with a search tool.
5. Fix the number of trees using early stopping on validation, then retrain on train plus validation with that count if you need the extra data.
6. Evaluate once on the held-out test set.
7. Check calibration and, if needed, calibrate probabilities.
8. Save the model, the feature list and the preprocessing code together.

Parameters to tune first

Imbalanced classes

• Use scale_pos_weight roughly equal to negatives divided by positives, or sample weights.
• Prefer ranking metrics such as AUC-PR or log loss for monitoring, not accuracy.
• Choose the decision threshold on validation data, not by default 0.5.

Categorical features

• For few categories, one-hot or native categorical support works well.
• For high-cardinality categories, consider target encoding done inside cross-validation folds, or try CatBoost.
• Always handle unseen categories at prediction time.

Speed

• Use tree_method="hist" and, if available, a GPU device.
• Reduce max_bin for faster training with small accuracy loss.
• Use early stopping to avoid wasted trees.
• Convert large data to DMatrix or QuantileDMatrix once.

Reproducibility

• Set random_state and a fixed number of threads when you need identical runs.
• Record library version, because defaults change between versions.
• Store the training data hash with the model.

Choosing an objective and metric

The objective is what training minimizes. The eval metric is what you watch. They can differ, and it is common to train on log loss while stopping on AUC.

Monotonic and interaction constraints

You can force a feature to have a non-decreasing effect (monotone_constraints) or limit which features may interact (interaction_constraints). These are valuable in regulated settings such as pricing or credit, where a higher income should never lower the score by design.

Deployment notes

• Models can be saved in a portable JSON format.
• Prediction is fast and light on memory, so it suits low-latency services.
• Watch for feature drift. A boosted model has no built-in way to flag inputs far from its training data.

Common mistakes

• Tuning on the test set. Always keep a final untouched test set.
• Leaking the target. Features built using the label or future information produce great validation scores and poor real use.
• Random splits on time series. Use forward-looking splits that respect time.
• Too many trees without early stopping. The model memorizes noise.
• Using accuracy on imbalanced data. A model that always predicts the majority class can look excellent.
• Scaling features for XGBoost. It is harmless but unnecessary, and wastes effort.
• Imputing missing values blindly. Native handling often does better, and "missing" can be a signal.
• Ignoring categorical encoding leakage. Target encoding fitted on the full data leaks labels.
• Reading feature importance as causation. Importance shows what the model used, not what causes the outcome.
• Comparing against a foundation model with an untuned baseline. A fair comparison needs both sides to be reasonably tuned.
• Expecting extrapolation. Trees cannot predict beyond the target range seen in training.
• Forgetting version drift. Re-running old code on a new library version can change results.
• Trusting default importance blindly. Split-count importance favors features with many possible thresholds. Prefer permutation importance or SHAP.
• Deep trees on tiny data. A depth of 10 on a few hundred rows memorizes the sample.
• Ignoring duplicated rows across splits. Near-copies in train and validation inflate scores.
• Skipping a simple baseline. A logistic regression or a mean predictor shows whether the boosted model adds real value.

Frequently asked questions

What is XGBoost used for?

XGBoost is used to predict outcomes from structured tables, such as whether a customer will leave, how much a house will sell for, whether a transaction is fraudulent, or how to rank search results. It handles classification, regression and ranking. It is popular because it gives strong accuracy with modest compute and little preprocessing.

How is XGBoost different from a random forest?

A random forest trains many deep trees independently on random samples and averages them, which reduces variance. XGBoost trains shallow trees one after another, each correcting the previous ones, which reduces bias as well. As a result boosting is usually more accurate when tuned, while a random forest is more forgiving with default settings.

Does XGBoost need feature scaling?

No. Trees split on thresholds, so only the order of values matters, and scaling does not change the result. Scaling is only needed if you combine XGBoost with other models that need it, or if you feed it linear-booster features.

How does XGBoost handle missing values?

It learns a default direction at each split. During training, rows with a missing value are tried on both sides of the split, and the side that gives the better gain becomes the default. At prediction time, missing values simply follow that stored path, so you do not have to impute them.

Which parameters should I tune first?

Start with the learning rate and the number of trees (using early stopping), then maximum depth and minimum child weight, then row and column subsampling, and finally the L1 and L2 regularization terms. Tuning in that order covers most of the possible gain with the least effort.

Is XGBoost better than deep learning for tabular data?

On many tabular benchmarks, tuned boosted trees match or beat standard neural networks, especially on small and medium datasets. Newer pre-trained tabular models narrow the gap and sometimes win on small tables. The honest answer is that it depends on data size, feature types and how much tuning each side receives.

Can XGBoost handle categorical features?

Yes, in two ways. You can encode categories yourself, using one-hot or target encoding, or you can use the native categorical support by marking columns as category type and enabling it. For very high-cardinality columns, a library built around categorical statistics, such as CatBoost, may do better.

How many trees should I use?

Set a large upper limit and let early stopping decide. The right number depends on the learning rate and data. With a learning rate near 0.05, a few hundred to a few thousand trees is common. The validation curve, not a fixed rule, should set the final count.

Can XGBoost run on a GPU?

Yes. The histogram method can run on a compatible GPU, which speeds up training on large datasets significantly. For small datasets the gain may be small, because data transfer and setup overhead can dominate.

Is XGBoost still relevant now that tabular foundation models exist?

Yes. It is still the standard baseline, it scales to very large tables, it is cheap to train and serve, and it works well when you have plenty of labeled rows. Tabular foundation models are strongest on small tables and fast prototyping, so the two are best seen as complements.

Glossary

• Boosting: a method that builds a strong model from many weak models trained in sequence.
• Gradient boosting: boosting where each new model fits the gradient of the loss.
• Decision tree: a model that splits data by feature thresholds into groups with similar targets.
• Leaf: a final node of a tree that holds an output value.
• Learning rate (eta): a factor that scales each tree's contribution.
• Regularization: a penalty that discourages overly complex models.
• Early stopping: halting training when validation performance stops improving.
• Histogram split: a split search over binned feature values for speed.
• Hessian: the second derivative of the loss, used to weigh updates.
• Feature importance: a measure of how much each feature contributed to the model.
• SHAP value: a per-feature contribution to a single prediction.
• Overfitting: fitting noise in the training data so that new data is predicted poorly.
• Calibration: agreement between predicted probabilities and real frequencies.
• In-context learning: predicting from labeled examples placed in the model input, with no weight updates.
• Tabular foundation model: a network pre-trained on many tables to transfer to new ones.

Summary

XGBoost is a regularized, fast and widely used gradient-boosting library that builds an ensemble of shallow trees, each correcting the last. It works well on tabular data with minimal preprocessing, handles missing values natively, and offers strong tooling for explanation and tuning. Its main costs are the need for per-dataset tuning, weak extrapolation, and no transfer between tables.

In the age of tabular foundation models, XGBoost is the benchmark every new method must meet. Pre-trained models such as TabPFN and TabICL offer a different trade: no per-table training and strong small-data results, in exchange for scale limits. The practical path is to keep a tuned boosted-tree baseline, compare it fairly with a foundation model on the same splits, and choose or blend based on data size, latency and accuracy needs. Related reading: LightGBM, CatBoost, Random forest, /model/architecture and /model/benchmarks.

Related

It belongs to Classical ML in my stack, under Foundation models and applied ML. Others in the same group: