SHAP

Overview

SHAP stands for SHapley Additive exPlanations. It is a method that explains one prediction of a machine learning model by giving each input feature a score. The score says how much that feature pushed the prediction up or down, compared with a typical prediction. The scores are computed with Shapley values, an idea from cooperative game theory that splits a shared reward fairly among players.

The result is easy to read. Start from a baseline, which is the average prediction. Add the SHAP value of every feature. You land exactly on the prediction for this row. Because the method works for almost any model, it has become a default tool for explaining models on tables, for auditing decisions, and for debugging data problems. It is a strong companion to models such as TabPFN and TabICL, and to any pipeline in the wider Supervised learning family.

What it is

SHAP is both a theory and a software toolkit. The theory defines a single "fair" way to assign credit to features. The toolkit offers fast algorithms that approximate or compute that credit for different model types.

The one-sentence idea

Imagine the features of a row as a team of players. The model prediction is the prize that the team wins. SHAP asks: how should the prize be shared so that each player gets what they really contributed?

What a SHAP value tells you

• A positive SHAP value means the feature moved this prediction above the baseline.
• A negative value means it moved the prediction below the baseline.
• A value near zero means the feature did little for this row.
• The sum of all values plus the baseline equals the model output.

Local and global views

SHAP is local by nature. It explains one row at a time. But if you compute values for many rows, you can summarize them. The average of the absolute SHAP values of a feature gives a global importance score. Plots of all values together show the shape of the effect, including direction and interactions.

What SHAP is not

• It is not a proof of cause and effect. It describes the model, not the world.
• It is not a model. It does not make predictions.
• It is not a single algorithm. It is a family of estimators for the same target quantity.

Where it came from

The roots of SHAP are older than machine learning. In 1953 a mathematician introduced the Shapley value as the only way to share the gains of a coalition that satisfies a short list of fairness rules. It earned a place in economics and in the study of voting power, cost sharing, and bargaining.

In the 2010s, researchers began to look for ways to explain complex models such as large tree ensembles and deep networks. Earlier tools gave feature importance for the whole model. Tools such as LIME, published in 2016, explained single predictions by fitting a small, simple model around one row.

The unifying paper

The key step came in 2017 with the paper "A Unified Approach to Interpreting Model Predictions". It showed that several existing explanation methods are all instances of one class, which the authors named additive feature attribution methods. It then proved that, within this class, only one solution satisfies three desirable properties, and that solution is the Shapley value. The paper also introduced practical estimators, including a model-agnostic one and one for deep networks.

Fast exact values for trees

A later line of work, published around 2018 and extended in 2020 under the title "From local explanations to global understanding with explainable AI for trees", gave a polynomial time algorithm for tree ensembles. This made exact SHAP values practical for gradient boosted trees and random forests, which dominate real-world tabular work. It also added interaction values and tools for global analysis.

Why it spread

• It had a clear theory, so people could trust it.
• It had fast code for the models that practitioners already used.
• It had good plots, so results were easy to share.
• It fit regulatory needs for explanations of automated decisions.

How it works

The method has one central quantity and several ways to estimate it. First the central quantity, with an everyday analogy.

The restaurant bill analogy

Four friends share a large dinner. One orders a costly bottle of wine. One orders only a salad. Splitting the bill equally feels unfair. Splitting it by what each person ordered works if every dish is separate. But what about a shared pizza that exists only because three people wanted it?

Shapley values handle this. For each friend, imagine every possible order in which the group could have arrived at the table. Each time, record how much the bill grew when that friend sat down. Average that growth across all arrival orders. That average is the friend's fair share.

SHAP does the same with features. "Arriving at the table" means "being revealed to the model".

Step by step

1. Pick a row you want to explain and a background data set that represents typical rows.
2. Define a value function for any subset of features. It answers: what would the model predict if it only knew these features?
3. For a feature of interest, consider every subset of the other features.
4. For each subset, compute the change in the value function when the feature is added.
5. Take a weighted average of those changes. The weights favor very small and very large subsets in a precise way that comes from counting orderings.
6. Repeat for each feature. The results are the SHAP values.

The value function

Models cannot usually run with a feature missing. So SHAP needs a way to "remove" a feature. The common choice is to replace the missing features with values sampled from the background data and average the model output. This is called the interventional approach. A second choice conditions on the known features and samples the rest from the matching conditional distribution. That is the observational approach.

The two choices answer slightly different questions. The interventional version asks how the model uses a feature. The conditional version asks what the feature tells us given the others. They differ most when features are strongly correlated.

The exact formula in words

The value for feature i is the sum, over all subsets S that do not contain i, of a weight times the gain from adding i to S. The weight equals the number of orderings that produce S before i, divided by the number of all orderings. In plain terms, it is a fair average over every order in which features could be added.

With n features there are 2 to the power n subsets. Computing it directly is impossible for large n. So practical estimators exist.

The main estimators

• KernelSHAP treats the problem as a weighted linear regression over sampled subsets. It works with any model, but it is slow, because it calls the model many times.
• TreeSHAP uses the tree structure to compute exact values in polynomial time. It is fast and exact for tree ensembles.
• DeepSHAP combines ideas from a layer-wise attribution method for networks with Shapley theory. It is fast but approximate.
• LinearSHAP has a closed form for linear models, based on the coefficients and the feature means.
• Permutation and sampling estimators draw random feature orderings and average the marginal changes. They are simple and unbiased, though noisy.
• Partition estimators group correlated features into a hierarchy to reduce cost and handle dependence.

The four fairness properties

The Shapley value is the only attribution that satisfies all four rules below.

These properties explain why many teams prefer SHAP to ad hoc importance scores.

Interaction values

Features rarely act alone. SHAP can split each value into a main effect and pairwise interaction terms. For trees this is exact. The interaction matrix for a row has main effects on the diagonal and shared effects off the diagonal. This helps to see that two features matter only together, such as income and loan amount in a credit model.

The common plots

• Waterfall plot. Starts at the baseline and adds one bar per feature until it reaches the prediction. Best for one row.
• Force plot. A compact one-line version of the same story.
• Beeswarm plot. One dot per row per feature, colored by feature value. Shows importance, direction, and spread.
• Bar plot. Mean absolute SHAP value per feature. A global ranking.
• Dependence plot. Feature value on the x-axis, SHAP value on the y-axis, colored by a second feature. Shows the shape of the effect and interactions.
• Heatmap. Rows grouped by similarity of their explanations.

Cost and scaling

The cost of an explanation depends on three things.

• The number of features. Exact enumeration doubles with each added feature.
• The number of background rows. Each coalition is evaluated once per background row.
• The cost of one model call. Large networks make every call expensive.

Practical estimators trade accuracy for speed:

• Fewer sampled coalitions means more noise.
• A smaller background means a rougher baseline.
• Grouping features cuts the number of players.

A good habit is to plot the estimate against the number of samples. When the curve flattens, you have enough.

Explaining different outputs

SHAP explains whatever number the model returns. That choice is yours.

• For regression, explain the predicted value.
• For classification, explain the log-odds of a class or the probability of a class.
• For multi-class models, you get one set of values per class.
• For anomaly detectors, explain the anomaly score.
• For a ranking model, explain the score before sorting.
• For a loss, explain the per-row error to see which features drive mistakes.

Explaining the loss is an underused trick. It points to the columns that cause the model to fail, which is often more useful than knowing which columns it likes.

Group and hierarchical explanations

Sometimes a single column is too small a unit. A date might be split into year, month and weekday. An address might be split into city and postal code. Treat each family as one player by summing the values of its members, or by defining a masker that hides the whole family together. This keeps the story readable and avoids splitting credit among pieces that belong together.

A mental checklist for reading any SHAP plot

1. What is the output being explained, and on what scale?
2. What is the baseline, and what background produced it?
3. Which rows are shown, and how were they chosen?
4. Are any features grouped or merged?
5. Is the story stable across seeds and samples?

Key concepts and terms

A worked example

A tiny numeric walk-through

Consider a model with two features, a and b:

f(a, b) = 10 + 4a + 6b + 2ab

Let the baseline be a = 0 and b = 0. We want to explain the row a = 1, b = 1.

The value function for each coalition:

There are two orderings: a first or b first.

For feature a:

• If a joins first, it adds 14 minus 10, which is 4.
• If a joins after b, it adds 22 minus 16, which is 6.
• The average is 5.

For feature b:

• If b joins first, it adds 16 minus 10, which is 6.
• If b joins after a, it adds 22 minus 14, which is 8.
• The average is 7.

Check efficiency: 10 + 5 + 7 = 22. The values add up to the prediction.

Notice the interaction term 2ab. It is worth 2 in total, and SHAP splits it evenly, 1 for each feature. Feature a gets 4 from its own term plus 1. Feature b gets 6 plus 1. That is the symmetry property at work.

A code example with a tree model

The following sketch trains a gradient boosted model on a table and explains it. It uses the open source shap package.

import numpy as np
import pandas as pd
import shap
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split

X, y = fetch_california_housing(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0
)

model = GradientBoostingRegressor(n_estimators=200, max_depth=3)
model.fit(X_train, y_train)

# TreeSHAP: exact and fast for tree ensembles
explainer = shap.TreeExplainer(model)
shap_values = explainer(X_test.iloc[:500])

# Local view: one row
shap.plots.waterfall(shap_values[0])

# Global views
shap.plots.bar(shap_values)
shap.plots.beeswarm(shap_values)

# Check the efficiency property
pred = model.predict(X_test.iloc[:500])
recon = shap_values.base_values + shap_values.values.sum(axis=1)
print(np.allclose(pred, recon, atol=1e-6))

The last check is worth keeping in any pipeline. If the sum of the values does not match the prediction, something is wrong with the explainer settings or the output scale.

Explaining a black box with KernelSHAP

For a model that is not a tree, such as a neural network or an in-context learner, use a model-agnostic explainer. Wrap the prediction function and give it a small background sample.

background = shap.sample(X_train, 100, random_state=0)

def predict_fn(data):
    return model.predict(pd.DataFrame(data, columns=X_train.columns))

explainer = shap.KernelExplainer(predict_fn, background)
values = explainer.shap_values(X_test.iloc[:20], nsamples=500)

Keep the background small, the explained set small, and the number of samples moderate. Cost grows quickly with all three.

Reading the result

Suppose the waterfall shows a baseline of 2.1 and the prediction is 3.4 for a house price in some units. The bars might say that median income adds 0.9, latitude adds 0.4, and house age subtracts 0.1. Each bar is a statement about this model on this row. It is not a statement about what would happen to the real house price if income changed.

Strengths and limitations

Strengths

• Solid theory. Unique solution under fairness axioms.
• Local accuracy. Values add up to the prediction.
• Model agnostic options. Works with trees, networks, linear models, and black boxes.
• Exact and fast for trees. Suits the models most used on tables.
• Rich visuals. Standard plots make results easy to communicate.
• Both local and global. One method gives per-row stories and overall rankings.
• Interactions. Pairwise terms are available for trees.

Limitations

• Cost. Model-agnostic estimation can be slow with many features or rows.
• Choice of background. Results depend on the reference data. A poor background gives misleading numbers.
• Correlated features. Credit is split among related features in ways that can surprise people. Interventional methods may also evaluate the model on unrealistic combinations of values.
• Not causal. A large SHAP value does not mean that changing the feature would change the outcome in the world.
• Instability. For sampling methods, values vary with the random seed. For very flexible models, small input changes can reshape explanations.
• Scale matters. For classifiers, values on the probability scale and on the log-odds scale tell different stories.
• Potential for misuse. Explanations can be manipulated by building models that behave well on the background and badly elsewhere.
• Human limits. A table of 200 SHAP values is no more readable than a table of 200 features.

How it compares with alternatives

Choosing between them

• Use SHAP when you need consistent, additive, per-row explanations and a global summary from the same numbers.
• Use permutation importance style checks, that is, score drops on held-out data, when you want a quick global ranking tied to real performance.
• Use counterfactuals when the audience wants to know what to change.
• Use an interpretable model when the stakes demand full transparency.

Role in tabular foundation models

Tabular foundation models learn directly from raw tables. A table has rows and columns, mixed types such as numbers, categories and text, missing cells, and sometimes no meaningful column names. These models, such as TabPFN and TabICL, are pre-trained on large collections of synthetic or real tables. At prediction time they use in-context learning: the labeled rows are given as context and the model predicts the new rows in a single forward pass, with no weight updates. See /model/research and /model/architecture for how such systems are built.

SHAP matters here because these models are black boxes by default. A buyer of a tabular prediction needs to know why. SHAP is the standard language for that answer.

Explaining an in-context learner

An in-context model is a function from a row to a prediction, given a fixed context set. That is exactly what a model-agnostic SHAP explainer needs. Wrap the prediction call, fix the context, and run KernelSHAP or a permutation explainer over the feature columns. Three points are important:

• Keep the context set fixed during explanation. Changing the context changes the function.
• Use a background drawn from the same table. This respects the mixed types and the real value ranges.
• Budget the cost. Each call is a forward pass through a large network. Sample rows and use a modest number of coalitions.

Row and column structure

A tabular foundation model treats a row as a set of cells that carry both a value and a column identity. Attention often runs across the rows and across the columns, as in axial attention. SHAP maps naturally onto this structure: one score per column for each row. Because the models are built to be indifferent to column order, SHAP scores should be too. This is a useful sanity test. Shuffle the columns, rerun, and compare.

Mixed types and missing values

Real tables mix numbers with categories and have gaps. SHAP handles this if the value function does. For a categorical column, "removing" it means sampling a category from the background. For a missing cell, there are two readings. If missingness is informative, treat "missing" as a value and let it carry credit. If it is not informative, impute first and explain the imputed table. Pairing SHAP with imputation networks lets you see which filled-in values changed a decision and which did not.

No column names

Some tabular models are designed to work when column names are absent or untrustworthy. SHAP does not need names. It scores positions. But a human cannot read "column 17". The usual pattern is to attach names or descriptions after the fact. When a semantic layer assigns meaning to columns, for example through semantic sector heads, the SHAP output can be reported by meaning, such as "payment history" or "device type", rather than by raw header.

Pre-training on synthetic tables

Models pre-trained on synthetic tables learn general rules about how columns relate: monotone trends, interactions, thresholds, noise. SHAP is a way to check what they learned on your real data. If a model trained on synthetic worlds attributes a large share of a prediction to a column that makes no business sense, that is a flag for data leakage, an ID column, or a distribution shift between pre-training and your table.

Continual learning

When a model is updated over time, explanations may drift. Treat the SHAP summary as a monitored artifact. Store the mean absolute values per feature for each release. A sudden jump in the share of one feature signals a change in data or in model behavior. This ties into meta-learning and transfer learning practices where reuse is common.

Imputation

When a foundation model fills missing cells, SHAP can explain each filled value by treating the imputed column as the target and the observed columns as inputs. This tells you which neighbors in the row drove the guess. It is a way to audit masked cell modeling outputs, where the model has learned to predict hidden cells from visible ones.

Anomaly detection

A detector flags a row as strange. The next question is always why. SHAP applied to the anomaly score gives a ranked list of the columns that made the row look odd. It works with isolation forest, because the score is a function of tree path lengths and TreeSHAP can explain it. It also works with reconstruction-based detectors built from autoencoders, by explaining the reconstruction error.

Sector and semantic classification

When a model assigns a row, a company, or a transaction to a sector or semantic class, SHAP shows which fields drove the label. This is useful for review queues: a reviewer sees the label and the top three reasons. It also reveals shortcut features, such as a name token that mimics the label.

Mixture-of-experts models

In a mixture-of-experts layer, a router sends each input to a few experts. A feature-level explanation of the whole model still works through KernelSHAP. For deeper insight, combine it with logs of which experts fired. This gives two views: which inputs mattered, and which internal specialists handled them.

Links to benchmarks

Explanations should be evaluated, not assumed. See /model/benchmarks for how accuracy is measured across table collections. Explanation quality can be checked in parallel: delete the top features by SHAP rank and watch the score fall faster than when deleting random ones.

Practical guidance and best practices

Before you start

• Decide the question. Is it "why this row?", "what matters overall?", or "is the model fair across groups?" The plots differ.
• Decide the output scale. For classifiers, use log-odds for additivity or probability for readability, and say which one you used.
• Choose the explainer to match the model. Use TreeSHAP for trees. Use a model-agnostic explainer for the rest.

Choosing a background

• Use a sample of 50 to 500 training rows for model-agnostic explainers.
• Summarize with k-means centroids if cost is high.
• For tree explainers, use the interventional option with a background sample when correlation matters, or the tree-path option when speed matters.
• Make the background reflect the population you care about. Explaining a loan for a young applicant against an all-ages background is a legitimate choice, but state it.

Handling correlated features

• Group highly correlated columns and explain the group as one unit.
• Report that credit may be shared among them.
• Do not over-read the split between two near-duplicate columns.

Validating the explanations

1. Check the efficiency sum on a few rows.
2. Shuffle the order of columns and confirm that values are unchanged.
3. Run with two different seeds for sampling explainers and compare.
4. Run a deletion test: remove the top features and watch the performance drop.
5. Inspect a few rows by hand with domain experts.

Communicating results

• Show a waterfall for one decision and a beeswarm for the overall picture.
• Use clear feature names and units.
• Round values to a sensible precision.
• Explain that the baseline is an average, not a threshold.
• Avoid causal language. Say "the model relied on" instead of "caused".

Performance tips

• Explain a random sample of rows, not the whole table.
• Use the tree explainer for tree models even if the model-agnostic one is available.
• Cache the explainer and the background.
• Parallelize across rows.
• For text or image inputs, use the partition explainers with hierarchical masking.

Governance

• Version the explainer settings next to the model.
• Log the background set identifier.
• Keep the global summary per release in your experiment tracker, such as MLflow.
• Review explanations as part of evals before release.

Common mistakes

• Reading SHAP as causal. The values describe model behavior. Changing a feature in real life may not move the outcome the same way.
• Ignoring the background. Two analysts with different backgrounds get different numbers from the same model.
• Mixing scales. Comparing log-odds values from one plot with probability values from another.
• Explaining the training set only. Values on training rows can reflect overfitting. Check held-out rows.
• Trusting correlated splits. Two nearly identical columns may share credit in an arbitrary way.
• Using too few samples. Sampling explainers with a handful of draws give noisy results.
• Forgetting ID-like columns. A high rank for an identifier is a leakage alarm, not an insight.
• Over-explaining. Presenting 40 bars on a waterfall plot hides the story. Cap the display and merge the tail.
• Treating the global bar chart as the whole truth. A feature can matter a lot for a small group and little on average.
• Skipping the sanity checks. No efficiency check, no deletion test, no column shuffle.
• Applying one explanation to a changed model. After retraining, recompute.
• Hiding uncertainty. Report that values come from a model and an estimator, and both carry error.

Frequently asked questions

What does a SHAP value actually mean?

A SHAP value is the average amount by which a feature moves a model prediction away from the baseline, taken over all the orders in which features could be revealed to the model. If the value is positive, the feature raised this prediction. If it is negative, it lowered it. All the values for one row add up to the gap between the baseline and the prediction.

Is SHAP the same as feature importance?

Not exactly. Classic feature importance is usually a single global number per feature. SHAP gives a number per feature per row, and a global importance can be built from them by averaging absolute values. The per-row view shows direction, interactions, and subgroups that a single score hides.

Does SHAP work for any model?

Yes, in the sense that a model-agnostic estimator only needs a function that maps inputs to outputs. It can be slow for large models. For tree ensembles, linear models, and some neural networks, specialized estimators are much faster. The explanation always concerns the model as a function, not the learning algorithm.

Why do my SHAP values change when I change the background data?

Because the value function simulates missing features by drawing from the background. A different background means a different notion of "typical", so the baseline and the credit assigned to each feature shift. This is expected. State the background you used and keep it fixed when comparing runs.

How does SHAP handle correlated features?

The interventional version breaks correlations when it fills in absent features, which can create unrealistic rows and spread credit among related columns. The conditional version respects correlations but credits a feature for information it only carries through its neighbors. Neither is perfect. The practical remedy is to group correlated features and interpret the group.

Can SHAP explain a model that uses in-context learning on tables?

Yes. Treat the model, with its context rows fixed, as a function from a new row to a prediction, and run a model-agnostic explainer over the feature columns. Keep the context unchanged during the run, use a background from the same table, and budget for the cost of repeated forward passes.

Is SHAP evidence of fairness or bias?

It can help reveal bias, for example by showing that a protected attribute or a close proxy has a large effect on predictions. It cannot prove fairness. Fairness needs outcome-level metrics across groups, and careful thought about the data, the decision, and its consequences.

How many rows should I explain?

For a global picture, a few hundred to a few thousand well-chosen rows are usually enough, because the averages stabilize. For audits of individual decisions, explain exactly the rows in question. For tree models, exact values for the whole table are often affordable.

What is the difference between SHAP and LIME?

LIME fits a small interpretable model around one row using perturbed samples, and the weights depend on settings such as the kernel width. SHAP defines credit through Shapley values and has the additivity and consistency properties. KernelSHAP is in fact a LIME-style regression with specific weights chosen so that the result equals the Shapley value.

Can SHAP be used for time series or text?

Yes, with care. For text, features are tokens or groups of tokens and masking is done by hierarchy. For time series, features are time steps or windows, and correlations over time are strong, so group values by window. In both cases the choice of masking strategy shapes the result.

Glossary

• Additive explanation: an explanation in which feature contributions sum to the model output minus a baseline.
• Background data: reference rows used to simulate the absence of a feature.
• Baseline: the expected model output over the background data.
• Beeswarm plot: a summary plot with one dot per row per feature.
• Coalition: a subset of features considered together.
• Dependence plot: a plot of SHAP value against feature value.
• Efficiency: the property that values sum to the prediction minus the baseline.
• Interventional approach: filling absent features independently from the background.
• KernelSHAP: a model-agnostic estimator based on weighted regression.
• Log-odds: the logarithm of the odds, the natural scale of a logistic classifier.
• Marginal contribution: the change in output when one feature is added to a coalition.
• Shapley value: the fair share of a payout in a cooperative game.
• TreeSHAP: an exact, fast algorithm for tree ensembles.
• Value function: a function that gives the model output for a given subset of known features.

Summary

SHAP explains a prediction by sharing it fairly among input features using Shapley values. Its values add up to the gap between the baseline and the prediction, and it is the only attribution that meets a small set of fairness rules. TreeSHAP makes it exact and fast for tree models, while KernelSHAP and permutation methods extend it to any model, including in-context tabular foundation models. Use it with a well-chosen background, check the sum property, group correlated features, and avoid causal claims. Combined with anomaly detectors such as isolation forest, imputation models, and semantic classifiers, SHAP turns black-box tabular predictions into reasons that people can review. For related reading see Bayesian optimization for tuning, and /model/benchmarks for evaluation.

Related

It belongs to Classical ML in my stack, under Foundation models and applied ML. Others in the same group: