LIME

Overview

LIME stands for Local Interpretable Model-agnostic Explanations. It explains one prediction from a complex model by fitting a small, simple model that imitates the complex one only in the neighbourhood of that single case. The simple model, often a weighted linear model, is easy to read. Its coefficients tell you which features pushed this particular prediction up or down.

LIME is popular because it is quick, works with almost any model and gives a result a non-specialist can read. It is also easy to misuse. The explanation depends on how the neighbourhood is sampled, so two runs can differ, and a local story does not describe the whole model. This article explains the idea in plain terms, walks through an example, compares LIME with SHAP and other tools, and shows how local explanations apply to models that learn from raw tables, including the ones covered on the model research page.

What it is

LIME is a method for explaining individual predictions of any classifier or regressor. "Model-agnostic" means it does not need to see inside the model. It only needs to send inputs in and read outputs back. The model can be a boosted tree, a neural network or an ensemble. LIME treats it as a black box.

"Local" means the explanation is only valid near the chosen instance. "Interpretable" means the explanation uses a form people can read, such as a short list of features with weights.

The central intuition

Imagine a very curvy mountain road seen from a plane. From above, the road twists in complicated ways. But if you zoom in on a few metres of it, the road looks almost straight. A straight line is a good local description of a curvy road, even though it is a poor global one.

LIME does the same thing to a model. The full decision surface of a complex model may be curved in many dimensions. Near one point, a straight line (a linear model) can approximate it well enough to read off slopes. Those slopes are the explanation.

What LIME produces

For a tabular row, LIME produces a short ranked list such as:

• usage_hours above a threshold: pushes the prediction towards "stays".
• support_tickets above a threshold: pushes towards "leaves".
• plan_age in a range: small push towards "stays".

Each item is a feature condition and a weight. Positive weights support the predicted class and negative weights oppose it. The list is for one row only.

What LIME is not

• It is not a global summary of the model.
• It is not a causal analysis. A feature can push a prediction without causing the real outcome.
• It is not deterministic unless you fix the random seed.
• It is not a guarantee of fairness or correctness.

Where it came from

LIME was introduced in a 2016 paper titled "Why Should I Trust You?: Explaining the Predictions of Any Classifier" by Ribeiro, Singh and Guestrin. The paper presented the method together with an open-source implementation. It arrived when complex models were spreading in sensitive areas and people wanted to know why a model said what it said.

The idea belongs to a wider family of surrogate models, where a simple model stands in for a complex one. Earlier work already used global surrogates, such as fitting a decision tree to mimic a neural network. LIME's contribution was to make the surrogate local and to make the whole process model-agnostic and easy to apply to text, images and tables.

After LIME, other methods appeared. SHAP, introduced in 2017, grounded feature attribution in Shapley values from cooperative game theory. Anchors, from some of the same authors, gave rule-style explanations. Counterfactual explanations asked a different question: what is the smallest change that would flip the prediction? LIME remains a common first tool because of its simplicity.

How it works

LIME follows a recipe with a handful of steps. Here they are for tabular data.

Step 1: choose the instance to explain

Pick the row whose prediction you want to understand. Call it x. For example, a customer the model flagged as likely to leave.

Step 2: create a cloud of nearby fake rows

LIME perturbs x many times to make new rows around it. For numeric features, it samples values from a distribution fitted to the training data, often a normal distribution using each feature's mean and spread. For categorical features, it samples categories according to how often they occur in training data.

Think of a chef tasting a soup and then making many slightly different versions, with a little more salt here, a little less pepper there, to learn which ingredient changes the taste most.

Step 3: ask the black box about each fake row

Each perturbed row is sent to the original model. LIME records the predicted probability or value. Now there is a table of fake rows and the model's answers for them.

Step 4: weight the fake rows by closeness

Rows near x matter more than rows far away. LIME applies a kernel that gives high weight to close rows and low weight to distant ones. The width of this kernel decides what "local" means. A narrow kernel looks only at very near points. A wide kernel looks at a bigger area.

Step 5: fit a simple model

LIME fits a simple model, usually a linear one with regularisation, to the weighted fake rows. It tries to match the black box's answers near x. To keep the explanation short, it limits the number of features used, using a selection step such as picking the top K features.

Step 6: read the weights

The coefficients of the simple model are the explanation. A large positive weight means the feature condition pushes the prediction up for this instance. A large negative weight pushes it down.

Why discretisation is used

By default, LIME for tables often bins numeric features into ranges, such as quartiles. The explanation then reads "usage_hours is above the upper quartile" rather than a slope on a raw number. This is easier for people to read, and it handles non-linear effects a little better. The cost is that bin boundaries affect the result.

The objective, in words

LIME balances two goals. The simple model should be faithful to the black box near x. It should also be simple enough to read. The method picks the simple model that gives the best fit near x while keeping complexity low. In the original paper the objective is written as a loss on local fidelity plus a penalty on the complexity of the explanation.

Key concepts and terms

A worked example

The following Python code explains one prediction of a random forest on a made-up customer table. File names are placeholders.

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from lime.lime_tabular import LimeTabularExplainer

df = pd.read_csv("customers.csv")
y = df["churned"].values
X = df.drop(columns=["churned"])

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0
)

model = RandomForestClassifier(n_estimators=200, random_state=0)
model.fit(X_train, y_train)

explainer = LimeTabularExplainer(
    training_data=X_train.values,
    feature_names=list(X_train.columns),
    class_names=["stays", "leaves"],
    mode="classification",
    discretize_continuous=True,
    random_state=0,
)

row = X_test.iloc[3].values
exp = explainer.explain_instance(
    data_row=row,
    predict_fn=model.predict_proba,
    num_features=5,
    num_samples=5000,
)

for feature_rule, weight in exp.as_list():
    print(f"{feature_rule:45s} {weight:+.3f}")

Reading the output

A printout might look like this (the numbers are illustrative):

support_tickets > 4.00                        +0.210
usage_hours <= 12.00                          +0.150
plan_age > 24.00                              -0.090
region_code <= 2.00                           +0.030
discount_used > 0.50                          -0.020

Positive values support the class being explained (here, "leaves"). Negative values oppose it. The first line says that having more than four support tickets pushed this customer's score towards leaving by about 0.21 in the local linear model. It does not say that tickets cause leaving.

A small numeric walk-through

Say the black box gives these probabilities for five fake rows near x, and the closeness weights are in the last column.

A weighted linear fit on this tiny table would give a positive slope on tickets and a negative slope on usage hours. Row 4 is far from x, so it gets a low weight and hardly shapes the line. This is the heart of LIME: fit a line, weighted by closeness, then read the slopes.

Checking the explanation

A good habit is to check local fidelity. LIME can report a score, often the R squared of the weighted fit, as exp.score. If it is low, the simple model does not match the black box well near x, and the explanation should not be trusted. You can also run LIME several times with different seeds and compare. If the top features change a lot, the explanation is unstable.

Strengths and limitations

Strengths

• Works with any model that returns predictions.
• Easy to understand: a short list of features and weights.
• Quick to run for a single row.
• Supports tables, text and images.
• Helps with debugging: odd explanations can reveal leakage or strange features.
• Useful for communication with non-technical readers, when used carefully.

Limitations

• Instability. Random sampling means results can change between runs unless the seed is fixed, and small changes in settings can change the ranking.
• Kernel width sensitivity. The notion of "local" is a choice, and a poor choice gives a poor explanation.
• Unrealistic samples. Perturbations treat features as if they vary independently. In real data features are correlated, so some fake rows could never occur. The black box may behave strangely on them.
• Local only. A good local story may hide global behaviour.
• Not causal. It describes the model, not the world.
• Linear assumption. If the model is highly non-linear even near x, a linear surrogate fits badly.
• Manipulability. Research has shown that a model can be built to behave differently on LIME's fake rows than on real data, hiding bias from the explanation.

The independence problem, explained simply

Suppose a table has "age" and "years_of_experience". Real people with age 20 do not have 30 years of experience. LIME may still sample such a pair, because it varies each feature on its own. The black box has never seen this combination, so its answer may be arbitrary. That answer then leaks into the explanation. This is why good practice includes checking explanations with domain knowledge.

How it compares with alternatives

LIME versus SHAP

Both give per-feature contributions for one row. LIME fits a local linear model to random perturbations. SHAP computes contributions based on how the prediction changes when features are added in different orders. SHAP values add up exactly to the difference between the prediction and the average prediction, which makes them easier to reason about. SHAP for tree models can be fast. LIME is lighter to set up for arbitrary models and often fine for a quick look. Many teams use both and check that they agree.

When LIME is the right pick

• You need a quick look at one prediction.
• The model is a black box with only a prediction function available.
• You need an explanation in a short list form.
• You will validate the explanation with a human who knows the domain.

When to prefer something else

• You need a stable, additive, auditable attribution: prefer SHAP.
• You need global behaviour: use global importance and dependence plots.
• You need to advise a person what to change: use counterfactuals.

Role in tabular foundation models

A tabular foundation model is pre-trained on many tables so that it can predict on a new table without much per-table training. It reads rows and columns of mixed types, copes with missing cells and often does not depend on column names. Such models are powerful but opaque. LIME matters here because it is one of the few tools that can explain them without opening them up.

Black-box access is enough

Models like TabPFN and TabICL predict in context: you provide training rows and query rows, and the model outputs predictions in one forward pass. LIME needs only a function that maps a row to a probability. You can wrap the call, including the context rows, in a prediction function and pass it to LIME. No access to weights or attention maps is needed. That makes LIME a natural first tool for explaining pre-trained table models.

Fixed context, changing query

When explaining an in-context model, keep the context (the training rows given to the model) fixed. LIME then perturbs only the query row. If you also changed the context, you would be explaining a different model each time. This is a common source of confusion and worth stating in any report.

Mixed types and missing values

Tabular foundation models take both numbers and categories, and they may see missing cells. LIME's sampling must respect this. Good practice includes:

• Sample categories from their observed frequencies.
• Treat missing as its own possible value if the model can handle it.
• Do not create values outside plausible ranges.
• Compare explanations across several perturbation settings.

No column names

Some table models are built so that they do not rely on column names, which helps when names are missing, shuffled or in another language. LIME's output still lists feature names, but they are labels you supply for people to read. The model itself sees only positions and values. If the model is truly name-agnostic, an explanation should not change when you rename columns. That makes a simple test: rename or reorder the columns and confirm that the attributions follow the data, not the names.

Rows and columns as two axes

These models often use attention across rows and across columns. LIME does not see that structure. Its explanation says which features of the query row matter for the prediction. It does not say which training rows in the context mattered. For that, you need other tools, such as influence methods or attention inspection, and these are less standard. Be clear in reports about which question your explanation answers.

Imputation and masked-cell training

Some foundation models are trained by hiding cells and asking the model to recover them, an idea related to imputation networks. LIME can explain an imputed value by treating "predict this masked column" as the target. The explanation then lists the other columns that drove the imputed value. This is a practical way to check that an imputation relies on sensible neighbours, such as a postal code helping to fill a city name.

Anomaly detection

When a model flags a row as unusual, a human wants to know why. LIME can be applied to an anomaly score function. The weights show which features made the row look odd. Pair this with the anomaly detection page for background on scoring rows. Because perturbations of an already odd row can land in sparse regions, results can be noisy, so check stability.

Sector and semantic classification

Another task is classifying what kind of table or column something is, such as finance, retail or telecom. Here the "features" may be summary statistics or value patterns of columns. LIME can show which patterns drove the label, for instance that a column of values with two decimals and a narrow range pushed a column towards "price". This supports trust and debugging of semantic heads.

Continual learning

When a model is updated over time, explanations may drift. A useful practice is to keep a fixed set of audit rows and re-run LIME after each update, with the same seed and settings. Large shifts in the top features for the same rows can signal that the update changed behaviour, even if accuracy looks the same. This is a light-weight regression test for model behaviour. See the architecture page for how such models are structured, and the benchmarks page for how they are compared.

Limits specific to foundation models

• Pre-trained models may be sensitive to rows that look unlike their training tables, which LIME's fake rows sometimes do.
• Context length and size can make each call expensive, so thousands of LIME samples may be slow. Reduce the number of samples, batch calls or cache the context.
• Explanations describe the model with a given context, not the model alone.

Practical guidance and best practices

Fix the random seed

Set the seed so that results are repeatable. Then run with a few other seeds to check stability, and report the range.

Use enough samples

The default number of samples may be too low for wide tables. Increase it until the top features stop changing.

Check the local fit score

If the fit score is low, widen or narrow the kernel, change the number of features or consider another method.

Choose the number of features deliberately

Five to ten features is readable. More is rarely useful. Choose by looking at how the fit score changes.

Sample realistically

If you can, use a perturbation scheme that respects correlations between features, or at least filter out impossible rows. Domain checks help.

Explain a set of rows, not one

Pick a handful of representative rows: confident correct ones, confident wrong ones and borderline ones. Look for patterns across them. Some libraries offer a way to pick representative explanations, but a manual selection often works well.

Combine with global tools

Use LIME for local stories and global importance for the big picture. If a feature is top-ranked locally for most rows but low globally, investigate.

Validate with an expert

An explanation is a claim about the model. A domain expert can say whether the claim makes sense. If an explanation says that the row id drove the prediction, you have found a bug.

Document the settings

Record the kernel width, number of samples, discretisation choice, seed and context used. Without these, an explanation cannot be reproduced.

Be careful with sensitive uses

If explanations are shown to customers or used in regulated decisions, treat them as a communication aid, not as proof. Review wording with legal and ethics advisers.

A short workflow

1. Pick representative rows.
2. Wrap the model as a function that returns probabilities.
3. Fix seeds and run LIME with enough samples.
4. Check local fit scores.
5. Repeat with other seeds and settings.
6. Compare with SHAP or other tools.
7. Review with a domain expert.
8. Record settings and findings.

Common mistakes

• Treating a LIME explanation as a global truth.
• Reporting an explanation from a single run with no stability check.
• Ignoring a low local fit score.
• Explaining a model with a changing context or changing preprocessing.
• Forgetting that discretisation affects the wording and sometimes the result.
• Reading weights as causal effects.
• Passing a prediction function that returns labels rather than probabilities, which gives poor fits.
• Using the wrong class index, so the explanation describes the opposite class.
• Passing training data in a different encoding from what the model expects.
• Letting categorical columns be treated as numbers.
• Using LIME to claim that a model is fair after looking at a few rows.
• Showing raw weights to non-experts without explaining what they mean.

A subtle mistake with encodings

Suppose the model was trained on one-hot encoded categories, but the explainer is given label-encoded data. The perturbations then vary the wrong thing, and the explanation is nonsense. Always make the explainer's view of the data match what the prediction function expects, or wrap the full preprocessing inside the prediction function.

LIME for text and images

Although this page focuses on tables, LIME was designed for other data too, and the contrast is useful.

Text

For text, LIME removes words at random from a sentence and watches how the prediction changes. The explanation highlights words that pushed the label up or down. A review classifier might show that the word "refund" pushed towards "negative".

Images

For images, LIME splits the picture into regions called superpixels and switches some of them off. The regions that matter most when present are shown as the explanation.

Why tables are trickier

In text and images, removing a part is a natural perturbation. In tables, removing a feature has no obvious meaning, so LIME replaces values with samples from the training distribution. That choice carries all the independence problems described above.

Tuning the explainer

LIME has a few settings worth understanding.

Kernel width

A small width makes the explanation very local. It follows the model closely near the point but may rest on few effective samples and be noisy. A large width uses more of the space and gives smoother but less local results. Try two or three values and see whether the top features hold.

Number of samples

More samples reduce noise but cost more model calls. For a cheap model, thousands are fine. For a heavy model, use fewer and rely on repeated runs.

Feature selection method

LIME can pick features by forward selection, by highest weights or by an automatic choice. The choice affects which features appear. Highest weights is a fast default for many cases.

Sampling around the training distribution

The default sampling uses statistics from the training data. If your row is an outlier, its neighbourhood is drawn from the bulk of the data rather than from around the row. Some setups allow sampling around the instance instead. Know which one you use.

A stability test script

import collections

counts = collections.Counter()
for seed in range(20):
    explainer = LimeTabularExplainer(
        X_train.values,
        feature_names=list(X_train.columns),
        class_names=["stays", "leaves"],
        mode="classification",
        random_state=seed,
    )
    exp = explainer.explain_instance(row, model.predict_proba, num_features=5)
    for rule, _ in exp.as_list():
        counts[rule.split(" ")[0]] += 1

print(counts.most_common())

This counts how often each feature appears in the top five across twenty seeds. A feature that appears in all twenty is a stable driver. A feature that appears in three is probably noise.

Presenting explanations to people

An explanation is only useful if the reader understands it. Some guidelines:

• Use plain feature names, not internal codes.
• Show direction with words and colour, and keep the number of items small.
• Say that the explanation is about the model, not the world.
• Show the prediction and its probability next to the list.
• Give a one-line caveat about stability.
• Avoid suggesting actions the explanation cannot support.

Good practice is to test explanations with real users. Ask them what they would do after reading it. If they draw conclusions the method cannot support, rewrite it.

When an explanation looks wrong

Sometimes a LIME result surprises you. Work through these checks.

1. Is the class index right? Confirm which class the weights refer to.
2. Is the encoding consistent between explainer and model?
3. Is the local fit score low? If so, distrust the result.
4. Does the list change with another seed? If so, increase samples.
5. Does a leaky or ID-like column appear? If so, fix the data.
6. Do correlated features split the credit? If so, group them in the report.
7. Does the explanation disagree with SHAP? Look at a few more rows to see which one matches domain sense.

Frequently asked questions

What does LIME stand for?

LIME stands for Local Interpretable Model-agnostic Explanations. Local means it explains a single prediction, interpretable means it uses a simple readable model, and model-agnostic means it works with any model by using only inputs and outputs.

How does LIME explain a prediction?

It creates many slightly changed copies of the row, asks the original model to predict each copy, then fits a simple weighted model that imitates those predictions near the row. The coefficients of that simple model show which features pushed the prediction up or down.

Is LIME the same as SHAP?

No. Both give feature contributions for one prediction, but they work differently. LIME fits a local linear model to random samples, while SHAP uses Shapley values from game theory and gives contributions that add up exactly to the prediction. SHAP is usually more stable, and LIME is often lighter to set up.

Why do LIME explanations change between runs?

Because LIME relies on random sampling to build the neighbourhood. A different seed gives different fake rows and so a slightly different fit. Fixing the seed makes runs repeatable, and using more samples makes the results steadier.

Can LIME explain any machine learning model?

In principle yes, as long as you can call the model and get a prediction. This includes tree ensembles, neural networks and pre-trained tabular foundation models. The quality of the explanation still depends on how the model behaves on the perturbed inputs.

Does LIME show cause and effect?

No. It shows which features influenced the model's output for one row, not which features cause the real-world outcome. A feature can be a strong signal for the model simply because it is correlated with something else.

What is a good local fit score for LIME?

There is no universal threshold, but a higher score means the simple model matches the black box better near the row. If the score is low, the explanation is unreliable and you should adjust the kernel width, the number of features or the method.

Can LIME be fooled?

Yes. Research has shown that a model can be built to detect LIME's unrealistic perturbed rows and behave differently on them, which hides its true behaviour from the explanation. Cross-checking with other methods and with real data slices reduces this risk.

How many features should I show in a LIME explanation?

Usually five to ten. Fewer than five may miss important drivers, and more than ten is hard to read. Look at how the local fit changes as you add features and stop when gains become small.

Can LIME be used with tabular foundation models?

Yes. It only needs a prediction function, so you can wrap a pre-trained table model together with its fixed context rows and explain each query row. Keep the context unchanged during explanation, and be aware that calls may be costly, so reduce the sample count when needed.

Glossary

• Anchor: a rule that locally guarantees a prediction with high precision.
• Black-box model: a model whose internals are hard to inspect.
• Counterfactual: a changed version of an input that gives a different prediction.
• Discretisation: grouping numeric values into ranges.
• Explainability: the ability to describe why a model made a decision.
• Fidelity: how closely a surrogate matches the original model.
• Global explanation: a description of overall model behaviour.
• Instance: one row of data.
• Kernel: a function that gives more weight to nearer points.
• Local explanation: a description of behaviour near one input.
• Perturbation: a small random change applied to an input.
• Regularisation: a penalty that discourages complex models.
• Shapley value: a fair share of credit among features, from game theory.
• Sparse: using only a few non-zero terms.
• Surrogate: a simple model that stands in for a complex one.
• Stability: consistency of results across repeated runs.

Summary

LIME explains one prediction at a time by building a cloud of nearby fake rows, asking the black-box model about each, and fitting a simple weighted model whose coefficients are easy to read. It is model-agnostic, quick and good for first-look debugging and communication.

Its weaknesses are random variation, sensitivity to the neighbourhood definition, unrealistic samples and a purely local scope. Use a fixed seed, check the local fit score, repeat runs and compare with SHAP and global tools. For tabular foundation models that predict in context, LIME is a practical way to explain query rows as long as the context stays fixed and the model sees realistic inputs. For related reading, see Random Forest, TabPFN, anomaly detection and the model research page.

Related

It belongs to Classical ML in my stack, under Foundation models and applied ML. Others in the same group: