Conformal prediction: intervals with a guaranteed coverage
A model that says 42 is less useful than one that says 'between 38 and 46, 90% of the time'. Conformal prediction wraps any trained model and, using a held-out calibration set, produces intervals with a coverage guarantee that holds without assumptions about the model or the data distribution, as long as new data look like the calibration data.
Level AdvancedAbout 12 minRead first: Linear regression, Calibration1 challenge in the Lab
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The data follow a curve with noise that grows to the right. A k-nearest-neighbour model, the blue line, is fitted on 300 training points. A separate calibration set gives the residual scores, and their ⌈(n+1)(1−α)⌉-th smallest value becomes the interval's half-width. The band is shown over 600 test points; orange ones fall outside. The panel reports coverage overall and separately for the low-noise and high-noise halves, and Repeat runs 200 random calibration sets to show how coverage varies.
Things to try
- With the defaults, check overall coverage: close to 90%. Now compare the two halves: too wide on the left, too narrow on the right.
- Switch to the adaptive score: the band widens where the data are noisy and both halves get closer to 90%.
- Use only 20 calibration rows and press Repeat: coverage is still about 90% on average, but individual splits vary much more.
- Set α to 0.05: the intervals widen to cover 95%.
- Press New sample a few times: the guarantee holds whatever the data, because it does not depend on the model being right.
How it works in real systems
Conformal prediction is used to put error bars on regression models, to return sets of labels with guaranteed coverage in classification, and to flag uncertain predictions for review. Libraries such as MAPIE and crepes implement it for scikit-learn models, and conformalised quantile regression is a common way to get adaptive widths. The guarantee needs exchangeable data; under distribution shift, weighted or online variants are used.
Key formulas
$$\hat q=\text{the }\big\lceil (n+1)(1-\alpha)\big\rceil\text{-th smallest of } s_1,\dots,s_n,\qquad C(x)=\big[\hat f(x)-\hat q,\ \hat f(x)+\hat q\big]$$
Split conformal prediction with absolute residual scores s_i = |y_i − f̂(x_i)|.
$$\mathbb P\big(Y_{n+1}\in C(X_{n+1})\big)\ \ge\ 1-\alpha$$
The marginal coverage guarantee, for exchangeable data.
$$s_i=\frac{|y_i-\hat f(x_i)|}{\hat\sigma(x_i)}\ \Rightarrow\ C(x)=\hat f(x)\pm\hat q\,\hat\sigma(x)$$
Normalised scores give adaptive widths.
Common mistakes
- Reading the guarantee as conditional: 90% coverage on average can hide much lower coverage in the hardest region.
- Calibrating on the rows the model was trained on: residuals there are too small and the intervals too narrow.
- Assuming the guarantee survives distribution shift: it needs calibration and test data to be exchangeable.
Real-world numbers
- Conformal prediction was developed by Vovk, Gammerman and Shafer; their book Algorithmic Learning in a Random World appeared in 2005.
- Split conformal needs one model fit and one sort of the n calibration scores.
- Conformalized quantile regression (Romano, Patterson and Candès, 2019) combines quantile models with conformal calibration for adaptive widths.
Further reading
- A Gentle Introduction to Conformal Prediction (Angelopoulos and Bates)
- Distribution-Free Predictive Inference for Regression (Lei et al.)
- Conformalized Quantile Regression
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Split conformal by hand
Fit on one part, calibrate on another, test on a third.
pip install scikit-learn numpy
import numpy as np
from sklearn.neighbors import KNeighborsRegressor
rng = np.random.default_rng(1)
x = rng.random(3000) * 1.6
y = np.sin(3 * x) + 0.5 * x + rng.normal(0, 0.08 + 0.45 * x ** 2)
X = x[:, None]
tr, cal, te = slice(0, 300), slice(300, 500), slice(500, None)
model = KNeighborsRegressor(n_neighbors=20).fit(X[tr], y[tr])
alpha = 0.1
scores = np.abs(y[cal] - model.predict(X[cal]))
n = len(scores)
q = np.sort(scores)[int(np.ceil((n + 1) * (1 - alpha))) - 1]
inside = np.abs(y[te] - model.predict(X[te])) <= q
print(f'coverage {inside.mean():.3f} (target {1 - alpha}), half-width {q:.3f}')
print('low-noise half :', inside[x[te] < 0.8].mean().round(3))
print('high-noise half:', inside[x[te] >= 0.8].mean().round(3))
Step 2 · Adaptive widths with a spread model
Divide the residual by an estimate of the noise level, then scale the interval back.
resid_tr = np.abs(y[tr] - model.predict(X[tr]))
spread = KNeighborsRegressor(n_neighbors=40).fit(X[tr], resid_tr)
sig = lambda X_: np.maximum(spread.predict(X_), 0.03)
scores = np.abs(y[cal] - model.predict(X[cal])) / sig(X[cal])
q = np.sort(scores)[int(np.ceil((n + 1) * (1 - alpha))) - 1]
inside = np.abs(y[te] - model.predict(X[te])) <= q * sig(X[te])
print(f'adaptive coverage {inside.mean():.3f}; low half {inside[x[te] < 0.8].mean():.3f}, high half {inside[x[te] >= 0.8].mean():.3f}')
Both versions cover at least 90% overall; the adaptive one spreads the coverage much more evenly across the noise levels.
More simulations
← Time-series forecasting · All AI Simulations · Training run →