Calibration: do a tabular model's probabilities mean what they say?
Among all the rows a model calls 70% likely, about 70% should be positive. Accuracy cannot tell you that; a reliability diagram can. Make the model overconfident, then fix it with one number.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The diagram plots the average predicted probability in each of ten bins against how often the event really happened; the diagonal is perfect calibration and dot size is the number of rows. The table reports ECE, Brier score, log loss and accuracy on 1,500 unseen rows.
Things to try
- Make attention very sharp: the curve flattens away from the diagonal (overconfidence).
- Switch on temperature scaling and watch ECE drop while accuracy stays the same.
- Compare with logistic regression.
How it works in real systems
scikit-learn provides calibration_curve, brier_score_loss and CalibratedClassifierCV (Platt scaling and isotonic regression); temperature scaling is the standard fix for neural network classifiers.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Data and two models
The same synthetic table as the simulation, so the numbers are comparable.
import numpy as np
rng = np.random.default_rng(7)
def churn(n):
t = rng.integers(1, 73, n); f = rng.integers(20, 121, n); c = np.minimum(9, rng.exponential(2.2, n).astype(int))
a = (rng.random(n) < 0.4).astype(int); u = rng.integers(1, 81, n)
z = 1.1 - 0.05 * t + 0.025 * (f - 60) + 0.45 * c - 1.6 * a - 0.015 * u + 1.1 * ((c > 5) & (a == 0))
y = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
return np.c_[t, f, c, a, u].astype(float), y
X, y = churn(60); X_test, y_test = churn(300)
X, y = churn(500); X_te, y_te = churn(3000)
from tabpfn import TabPFNClassifier
from sklearn.linear_model import LogisticRegression
p_tf = TabPFNClassifier().fit(X, y).predict_proba(X_te)[:, 1]
p_lr = LogisticRegression(max_iter=1000).fit(X, y).predict_proba(X_te)[:, 1]
Step 2 · Reliability diagram, ECE and Brier score
from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss
import matplotlib.pyplot as plt
def ece(p, y, bins=10):
b = np.minimum((p * bins).astype(int), bins - 1)
return sum(abs(p[b == k].mean() - y[b == k].mean()) * (b == k).mean() for k in range(bins) if (b == k).any())
for name, p in [("TabPFN", p_tf), ("logistic", p_lr)]:
obs, pred = calibration_curve(y_te, p, n_bins=10)
plt.plot(pred, obs, marker="o", label=f"{name} ECE {ece(p, y_te):.3f} Brier {brier_score_loss(y_te, p):.3f}")
plt.plot([0, 1], [0, 1], "--", color="grey"); plt.legend(); plt.show()
Step 3 · Temperature scaling
Fit one number T on held-out rows by minimising log loss, then divide the log-odds by it. Accuracy does not change; the probabilities become honest.
from scipy.optimize import minimize_scalar
def temp_scale(p_cal, y_cal):
z = np.log(p_cal / (1 - p_cal + 1e-9) + 1e-9)
nll = lambda T: -np.mean(y_cal * np.log(1 / (1 + np.exp(-z / T))) + (1 - y_cal) * np.log(1 - 1 / (1 + np.exp(-z / T)) + 1e-12))
return minimize_scalar(nll, bounds=(0.2, 10), method="bounded").x