Özer Cem KELAHMETKELAHMET
AI Simulations · Large tabular foundation model

Calibration: do a tabular model's probabilities mean what they say?

Among all the rows a model calls 70% likely, about 70% should be positive. Accuracy cannot tell you that; a reliability diagram can. Make the model overconfident, then fix it with one number.

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

The diagram plots the average predicted probability in each of ten bins against how often the event really happened; the diagonal is perfect calibration and dot size is the number of rows. The table reports ECE, Brier score, log loss and accuracy on 1,500 unseen rows.

Things to try

  • Make attention very sharp: the curve flattens away from the diagonal (overconfidence).
  • Switch on temperature scaling and watch ECE drop while accuracy stays the same.
  • Compare with logistic regression.

How it works in real systems

scikit-learn provides calibration_curve, brier_score_loss and CalibratedClassifierCV (Platt scaling and isotonic regression); temperature scaling is the standard fix for neural network classifiers.

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · Data and two models

The same synthetic table as the simulation, so the numbers are comparable.

import numpy as np
rng = np.random.default_rng(7)
def churn(n):
    t = rng.integers(1, 73, n); f = rng.integers(20, 121, n); c = np.minimum(9, rng.exponential(2.2, n).astype(int))
    a = (rng.random(n) < 0.4).astype(int); u = rng.integers(1, 81, n)
    z = 1.1 - 0.05 * t + 0.025 * (f - 60) + 0.45 * c - 1.6 * a - 0.015 * u + 1.1 * ((c > 5) & (a == 0))
    y = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
    return np.c_[t, f, c, a, u].astype(float), y
X, y = churn(60); X_test, y_test = churn(300)
X, y = churn(500); X_te, y_te = churn(3000)
from tabpfn import TabPFNClassifier
from sklearn.linear_model import LogisticRegression
p_tf = TabPFNClassifier().fit(X, y).predict_proba(X_te)[:, 1]
p_lr = LogisticRegression(max_iter=1000).fit(X, y).predict_proba(X_te)[:, 1]

Step 2 · Reliability diagram, ECE and Brier score

from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss
import matplotlib.pyplot as plt
def ece(p, y, bins=10):
    b = np.minimum((p * bins).astype(int), bins - 1)
    return sum(abs(p[b == k].mean() - y[b == k].mean()) * (b == k).mean() for k in range(bins) if (b == k).any())
for name, p in [("TabPFN", p_tf), ("logistic", p_lr)]:
    obs, pred = calibration_curve(y_te, p, n_bins=10)
    plt.plot(pred, obs, marker="o", label=f"{name}  ECE {ece(p, y_te):.3f}  Brier {brier_score_loss(y_te, p):.3f}")
plt.plot([0, 1], [0, 1], "--", color="grey"); plt.legend(); plt.show()

Step 3 · Temperature scaling

Fit one number T on held-out rows by minimising log loss, then divide the log-odds by it. Accuracy does not change; the probabilities become honest.

from scipy.optimize import minimize_scalar
def temp_scale(p_cal, y_cal):
    z = np.log(p_cal / (1 - p_cal + 1e-9) + 1e-9)
    nll = lambda T: -np.mean(y_cal * np.log(1 / (1 + np.exp(-z / T))) + (1 - y_cal) * np.log(1 - 1 / (1 + np.exp(-z / T)) + 1e-12))
    return minimize_scalar(nll, bounds=(0.2, 10), method="bounded").x

More simulations

← Any table · All AI Simulations · Anomaly detection →

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open