# Calibration: do a tabular model's probabilities mean what they say?

Among all the rows a model calls 70% likely, about 70% should be positive. Accuracy cannot tell you that; a reliability diagram can. Make the model overconfident, then fix it with one number.

<div data-sim="tfm" data-v="cal"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

The diagram plots the average predicted probability in each of ten bins against how often the event really happened; the diagonal is perfect calibration and dot size is the number of rows. The table reports ECE, Brier score, log loss and accuracy on 1,500 unseen rows.

## Things to try

- Make attention very sharp: the curve flattens away from the diagonal (overconfidence).
- Switch on temperature scaling and watch ECE drop while accuracy stays the same.
- Compare with logistic regression.

## How it works in real systems

scikit-learn provides calibration_curve, brier_score_loss and CalibratedClassifierCV (Platt scaling and isotonic regression); temperature scaling is the standard fix for neural network classifiers.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Data and two models

The same synthetic table as the simulation, so the numbers are comparable.

```python
import numpy as np
rng = np.random.default_rng(7)
def churn(n):
    t = rng.integers(1, 73, n); f = rng.integers(20, 121, n); c = np.minimum(9, rng.exponential(2.2, n).astype(int))
    a = (rng.random(n) < 0.4).astype(int); u = rng.integers(1, 81, n)
    z = 1.1 - 0.05 * t + 0.025 * (f - 60) + 0.45 * c - 1.6 * a - 0.015 * u + 1.1 * ((c > 5) & (a == 0))
    y = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
    return np.c_[t, f, c, a, u].astype(float), y
X, y = churn(60); X_test, y_test = churn(300)
```

```python
X, y = churn(500); X_te, y_te = churn(3000)
from tabpfn import TabPFNClassifier
from sklearn.linear_model import LogisticRegression
p_tf = TabPFNClassifier().fit(X, y).predict_proba(X_te)[:, 1]
p_lr = LogisticRegression(max_iter=1000).fit(X, y).predict_proba(X_te)[:, 1]
```

### Step 2 · Reliability diagram, ECE and Brier score

```python
from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss
import matplotlib.pyplot as plt
def ece(p, y, bins=10):
    b = np.minimum((p * bins).astype(int), bins - 1)
    return sum(abs(p[b == k].mean() - y[b == k].mean()) * (b == k).mean() for k in range(bins) if (b == k).any())
for name, p in [("TabPFN", p_tf), ("logistic", p_lr)]:
    obs, pred = calibration_curve(y_te, p, n_bins=10)
    plt.plot(pred, obs, marker="o", label=f"{name}  ECE {ece(p, y_te):.3f}  Brier {brier_score_loss(y_te, p):.3f}")
plt.plot([0, 1], [0, 1], "--", color="grey"); plt.legend(); plt.show()
```

### Step 3 · Temperature scaling

Fit one number T on held-out rows by minimising log loss, then divide the log-odds by it. Accuracy does not change; the probabilities become honest.

```python
from scipy.optimize import minimize_scalar
def temp_scale(p_cal, y_cal):
    z = np.log(p_cal / (1 - p_cal + 1e-9) + 1e-9)
    nll = lambda T: -np.mean(y_cal * np.log(1 / (1 + np.exp(-z / T))) + (1 - y_cal) * np.log(1 - 1 / (1 + np.exp(-z / T)) + 1e-12))
    return minimize_scalar(nll, bounds=(0.2, 10), method="bounded").x
```

<div class="cl-end"></div>

## More simulations

[← Any table](/ai-simulations/tfm-any-table) · [All AI Simulations](/ai-simulations) · [Anomaly detection →](/ai-simulations/tfm-anomaly-detection)
