Probabilistic regression: the whole predictive distribution, scored properly
A single number hides how sure the model is. A tabular foundation model predicts the whole distribution of the target. Watch it widen where data are noisy, split in two when there are two outcomes, and get scored with proper metrics.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The shading is the predicted density of y at every x; the solid line is the median and the dashed lines the 10% and 90% quantiles. The grey dashed lines are a Gaussian baseline with one constant spread. Hover to move the slice shown on the lower left as a histogram; the table scores both on 400 unseen points.
Things to try
- Pick 'two possible outcomes': the Gaussian puts its peak where no data fall.
- Pick 'noise that grows with x' and compare interval coverage.
- Click to add points in an empty region and watch the uncertainty there.
How it works in real systems
TabPFN predicts a 'bar distribution' over the target, a fine histogram learned in pre-training; probabilistic benchmarks score models with CRPS, negative log-likelihood and interval coverage rather than RMSE.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Data whose noise changes
import numpy as np
rng = np.random.default_rng(0)
x = rng.random(300); y = np.sin(5 * x) * .6 + x + rng.normal(0, .05 + .45 * x)
X, Xt, Y, Yt = x[:70, None], x[70:, None], y[:70], y[70:]
Step 2 · A full distribution with TabPFN
TabPFN's regressor predicts a distribution over the target (a fine histogram, the "bar distribution"), so quantiles come out directly.
from tabpfn import TabPFNRegressor
reg = TabPFNRegressor().fit(X, Y)
q10, q50, q90 = reg.predict(Xt, output_type="quantiles", quantiles=[0.1, 0.5, 0.9])
print("80% interval covers", ((Yt >= q10) & (Yt <= q90)).mean())
Step 3 · Score it properly
Coverage alone can be gamed; use proper scores. The pinball (quantile) loss averaged over many quantiles approximates CRPS.
def pinball(y, qhat, tau): d = y - qhat; return np.mean(np.maximum(tau * d, (tau - 1) * d))
taus = np.linspace(.05, .95, 19)
qs = reg.predict(Xt, output_type="quantiles", quantiles=list(taus))
print("mean pinball loss", np.mean([pinball(Yt, q, t) for q, t in zip(qs, taus)]))
# baseline: a line plus one constant sigma
from scipy.stats import norm
b = np.polyfit(X[:, 0], Y, 1); sig = np.std(Y - np.polyval(b, X[:, 0]))
qb = [np.polyval(b, Xt[:, 0]) + norm.ppf(t) * sig for t in taus]
print("baseline pinball", np.mean([pinball(Yt, q, t) for q, t in zip(qb, taus)]))