Large tabular foundation model lab: in-context learning, uncertainty and imputation, live
Eleven live experiments in one place: how a large tabular foundation model predicts by reading labelled rows, scales to large tables and any schema, predicts calibrated distributions, fills missing cells, stays robust when missingness changes, and finds anomalies.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What is in the lab
- Classification and Regression: in-context learning, where labelled rows are the context and new rows attend to them; no training on the table.
- Probabilistic regression: a full predictive distribution, scored with coverage, negative log-likelihood and CRPS.
- Imputation and Masked value: filling hidden cells from the observed cells of all rows, compared with mean, kNN, MICE and MIDAS.
- MIDAS: a denoising autoencoder trained in your browser, drawing several imputations to show their uncertainty.
- Missingness shift (MIRRAMS): models trained at one missing rate and tested as the rate and the mechanism change.
- Large tables: choosing which rows to read when the table is bigger than the context: random, balanced, ensembled or retrieved.
- Any table: one predictor on unrelated tables, unchanged by column order, names and units.
- Calibration: reliability diagrams, ECE and temperature scaling.
- Anomalies: rows found by how improbable their cells are under masked-cell prediction.
How the simulation relates to the real model
Every number is computed live. The predictor is a small stand-in that keeps the mechanism of a large tabular foundation model, a query row attending to the labelled rows of the same table and reading its answer off them, and drops the scale: the real model is a transformer pre-trained on very large collections of tables, with learned attention over both rows and columns. See In-context prediction, Probabilistic regression benchmark, Imputation benchmark and Missing-data robustness for the model itself.
Each experiment also has its own page among the AI Simulations, with things to try and a code lab.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Make the table
The same synthetic table as the simulation, so the numbers are comparable.
import numpy as np
rng = np.random.default_rng(7)
def churn(n):
t = rng.integers(1, 73, n); f = rng.integers(20, 121, n); c = np.minimum(9, rng.exponential(2.2, n).astype(int))
a = (rng.random(n) < 0.4).astype(int); u = rng.integers(1, 81, n)
z = 1.1 - 0.05 * t + 0.025 * (f - 60) + 0.45 * c - 1.6 * a - 0.015 * u + 1.1 * ((c > 5) & (a == 0))
y = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
return np.c_[t, f, c, a, u].astype(float), y
X, y = churn(60); X_test, y_test = churn(300)
Step 2 · In-context learning from scratch
The mechanism in a dozen lines: standardise with the context's statistics, weight features by how they relate to the label, attend from each query row to every labelled row, read the label off the attended rows. No weights are fitted.
def icl_proba(Xc, yc, Xq, tau=0.5):
mu, sd = Xc.mean(0), Xc.std(0) + 1e-9
Zc, Zq = (Xc - mu) / sd, (Xq - mu) / sd
w = np.abs([np.corrcoef(Zc[:, f], yc)[0, 1] for f in range(Xc.shape[1])])
w = 0.25 + 0.75 * w / w.mean() # feature relevance read from the context
d = (((Zq[:, None, :] - Zc[None]) ** 2) * w).sum(-1) / w.sum() # (queries, context rows)
a = np.exp(-(d - d.min(1, keepdims=True)) / tau); a /= a.sum(1, keepdims=True) # attention
return a @ np.eye(2)[yc], a
p, att = icl_proba(X, y, X_test)
print("accuracy", ((p[:, 1] > .5) == y_test).mean())
Step 3 · A real tabular foundation model
TabPFN is an open tabular foundation model that works this way, with a transformer trained on millions of synthetic tables. fit only stores the context; the prediction is one forward pass.
# pip install tabpfn (runs on CPU for small tables; a GPU is much faster)
from tabpfn import TabPFNClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, log_loss
clf = TabPFNClassifier()
clf.fit(X, y) # stores the 60 labelled rows: no gradient steps
proba = clf.predict_proba(X_test)
print("TabPFN ", accuracy_score(y_test, proba.argmax(1)), log_loss(y_test, proba))
lr = LogisticRegression(max_iter=1000).fit(X, y)
print("logistic", accuracy_score(y_test, lr.predict(X_test)), log_loss(y_test, lr.predict_proba(X_test)))
Step 4 · Data whose noise changes
import numpy as np
rng = np.random.default_rng(0)
x = rng.random(300); y = np.sin(5 * x) * .6 + x + rng.normal(0, .05 + .45 * x)
X, Xt, Y, Yt = x[:70, None], x[70:, None], y[:70], y[70:]
Step 5 · A full distribution with TabPFN
TabPFN's regressor predicts a distribution over the target (a fine histogram, the "bar distribution"), so quantiles come out directly.
from tabpfn import TabPFNRegressor
reg = TabPFNRegressor().fit(X, Y)
q10, q50, q90 = reg.predict(Xt, output_type="quantiles", quantiles=[0.1, 0.5, 0.9])
print("80% interval covers", ((Yt >= q10) & (Yt <= q90)).mean())
Step 6 · Hide cells you know
import numpy as np
rng = np.random.default_rng(11)
def houses(n):
s = rng.integers(35, 181, n); ro = np.clip(np.round(s / 32 + rng.normal(0, .6, n)), 1, 6); ag = rng.integers(0, 81, n); d = np.round(rng.uniform(.5, 25, n), 1)
p = 2.4 * s * (1 + .45 * np.exp(-d / 4)) - .9 * ag + 14 * ro + 40
return np.c_[s, ro, ag, d].astype(float), np.round(p * (1 + rng.normal(0, .07, n)))
X, y = houses(60); X_test, y_test = houses(300)
T = np.c_[X, y] # five columns, price included
mask = rng.random(T.shape) < 0.2 # MCAR: 20% of cells hidden
T_obs = T.copy(); T_obs[mask] = np.nan
def nrmse(T_hat): return np.sqrt(np.mean(((T_hat - T) / T.std(0))[mask] ** 2))
Step 7 · Impute with a large tabular foundation model
Treat each column with gaps as a target: the rows where it is present are the context, the rows where it is missing are the queries. TabPFN accepts missing values in the features, so nothing else needs filling first.
from tabpfn import TabPFNRegressor
T_tf = T_obs.copy()
for f in range(T.shape[1]):
miss = np.isnan(T_obs[:, f])
if not miss.any(): continue
feats = np.delete(T_obs, f, axis=1)
reg = TabPFNRegressor().fit(feats[~miss], T_obs[~miss, f])
T_tf[miss, f] = reg.predict(feats[miss])
print("TabPFN", round(nrmse(T_tf), 3))
Step 8 · Train at one missing rate, test at many
The same synthetic table as the simulation, so the numbers are comparable.
import numpy as np
rng = np.random.default_rng(7)
def churn(n):
t = rng.integers(1, 73, n); f = rng.integers(20, 121, n); c = np.minimum(9, rng.exponential(2.2, n).astype(int))
a = (rng.random(n) < 0.4).astype(int); u = rng.integers(1, 81, n)
z = 1.1 - 0.05 * t + 0.025 * (f - 60) + 0.45 * c - 1.6 * a - 0.015 * u + 1.1 * ((c > 5) & (a == 0))
y = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
return np.c_[t, f, c, a, u].astype(float), y
X, y = churn(60); X_test, y_test = churn(300)
X_tr, y_tr = churn(2000); X_te, y_te = churn(2000)
def hide(X, p, seed): r = np.random.default_rng(seed); X = X.copy(); X[r.random(X.shape) < p] = np.nan; return X
Step 9 · Four strategies
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier
p_train = 0.0
Xtr = hide(X_tr, p_train, 1)
aug = np.vstack([hide(X_tr, p, 10 + i) for i, p in enumerate(np.linspace(0, .7, 8))]); y_aug = np.tile(y_tr, 8)
models = {
"mean imputation": make_pipeline(SimpleImputer(), StandardScaler(), LogisticRegression(max_iter=1000)).fit(Xtr, y_tr),
"mean + indicators": make_pipeline(SimpleImputer(add_indicator=True), StandardScaler(), LogisticRegression(max_iter=1000)).fit(Xtr, y_tr),
"mask augmentation": make_pipeline(SimpleImputer(add_indicator=True), StandardScaler(), LogisticRegression(max_iter=1000)).fit(aug, y_aug),
"boosting, native NaN": HistGradientBoostingClassifier().fit(aug, y_aug),
}
Step 10 · Sweep the test-time rate
for p in (0, .2, .4, .6):
Xt = hide(X_te, p, 99)
print(f"{int(p*100):2d}% missing", {k: round(m.score(Xt, y_te), 3) for k, m in models.items()})
Add a large tabular foundation model (TabPFN accepts NaN) to the dictionary to compare; with p_train = 0 the mean-imputation model degrades fastest, which is the failure MIRRAMS-style evaluation is designed to expose.