Özer Cem KELAHMETKELAHMET
Large tabular foundation model · Lab

Large tabular foundation model lab: in-context learning, uncertainty and imputation, live

Eleven live experiments in one place: how a large tabular foundation model predicts by reading labelled rows, scales to large tables and any schema, predicts calibrated distributions, fills missing cells, stays robust when missingness changes, and finds anomalies.

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What is in the lab

  • Classification and Regression: in-context learning, where labelled rows are the context and new rows attend to them; no training on the table.
  • Probabilistic regression: a full predictive distribution, scored with coverage, negative log-likelihood and CRPS.
  • Imputation and Masked value: filling hidden cells from the observed cells of all rows, compared with mean, kNN, MICE and MIDAS.
  • MIDAS: a denoising autoencoder trained in your browser, drawing several imputations to show their uncertainty.
  • Missingness shift (MIRRAMS): models trained at one missing rate and tested as the rate and the mechanism change.
  • Large tables: choosing which rows to read when the table is bigger than the context: random, balanced, ensembled or retrieved.
  • Any table: one predictor on unrelated tables, unchanged by column order, names and units.
  • Calibration: reliability diagrams, ECE and temperature scaling.
  • Anomalies: rows found by how improbable their cells are under masked-cell prediction.

How the simulation relates to the real model

Every number is computed live. The predictor is a small stand-in that keeps the mechanism of a large tabular foundation model, a query row attending to the labelled rows of the same table and reading its answer off them, and drops the scale: the real model is a transformer pre-trained on very large collections of tables, with learned attention over both rows and columns. See In-context prediction, Probabilistic regression benchmark, Imputation benchmark and Missing-data robustness for the model itself.

Each experiment also has its own page among the AI Simulations, with things to try and a code lab.

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · Make the table

The same synthetic table as the simulation, so the numbers are comparable.

import numpy as np
rng = np.random.default_rng(7)
def churn(n):
    t = rng.integers(1, 73, n); f = rng.integers(20, 121, n); c = np.minimum(9, rng.exponential(2.2, n).astype(int))
    a = (rng.random(n) < 0.4).astype(int); u = rng.integers(1, 81, n)
    z = 1.1 - 0.05 * t + 0.025 * (f - 60) + 0.45 * c - 1.6 * a - 0.015 * u + 1.1 * ((c > 5) & (a == 0))
    y = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
    return np.c_[t, f, c, a, u].astype(float), y
X, y = churn(60); X_test, y_test = churn(300)

Step 2 · In-context learning from scratch

The mechanism in a dozen lines: standardise with the context's statistics, weight features by how they relate to the label, attend from each query row to every labelled row, read the label off the attended rows. No weights are fitted.

def icl_proba(Xc, yc, Xq, tau=0.5):
    mu, sd = Xc.mean(0), Xc.std(0) + 1e-9
    Zc, Zq = (Xc - mu) / sd, (Xq - mu) / sd
    w = np.abs([np.corrcoef(Zc[:, f], yc)[0, 1] for f in range(Xc.shape[1])])
    w = 0.25 + 0.75 * w / w.mean()                                  # feature relevance read from the context
    d = (((Zq[:, None, :] - Zc[None]) ** 2) * w).sum(-1) / w.sum()   # (queries, context rows)
    a = np.exp(-(d - d.min(1, keepdims=True)) / tau); a /= a.sum(1, keepdims=True)   # attention
    return a @ np.eye(2)[yc], a
p, att = icl_proba(X, y, X_test)
print("accuracy", ((p[:, 1] > .5) == y_test).mean())

Step 3 · A real tabular foundation model

TabPFN is an open tabular foundation model that works this way, with a transformer trained on millions of synthetic tables. fit only stores the context; the prediction is one forward pass.

# pip install tabpfn          (runs on CPU for small tables; a GPU is much faster)
from tabpfn import TabPFNClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, log_loss
clf = TabPFNClassifier()
clf.fit(X, y)                               # stores the 60 labelled rows: no gradient steps
proba = clf.predict_proba(X_test)
print("TabPFN  ", accuracy_score(y_test, proba.argmax(1)), log_loss(y_test, proba))
lr = LogisticRegression(max_iter=1000).fit(X, y)
print("logistic", accuracy_score(y_test, lr.predict(X_test)), log_loss(y_test, lr.predict_proba(X_test)))

Step 4 · Data whose noise changes

import numpy as np
rng = np.random.default_rng(0)
x = rng.random(300); y = np.sin(5 * x) * .6 + x + rng.normal(0, .05 + .45 * x)
X, Xt, Y, Yt = x[:70, None], x[70:, None], y[:70], y[70:]

Step 5 · A full distribution with TabPFN

TabPFN's regressor predicts a distribution over the target (a fine histogram, the "bar distribution"), so quantiles come out directly.

from tabpfn import TabPFNRegressor
reg = TabPFNRegressor().fit(X, Y)
q10, q50, q90 = reg.predict(Xt, output_type="quantiles", quantiles=[0.1, 0.5, 0.9])
print("80% interval covers", ((Yt >= q10) & (Yt <= q90)).mean())

Step 6 · Hide cells you know

import numpy as np
rng = np.random.default_rng(11)
def houses(n):
    s = rng.integers(35, 181, n); ro = np.clip(np.round(s / 32 + rng.normal(0, .6, n)), 1, 6); ag = rng.integers(0, 81, n); d = np.round(rng.uniform(.5, 25, n), 1)
    p = 2.4 * s * (1 + .45 * np.exp(-d / 4)) - .9 * ag + 14 * ro + 40
    return np.c_[s, ro, ag, d].astype(float), np.round(p * (1 + rng.normal(0, .07, n)))
X, y = houses(60); X_test, y_test = houses(300)
T = np.c_[X, y]                                          # five columns, price included
mask = rng.random(T.shape) < 0.2                         # MCAR: 20% of cells hidden
T_obs = T.copy(); T_obs[mask] = np.nan
def nrmse(T_hat): return np.sqrt(np.mean(((T_hat - T) / T.std(0))[mask] ** 2))

Step 7 · Impute with a large tabular foundation model

Treat each column with gaps as a target: the rows where it is present are the context, the rows where it is missing are the queries. TabPFN accepts missing values in the features, so nothing else needs filling first.

from tabpfn import TabPFNRegressor
T_tf = T_obs.copy()
for f in range(T.shape[1]):
    miss = np.isnan(T_obs[:, f])
    if not miss.any(): continue
    feats = np.delete(T_obs, f, axis=1)
    reg = TabPFNRegressor().fit(feats[~miss], T_obs[~miss, f])
    T_tf[miss, f] = reg.predict(feats[miss])
print("TabPFN", round(nrmse(T_tf), 3))

Step 8 · Train at one missing rate, test at many

The same synthetic table as the simulation, so the numbers are comparable.

import numpy as np
rng = np.random.default_rng(7)
def churn(n):
    t = rng.integers(1, 73, n); f = rng.integers(20, 121, n); c = np.minimum(9, rng.exponential(2.2, n).astype(int))
    a = (rng.random(n) < 0.4).astype(int); u = rng.integers(1, 81, n)
    z = 1.1 - 0.05 * t + 0.025 * (f - 60) + 0.45 * c - 1.6 * a - 0.015 * u + 1.1 * ((c > 5) & (a == 0))
    y = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
    return np.c_[t, f, c, a, u].astype(float), y
X, y = churn(60); X_test, y_test = churn(300)
X_tr, y_tr = churn(2000); X_te, y_te = churn(2000)
def hide(X, p, seed): r = np.random.default_rng(seed); X = X.copy(); X[r.random(X.shape) < p] = np.nan; return X

Step 9 · Four strategies

from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier
p_train = 0.0
Xtr = hide(X_tr, p_train, 1)
aug = np.vstack([hide(X_tr, p, 10 + i) for i, p in enumerate(np.linspace(0, .7, 8))]); y_aug = np.tile(y_tr, 8)
models = {
  "mean imputation": make_pipeline(SimpleImputer(), StandardScaler(), LogisticRegression(max_iter=1000)).fit(Xtr, y_tr),
  "mean + indicators": make_pipeline(SimpleImputer(add_indicator=True), StandardScaler(), LogisticRegression(max_iter=1000)).fit(Xtr, y_tr),
  "mask augmentation": make_pipeline(SimpleImputer(add_indicator=True), StandardScaler(), LogisticRegression(max_iter=1000)).fit(aug, y_aug),
  "boosting, native NaN": HistGradientBoostingClassifier().fit(aug, y_aug),
}

Step 10 · Sweep the test-time rate

for p in (0, .2, .4, .6):
    Xt = hide(X_te, p, 99)
    print(f"{int(p*100):2d}% missing", {k: round(m.score(Xt, y_te), 3) for k, m in models.items()})

Add a large tabular foundation model (TabPFN accepts NaN) to the dictionary to compare; with p_train = 0 the mean-imputation model degrades fastest, which is the failure MIRRAMS-style evaluation is designed to expose.

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open