In-context classification: a tabular foundation model reads labelled rows
A tabular foundation model does not train on your table. It reads the labelled rows as context and answers new rows by comparison. Edit the table, change how many rows it may read, and see every attention weight.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The left side is the table: labelled context rows (shaded by how much the selected query attends to them) and query rows whose labels the model never sees. The plot shows two chosen columns, the attention lines from the selected query and, as shading, the predicted probability on a slice through that query row. The curve compares accuracy on 300 unseen rows as the context grows, against logistic regression and nearest neighbours.
Things to try
- Lower the context to 4 rows, then raise it to 60, and watch the curve.
- Edit the support-calls value of a context row and see the query probabilities move.
- Tick 'ignore learned feature relevance' to treat all columns alike.
- Make attention very sharp: the model behaves like a 1-nearest-neighbour.
How it works in real systems
TabPFN, TabICL and the large tabular foundation model on this site are transformers pre-trained on millions of tables; their attention is learned, multi-headed and runs over both rows and columns. The stand-in here keeps the mechanism (attend to labelled rows, read the answer off) and drops the scale, so it runs live in the browser.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Make the table
The same synthetic table as the simulation, so the numbers are comparable.
import numpy as np
rng = np.random.default_rng(7)
def churn(n):
t = rng.integers(1, 73, n); f = rng.integers(20, 121, n); c = np.minimum(9, rng.exponential(2.2, n).astype(int))
a = (rng.random(n) < 0.4).astype(int); u = rng.integers(1, 81, n)
z = 1.1 - 0.05 * t + 0.025 * (f - 60) + 0.45 * c - 1.6 * a - 0.015 * u + 1.1 * ((c > 5) & (a == 0))
y = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
return np.c_[t, f, c, a, u].astype(float), y
X, y = churn(60); X_test, y_test = churn(300)
Step 2 · In-context learning from scratch
The mechanism in a dozen lines: standardise with the context's statistics, weight features by how they relate to the label, attend from each query row to every labelled row, read the label off the attended rows. No weights are fitted.
def icl_proba(Xc, yc, Xq, tau=0.5):
mu, sd = Xc.mean(0), Xc.std(0) + 1e-9
Zc, Zq = (Xc - mu) / sd, (Xq - mu) / sd
w = np.abs([np.corrcoef(Zc[:, f], yc)[0, 1] for f in range(Xc.shape[1])])
w = 0.25 + 0.75 * w / w.mean() # feature relevance read from the context
d = (((Zq[:, None, :] - Zc[None]) ** 2) * w).sum(-1) / w.sum() # (queries, context rows)
a = np.exp(-(d - d.min(1, keepdims=True)) / tau); a /= a.sum(1, keepdims=True) # attention
return a @ np.eye(2)[yc], a
p, att = icl_proba(X, y, X_test)
print("accuracy", ((p[:, 1] > .5) == y_test).mean())
Step 3 · A real tabular foundation model
TabPFN is an open tabular foundation model that works this way, with a transformer trained on millions of synthetic tables. fit only stores the context; the prediction is one forward pass.
# pip install tabpfn (runs on CPU for small tables; a GPU is much faster)
from tabpfn import TabPFNClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, log_loss
clf = TabPFNClassifier()
clf.fit(X, y) # stores the 60 labelled rows: no gradient steps
proba = clf.predict_proba(X_test)
print("TabPFN ", accuracy_score(y_test, proba.argmax(1)), log_loss(y_test, proba))
lr = LogisticRegression(max_iter=1000).fit(X, y)
print("logistic", accuracy_score(y_test, lr.predict(X_test)), log_loss(y_test, lr.predict_proba(X_test)))
Step 4 · Accuracy as the context grows
for n in (8, 20, 40, 60):
clf = TabPFNClassifier().fit(X[:n], y[:n])
print(n, "rows:", round(accuracy_score(y_test, clf.predict(X_test)), 3))
The curve is the in-context learning curve: more labelled rows, better answers, still without training.