Özer Cem KELAHMETKELAHMET
AI Simulations · Large tabular foundation model

Large tables: choosing the context when a table outgrows it

Attention cost grows with the number of rows the model reads, and real tables can be far larger than any context. Which rows should it read? Each strategy gets the same budget.

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

The chart shows accuracy on 400 test rows for budgets from 16 to 512 rows per prediction, for four ways of choosing the rows, with the accuracy of reading all 3,000 rows as a dashed line. The panel compares the strategies at 64 rows.

Things to try

  • Run and compare retrieval with random subsets at 16 rows.
  • Look at how close the 512-row strategies get to the full table.
  • Notice what ensembling buys at small budgets.

How it works in real systems

TabPFN handles large tables by subsampling the context and ensembling several subsets; retrieval-based variants pick the context per query. Both trade a little accuracy for a large cut in compute.

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · A table larger than the context

import numpy as np
rng = np.random.default_rng(0)
def sensors(n):
    t = rng.normal(55, 9, n); v = np.maximum(.2, rng.normal(2.2, 1.1, n)); r = rng.uniform(1500, 3000, n); a = rng.integers(0, 97, n)
    z = -3.2 + .11 * (t - 55) + 1.1 * (v - 2.2) + .0009 * (r - 2250) + .025 * a + 2 * ((t > 65) & (v > 3))
    return np.c_[t, v, r, a], (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
X, y = sensors(50_000); X_te, y_te = sensors(2_000)

Step 2 · Random subsets, balanced subsets, ensembles

from tabpfn import TabPFNClassifier
from sklearn.metrics import accuracy_score
def subset(n, balanced=False):
    if not balanced: return rng.choice(len(X), n, replace=False)
    pos, neg = np.flatnonzero(y == 1), np.flatnonzero(y == 0)
    return np.r_[rng.choice(pos, n // 2), rng.choice(neg, n - n // 2)]
for n in (128, 512, 2048):
    idx = subset(n); acc_r = accuracy_score(y_te, TabPFNClassifier().fit(X[idx], y[idx]).predict(X_te))
    ens = np.mean([TabPFNClassifier().fit(X[i], y[i]).predict_proba(X_te)[:, 1] for i in (subset(n) for _ in range(4))], axis=0)
    print(n, "random", round(acc_r, 3), "ensemble of 4", round(accuracy_score(y_te, ens > .5), 3))

Step 3 · Retrieve the context per query

Group the test rows by their nearest training neighbours, so each group shares one retrieved context.

from sklearn.neighbors import NearestNeighbors
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
sc = StandardScaler().fit(X); nn = NearestNeighbors(n_neighbors=512).fit(sc.transform(X))
groups = KMeans(20, n_init=4, random_state=0).fit_predict(sc.transform(X_te))
pred = np.zeros(len(X_te))
for g in range(20):
    q = groups == g
    ctx = np.unique(nn.kneighbors(sc.transform(X_te[q]), return_distance=False)[:, :64].ravel())[:1024]
    pred[q] = TabPFNClassifier().fit(X[ctx], y[ctx]).predict_proba(X_te[q])[:, 1]
print("retrieval", accuracy_score(y_te, pred > .5))

More simulations

← Missingness shift · All AI Simulations · Any table →

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open