# A mini tabular foundation model: predict any small table without training on it

Classical machine learning trains a new model for every table. A tabular foundation model is trained once, on millions of synthetic tables, and then predicts a new table in a single forward pass by reading its rows as context. This is a miniature one, a million parameters, small enough to run in your browser on well-known datasets or on your own CSV.

<div data-sim="tabfm"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

Pick a table and a target column. The rows are shuffled: 70% are given to the model as context, with their answers, and the other 30% are held out. The model reads all rows at once; every held-out row attends to the context rows and the model outputs, for each, a probability for every class or a whole distribution over the target's value. Nothing is trained on your table.

The three cards compare it with two classical methods computed in your browser on the same split: five nearest neighbours, and logistic or linear regression trained for 400 gradient steps. The left panel shows the first test rows with the model's probabilities, or for numbers an 80% interval and the true value. The right panel hides some cells of one column and lets the same model fill them in, by treating that column as the target, which is exactly how tabular foundation models impute.

## Things to try

- On Iris, press New split a few times: accuracy moves by several points between splits, which is why single numbers mislead.
- On Diabetes progression, look at how often the true value falls inside the 80% intervals: a well-calibrated model is close to 80%.
- Hide half of one column and compare the error with the column mean: the improvement shows how much the other columns explain it.
- Load your own CSV: keep the target in the last column, or choose it in the menu.

## How it works in real systems

The network was trained with the recipe of prior-fitted networks: draw a random synthetic table from a prior over data-generating processes (random causal networks, threshold trees, linear relations, noise), hide the labels of some rows, and learn to predict them from the rest. After 352,000 such tables, including tables whose classes form clusters, it has learned a general way of reading a table. Large tabular foundation models follow the same principle with far richer priors, real-data training stages, many millions or billions of parameters and support for thousands of rows and hundreds of columns; they are the strongest methods today on many small and medium tables. This miniature is limited to eight columns, three classes and 130 rows. Averaged over 20 random splits it reaches 96.0% on Iris (5-nearest neighbours 94.9%, logistic regression 96.2%), 97.7% on Wine (96.4% and 96.9%) and 94.4% on Breast cancer (95.5% and 95.4%); on Diabetes progression its error (RMSE 59.8) is between nearest neighbours (61.8) and linear regression (56.7), and its 80% intervals cover 73% of the true values, a little too narrow.





<!-- simx:start -->
## Key formulas

$$p\big(y_{\text{test}}\mid x_{\text{test}},\ \mathcal D_{\text{context}}\big)\approx q_\theta\big(y_{\text{test}}\mid x_{\text{test}},\ \mathcal D_{\text{context}}\big)$$

A prior-fitted network learns, from synthetic tables, to approximate the posterior predictive in one forward pass.

$$\theta^{\ast}=\arg\min_\theta\ \mathbb E_{\mathcal D\sim\text{prior}}\big[-\log q_\theta(y_{\text{test}}\mid x_{\text{test}},\mathcal D_{\text{context}})\big]$$

Training: draw a synthetic table, hide some labels, learn to predict them. No real data involved.

$$\mathrm{RMSE}=\sqrt{\tfrac1n\textstyle\sum_i(\hat y_i-y_i)^2}$$

The score for numeric targets and for filled-in cells.

## Common mistakes

- Reading this small model's scores as those of a large tabular foundation model: the method is the same, the size is not.
- Comparing on a single split: press New split a few times before drawing a conclusion.
- Feeding more columns than the model reads: this one uses up to eight and up to 130 rows.

## Real-world numbers

- This model: 1.0 million parameters, trained for 22,000 steps on 352,000 freshly drawn synthetic tables and no real data.
- Averaged over 20 splits: Iris 96.0%, Wine 97.7%, Breast cancer 94.4%, on par with 5-nearest neighbours and logistic regression.
- TabPFN (2022) introduced prior-fitted networks for small tables; later tabular foundation models scale the idea to larger tables and many more parameters.

## Further reading

- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://arxiv.org/abs/2207.01848)
- [Transformers Can Do Bayesian Inference (prior-fitted networks)](https://arxiv.org/abs/2112.10510)
- [Accurate predictions on small data with a tabular foundation model (Nature, 2025)](https://www.nature.com/articles/s41586-024-08328-6)

<!-- simx:end -->

## Code lab

The training loop of a prior-fitted network, cut to its core, in PyTorch.

<div class="cl-start"></div>

### Step 1 · A prior: random synthetic tables

```python
import numpy as np, torch
def table(n=100, f=5):
    X = np.random.randn(n, f)
    W = np.random.randn(f, 16) / np.sqrt(f)
    y = np.tanh(X @ W) @ np.random.randn(16) + 0.2 * np.random.randn(n)   # a random causal mechanism
    k = np.random.randint(2, 4)
    cls = np.digitize(y, np.quantile(y, np.linspace(0, 1, k + 1)[1:-1]))  # 2 or 3 classes
    return (X - X.mean(0)) / X.std(0), cls
```

### Step 2 · Rows attend to the labelled context rows

```python
import torch.nn as nn, torch.nn.functional as F
class PFN(nn.Module):
    def __init__(s, F_=8, D=128, L=4, H=4):
        super().__init__(); s.inp = nn.Linear(F_, D); s.lab = nn.Embedding(3, D); s.H = H
        s.layers = nn.ModuleList(nn.TransformerEncoderLayer(D, H, 4 * D, batch_first=True) for _ in range(L))
        s.out = nn.Linear(D, 3)
    def forward(s, X, y_ctx, n_ctx):
        h = s.inp(X); h = torch.cat([h[:, :n_ctx] + s.lab(y_ctx), h[:, n_ctx:]], 1)
        N = X.shape[1]; mask = torch.ones(N, N, dtype=torch.bool); mask[:, :n_ctx] = False   # True = may not attend
        for l in s.layers: h = l(h, src_mask=mask)
        return s.out(h[:, n_ctx:])
```

### Step 3 · Train on fresh tables, never on real data

```python
net = PFN(); opt = torch.optim.AdamW(net.parameters(), 1e-3)
for step in range(5000):
    batch = [table() for _ in range(16)]
    X = torch.tensor(np.stack([np.pad(b[0], ((0, 0), (0, 3))) for b in batch]), dtype=torch.float32)
    y = torch.tensor(np.stack([b[1] for b in batch]))
    out = net(X, y[:, :70], 70)
    loss = F.cross_entropy(out.reshape(-1, 3), y[:, 70:].reshape(-1))
    opt.zero_grad(); loss.backward(); opt.step()
```

### Step 4 · Predict a real table in one forward pass

```python
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True); idx = np.random.permutation(150)[:100]
X = (X[idx] - X[idx].mean(0)) / X[idx].std(0); y = y[idx]
Xt = torch.tensor(np.pad(X, ((0, 0), (0, 4))), dtype=torch.float32)[None]
with torch.no_grad():
    p = net(Xt, torch.tensor(y[:70])[None], 70).softmax(-1)[0]
print("accuracy", (p.argmax(-1).numpy() == y[70:]).mean())
```

<div class="cl-end"></div>

## More simulations

[All AI Simulations](/ai-simulations) · [Tabular model: classification →](/ai-simulations/tfm-classification) · [Imputation →](/ai-simulations/tfm-imputation)
