Masked-value prediction: the pre-training task of a tabular foundation model
Hide one cell, predict it from its row and from the other rows: that is how tabular foundation models are pre-trained, and it is the same skill that later classifies, regresses and imputes.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
Click any cell to mask it. The histogram is the predicted distribution for that cell, with the true value as a line. The table shades rows by how much they were read, and the panel lists the most-read rows with their values.
Things to try
- Mask a price, then a size, then a number of rooms.
- Make attention sharp and see the distribution narrow around a few rows.
- Load a new table and test a few cells: how often is the truth inside the 80% interval?
How it works in real systems
Masked-cell modelling is to tables what masked-word modelling is to language models: hide part of the input, predict it, learn from the error, across many tables.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Masked-cell prediction, from scratch
import numpy as np
rng = np.random.default_rng(11)
def houses(n):
s = rng.integers(35, 181, n); ro = np.clip(np.round(s / 32 + rng.normal(0, .6, n)), 1, 6); ag = rng.integers(0, 81, n); d = np.round(rng.uniform(.5, 25, n), 1)
p = 2.4 * s * (1 + .45 * np.exp(-d / 4)) - .9 * ag + 14 * ro + 40
return np.c_[s, ro, ag, d].astype(float), np.round(p * (1 + rng.normal(0, .07, n)))
X, y = houses(60); X_test, y_test = houses(300)
T = np.c_[X, y]
def masked_cell(T, i, f, tau=0.3):
others = np.delete(T, f, axis=1); keep = np.arange(len(T)) != i
mu, sd = others[keep].mean(0), others[keep].std(0) + 1e-9
d = (((others[keep] - others[i]) / sd) ** 2).mean(1)
a = np.exp(-(d - d.min()) / tau); a /= a.sum()
vals = T[keep, f]; mean = a @ vals
return mean, np.sqrt(a @ (vals - mean) ** 2), a
print(masked_cell(T, 3, 4), "true", T[3, 4])
Step 2 · The same skill, three tasks
Mask the label column and it is classification or regression; mask a feature and it is imputation. Pre-training a tabular foundation model means doing this on millions of tables:
import torch, torch.nn as nn
def masked_cell_loss(model, batch, p=0.15): # batch: (rows, columns), standardised
m = torch.rand_like(batch) < p
pred = model(batch.masked_fill(m, 0.0), m.float()) # the model sees the table with holes and the mask
return nn.functional.mse_loss(pred[m], batch[m])