Özer Cem KELAHMETKELAHMET
AI Simulations · Large tabular foundation model

Masked-value prediction: the pre-training task of a tabular foundation model

Hide one cell, predict it from its row and from the other rows: that is how tabular foundation models are pre-trained, and it is the same skill that later classifies, regresses and imputes.

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

Click any cell to mask it. The histogram is the predicted distribution for that cell, with the true value as a line. The table shades rows by how much they were read, and the panel lists the most-read rows with their values.

Things to try

  • Mask a price, then a size, then a number of rooms.
  • Make attention sharp and see the distribution narrow around a few rows.
  • Load a new table and test a few cells: how often is the truth inside the 80% interval?

How it works in real systems

Masked-cell modelling is to tables what masked-word modelling is to language models: hide part of the input, predict it, learn from the error, across many tables.

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · Masked-cell prediction, from scratch

import numpy as np
rng = np.random.default_rng(11)
def houses(n):
    s = rng.integers(35, 181, n); ro = np.clip(np.round(s / 32 + rng.normal(0, .6, n)), 1, 6); ag = rng.integers(0, 81, n); d = np.round(rng.uniform(.5, 25, n), 1)
    p = 2.4 * s * (1 + .45 * np.exp(-d / 4)) - .9 * ag + 14 * ro + 40
    return np.c_[s, ro, ag, d].astype(float), np.round(p * (1 + rng.normal(0, .07, n)))
X, y = houses(60); X_test, y_test = houses(300)
T = np.c_[X, y]
def masked_cell(T, i, f, tau=0.3):
    others = np.delete(T, f, axis=1); keep = np.arange(len(T)) != i
    mu, sd = others[keep].mean(0), others[keep].std(0) + 1e-9
    d = (((others[keep] - others[i]) / sd) ** 2).mean(1)
    a = np.exp(-(d - d.min()) / tau); a /= a.sum()
    vals = T[keep, f]; mean = a @ vals
    return mean, np.sqrt(a @ (vals - mean) ** 2), a
print(masked_cell(T, 3, 4), "true", T[3, 4])

Step 2 · The same skill, three tasks

Mask the label column and it is classification or regression; mask a feature and it is imputation. Pre-training a tabular foundation model means doing this on millions of tables:

import torch, torch.nn as nn
def masked_cell_loss(model, batch, p=0.15):              # batch: (rows, columns), standardised
    m = torch.rand_like(batch) < p
    pred = model(batch.masked_fill(m, 0.0), m.float())   # the model sees the table with holes and the mask
    return nn.functional.mse_loss(pred[m], batch[m])

More simulations

← Imputation · All AI Simulations · MIDAS →

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open