# Masked-value prediction: the pre-training task of a tabular foundation model

Hide one cell, predict it from its row and from the other rows: that is how tabular foundation models are pre-trained, and it is the same skill that later classifies, regresses and imputes.

<div data-sim="tfm" data-v="mask"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

Click any cell to mask it. The histogram is the predicted distribution for that cell, with the true value as a line. The table shades rows by how much they were read, and the panel lists the most-read rows with their values.

## Things to try

- Mask a price, then a size, then a number of rooms.
- Make attention sharp and see the distribution narrow around a few rows.
- Load a new table and test a few cells: how often is the truth inside the 80% interval?

## How it works in real systems

Masked-cell modelling is to tables what masked-word modelling is to language models: hide part of the input, predict it, learn from the error, across many tables.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Masked-cell prediction, from scratch

```python
import numpy as np
rng = np.random.default_rng(11)
def houses(n):
    s = rng.integers(35, 181, n); ro = np.clip(np.round(s / 32 + rng.normal(0, .6, n)), 1, 6); ag = rng.integers(0, 81, n); d = np.round(rng.uniform(.5, 25, n), 1)
    p = 2.4 * s * (1 + .45 * np.exp(-d / 4)) - .9 * ag + 14 * ro + 40
    return np.c_[s, ro, ag, d].astype(float), np.round(p * (1 + rng.normal(0, .07, n)))
X, y = houses(60); X_test, y_test = houses(300)
```

```python
T = np.c_[X, y]
def masked_cell(T, i, f, tau=0.3):
    others = np.delete(T, f, axis=1); keep = np.arange(len(T)) != i
    mu, sd = others[keep].mean(0), others[keep].std(0) + 1e-9
    d = (((others[keep] - others[i]) / sd) ** 2).mean(1)
    a = np.exp(-(d - d.min()) / tau); a /= a.sum()
    vals = T[keep, f]; mean = a @ vals
    return mean, np.sqrt(a @ (vals - mean) ** 2), a
print(masked_cell(T, 3, 4), "true", T[3, 4])
```

### Step 2 · The same skill, three tasks

Mask the label column and it is classification or regression; mask a feature and it is imputation. Pre-training a tabular foundation model means doing this on millions of tables:

```python
import torch, torch.nn as nn
def masked_cell_loss(model, batch, p=0.15):              # batch: (rows, columns), standardised
    m = torch.rand_like(batch) < p
    pred = model(batch.masked_fill(m, 0.0), m.float())   # the model sees the table with holes and the mask
    return nn.functional.mse_loss(pred[m], batch[m])
```

<div class="cl-end"></div>

## More simulations

[← Imputation](/ai-simulations/tfm-imputation) · [All AI Simulations](/ai-simulations) · [MIDAS →](/ai-simulations/midas-imputation)
