Masked cell modeling

Overview

Masked cell modeling is a self-supervised training method for tables. A fraction of the cells in a table is hidden, and the model is trained to predict the hidden values from the cells that remain visible. It is the tabular cousin of masked language modeling, where words are hidden in a sentence and the model must guess them. No human labels are needed, because the table itself supplies the answers.

The method matters because it turns any table into free training data. A model that practices on millions of tables learns how columns relate, how values are distributed, and how rows resemble each other. That knowledge transfers to many tasks: filling missing values, spotting anomalies, classifying rows and recognizing what a table is about. Masked cell modeling is therefore a core pre-training recipe for tabular foundation models, and it works best with architectures that see both rows and columns, such as axial attention or Perceiver IO.

What it is

Masked cell modeling is a training objective, not an architecture. You pick any network that can read a table and produce a prediction for each cell. You hide some cells, ask the network to reconstruct them, and measure the error only on the hidden cells.

The three ingredients

1. A masking rule. Which cells are hidden, and how many.
2. A reconstruction head. How the network turns its internal state into a value for each hidden cell.
3. A loss. How the error is measured, which depends on the cell type.

Why "cell"

In language, the unit is a token. In images, it is a patch of pixels. In tables, the natural unit is a cell: the value at the crossing of one row and one column. Masking at the cell level asks the model to learn from the structure that is unique to tables.

How it differs from related ideas

• Masked language modeling hides tokens in a sequence, where order matters. Cells in a table have no natural order.
• Masked autoencoders for images hide patches and rebuild them. See MAE.
• Denoising autoencoders corrupt the input with noise and rebuild the clean version. See denoising autoencoders.
• Imputation fills missing values in real data. Masked cell modeling creates artificial missingness to practice imputation. See imputation networks.

Where it came from

The idea has roots in older statistical practice. Methods for filling missing data, such as iterative regression imputation, treated each column as a target and predicted it from the others. Denoising autoencoders in the 2000s taught networks to repair corrupted inputs.

The modern form was shaped by masked language modeling, introduced with BERT in 2018. The notion that you can learn rich representations just by guessing hidden pieces became a cornerstone of pre-training. In vision, masked image modeling followed.

Tabular researchers adapted the idea in the following years. TabTransformer (2020) used transformer layers on categorical features and explored pre-training. VIME (2020) proposed self-supervised learning for tabular data with a mask estimation task and a feature reconstruction task. SAINT (2021) combined row and column attention and used contrastive and denoising pre-training. TaBERT (2020) and related models handled tables that contain text. Later work scaled the idea to many tables and to foundation-style models, where masking is one of several training signals.

Why it took time to catch on in tables

• Tables mix numbers, categories, dates and text, so a single reconstruction loss does not fit.
• There is no order, so position-based tricks from language do not apply.
• Gradient-boosted trees remained very strong on single tables, which reduced the pressure to pre-train.
• Large collections of clean tables were harder to assemble than text corpora. Generating synthetic tables partly solved this.

The core intuition

You learn a lot by solving puzzles whose answers you can check. A student who covers part of a worked solution and tries to rebuild it learns the logic behind the steps. A model that covers part of a table and rebuilds it learns the logic behind the columns. The table is both the question and the answer key.

Analogy: a chef tastes a soup with one ingredient missing from the recipe card and guesses what it was. After many soups, the chef knows which flavours go together.

Where the signal comes from

• Within a row: age and years of work move together; city and postal code move together.
• Within a column: values follow a shape, such as a skewed income curve or a few repeated categories.
• Across rows: similar rows have similar values, so a hidden cell can be borrowed from a neighbour.
• From the missing pattern: the fact that a value is absent can be informative.

A good model uses all four sources at once.

How it works

Here is the full cycle, step by step, with an everyday picture.

Step 1: Take a table

Any table will do, with any number of rows and columns. It may have missing values already, which are simply never used as targets.

Analogy: a crossword puzzle that is fully solved. You have the answer key.

Step 2: Choose a mask

Pick the cells to hide. The simplest rule hides each cell independently with a fixed probability, such as 15 to 30 percent. Richer rules hide whole columns, whole rows or blocks, so that the model learns different skills.

Analogy: you cover some squares of the solved crossword with sticky notes.

Step 3: Replace hidden cells

The hidden cells are replaced by a special mask marker. In some designs a small share of hidden cells is instead replaced by a random value or left unchanged, to stop the model from relying on the mask marker alone.

Step 4: Encode the table

Every visible cell becomes a vector, and the marker cells become a learned mask vector tagged with their column identity. The network, for example one with row and column attention, processes the grid and updates every vector.

Step 5: Predict the hidden cells

For each hidden cell, a head reads its vector and produces a prediction.

• Numeric cell: a number, or the parameters of a distribution.
• Categorical cell: a probability for each category.
• Binary cell: a probability of one.
• Date cell: components such as year, month and day, or a numeric offset.

Step 6: Compute the loss on hidden cells only

For numbers, use squared error, absolute error or a likelihood. For categories, use cross-entropy. The loss is averaged over the hidden cells. Visible cells do not count, otherwise the model could simply copy.

Step 7: Update and repeat

Backpropagation adjusts the weights. A new mask is drawn every time the table is seen, so the model sees a different puzzle each pass.

Why this teaches useful things

To guess a hidden cell, the model must use everything else in the table. It learns:

• which columns move together,
• the typical range and shape of each column,
• which rows are similar to this one,
• how missing patterns relate to values,
• and how to handle different cell types in one network.

A compact code sketch

import torch
import torch.nn as nn
import torch.nn.functional as F

def make_mask(shape, p=0.2):
    return torch.rand(shape) < p          # True means hidden

class CellMasker(nn.Module):
    def __init__(self, n_cols, dim=32):
        super().__init__()
        self.num_enc = nn.Linear(1, dim)
        self.col_id = nn.Embedding(n_cols, dim)
        self.mask_vec = nn.Parameter(torch.zeros(dim))
        self.body = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(dim, 4, 4 * dim, batch_first=True,
                                       norm_first=True), num_layers=3)
        self.head = nn.Linear(dim, 1)

    def forward(self, x, mask):
        # x: (rows, cols) numeric table, mask: same shape bool
        R, C = x.shape
        h = self.num_enc(x.unsqueeze(-1))                  # (R, C, D)
        h = torch.where(mask.unsqueeze(-1), self.mask_vec.expand_as(h), h)
        h = h + self.col_id(torch.arange(C)).unsqueeze(0)
        h = self.body(h)                                   # attend across columns
        return self.head(h).squeeze(-1)

x = torch.randn(256, 8)
x[:, 1] = x[:, 0] * 2 + 0.1 * torch.randn(256)             # a hidden relation
model = CellMasker(8)
opt = torch.optim.AdamW(model.parameters(), 1e-3)
for step in range(300):
    mask = make_mask(x.shape)
    pred = model(x, mask)
    loss = F.mse_loss(pred[mask], x[mask])                 # hidden cells only
    opt.zero_grad(); loss.backward(); opt.step()
print(float(loss))

The only line that is special to this method is the loss on pred[mask]. The network learns that column 1 is predictable from column 0 because that is the only way to reduce the loss.

Design choices in the reconstruction head

The head is small but important. Common options are listed here.

• Point estimate. The head outputs one number or one category. Simple, but gives no uncertainty.
• Mean and spread. The head outputs a mean and a standard deviation for numbers, trained with a Gaussian likelihood.
• Mixture of distributions. For skewed or multi-peaked columns, a mixture gives a better fit.
• Quantile outputs. The head outputs several quantiles, which gives intervals directly.
• Binned classification. A number is cut into bins and predicted as a category. This handles odd shapes well and is easy to train.

Handling different cell types

Scheduling the mask during training

Training does not need to keep the same mask ratio from start to finish.

1. Begin with a low ratio so the task is easy and the model finds the basic relations.
2. Raise the ratio gradually so the model must rely on weaker clues.
3. Add structured masks late in training, when the model is strong enough to use them.
4. In the final stage, sample the ratio at random per batch to cover many test conditions.

This is a form of curriculum learning.

Key concepts and terms

A worked example

Take a small table with four columns: age, income, years_of_work and owns_home.

One masked pass

We hide two cells at random: the age in row 2 and the years_of_work in row 4.

The model must output an age for row 2 and a years_of_work value for row 4.

What clues exist

• For row 2, years_of_work is 22 and income is 110. Across the other rows, years_of_work is about age minus 23 (28 gives 5, 33 gives 9, 52 gives roughly 28). So age should be around 44 to 46. The true value is 45.
• For row 4, age is 52. Using the same relation, years_of_work should be around 28 to 29. The true value is 28.

What the model learns

After many such passes on many tables, the network does not memorize these particular numbers. It learns the general skill of finding which other columns are informative for a hidden cell, in any table. This is the transferable part.

Loss for this example

If the model guessed 43 for the age and 27 for the years, the squared errors are 4 and 1, with a mean of 2.5. Over time, training pushes the mean down. Visible cells such as the income of row 1 do not enter the loss.

Different masks teach different skills

Using the last-column mask is a key link to supervised learning: predicting a label is just masked cell modeling where the masked cell happens to be the label.

What good performance looks like

On a single well-behaved table, the reconstruction error should fall well below that of a simple baseline such as the column mean. Compare against several baselines.

A masked model that does not beat the iterative regression fill on a table with strong column relations is probably undertrained or poorly masked.

Metrics for evaluation

• Numeric cells: mean absolute error and root mean squared error, after scaling each column.
• Categorical cells: accuracy and log loss.
• Distribution heads: coverage of prediction intervals, which should match the nominal level.
• Downstream: quality of a classifier trained on the imputed table.

Strengths and limitations

Strengths

• No labels required. Any table helps, so scale is easy.
• One objective for many tasks. Imputation, classification, regression and anomaly scoring all reduce to cell prediction.
• Natural fit for missing data. The model practices exactly the situation it will meet in real life.
• Handles mixed types. Each cell type gets its own head and loss.
• Learns column relations. Cross-column dependencies are the only way to reduce loss.
• Compatible with many architectures. Works with transformers, autoencoders and latent models.
• Supports uncertainty. If the head outputs a distribution, the model gives intervals along with guesses.

Limitations

• Mask design matters. A poor mask ratio or pattern teaches shortcuts.
• Identifiability. Some cells simply cannot be predicted from the others. The model then learns the average, which is correct but not impressive.
• Mismatch with real missingness. Real data is often missing not at random, while training masks are random.
• Mixed-scale losses. Summing errors of different scales needs care.
• Not a full replacement for supervision. For a specific label, direct fine-tuning or in-context labeled rows still help.
• Can learn trivial copies. If two columns are duplicates, the model learns to copy, which teaches little.
• Compute cost. Pre-training on many tables takes real resources.

Practical effect on downstream accuracy

Pre-training with masked cells tends to help most when labeled data is small, because the model arrives with a good understanding of table structure. It helps less when labeled data is plentiful and the table is simple, since a boosted tree can already extract most of the signal. For tasks with many missing values, the benefit is larger because the model has practiced exactly that situation.

How it compares with alternatives

Masked cell modeling and contrastive learning together

The two objectives complement each other. Masking teaches the model to predict values. Contrast teaches it to tell rows apart. Many strong tabular models use a mix, with the masked loss as the main signal and the contrastive loss as a regularizer.

Masked cell modeling and in-context learning

In-context learning teaches a model to predict a label from labeled rows in the same input. Masked cell modeling teaches it to predict any cell from the other cells. They are close relatives: one can view in-context prediction as masking the label cells of the query rows. A model trained on both has more skills.

Role in tabular foundation models

A tabular foundation model reads raw tables with different shapes, mixed types and missing values, often with no meaningful column names. Masked cell modeling is one of the best objectives for that setting, because it needs only the table.

Raw rows and columns

The objective is defined on the grid. It does not care about the table's meaning or the number of columns. This makes it simple to apply across millions of tables.

Mixed types

Each column type has its own encoder and its own prediction head. A numeric cell uses a regression-style loss. A categorical cell uses a classification loss. During pre-training, the model sees all of them, and the shared body learns to handle them in one space.

Missing values

Real tables have gaps. These gaps are ignored when computing loss but are marked in the input, so the model learns to treat "missing" as information. The artificial masks provide more practice. This directly builds an imputation network.

No column names

Because masked cell modeling works from values, it forces the model to understand columns by their content and relations rather than by a header. If headers exist, they are an extra clue. If they do not, the objective still works. This is important for the many tables with codes such as "col_17".

Pre-training on synthetic and real tables

Synthetic tables can be generated from random structural models, and the true values of every cell are known. Real tables bring genuine messy patterns. Masked cell modeling can run on both with no change. See synthetic data generation and the research notes at /model/research.

In-context learning

When the last column is the label, masking it for the query rows turns prediction into masked cell modeling. A model trained to do this with labeled rows visible learns to read the labeled rows and apply the pattern. See TabPFN, TabICL and TabSTAR for related designs.

Continual learning

When new tables arrive over time, the model can keep practicing masked cell modeling on them. To avoid forgetting, mix in a replay of earlier tables or use regularization. See experience replay, elastic weight consolidation and synaptic intelligence. Because the objective is the same across tables, new data fits into the existing training loop.

Imputation

Masked cell modeling is imputation practice. At test time, the real missing cells play the role of masked cells and the head produces estimates. Models with distribution heads can also output uncertainty, which is valuable in decision settings. See MIDAS for a related approach built on denoising autoencoders.

Anomaly detection

A well-trained model assigns low error to normal cells. A cell with a large reconstruction error, or a row with a large average error, is unusual given its context. Using the reconstruction error as an anomaly score is simple and transfers across tables. See anomaly detection, isolation forest and one-class SVM for classic baselines.

Sector and semantic classification

Deciding what kind of table or column something is relies on a representation that captures column meaning. Masked pre-training builds such a representation. A light head on top can then classify a column's semantic type or a table's sector. See semantic and sector heads. Evaluation ideas for these tasks appear at /model/benchmarks.

Putting masked cell modeling into a pipeline

1. Collect tables. Gather real tables and generate synthetic ones.
2. Clean lightly. Remove obvious duplicates and columns that are constants.
3. Type the columns. Detect numeric, categorical, date and text columns.
4. Scale and encode. Apply robust scaling and encodings with saved parameters.
5. Mask and train. Use mixed mask types and a per-type loss.
6. Validate on held-out tables. Track error per type and per column.
7. Probe. Train light heads for classification, anomaly scoring and column typing to see what the model has learned.
8. Fine-tune or use in context. Adapt to a task only if needed.

Monitoring a run

• Track the loss for each type separately.
• Track the reconstruction error as a function of the mask ratio.
• Check for columns whose error never improves, which may be pure noise.
• Check for columns whose error is almost zero, which may be duplicates.
• Plot predicted versus true values on a held-out table.

Practical guidance and best practices

• Start with a 15 to 30 percent mask ratio. Vary it during training for more robustness.
• Mix mask types. Include random cells, whole columns and block patterns.
• Resample masks every pass. Fixed masks lead to memorization.
• Use type-appropriate losses. Squared or absolute error for numbers, cross-entropy for categories.
• Normalize numeric columns. Robust scaling avoids large-valued columns dominating the loss.
• Weight columns fairly. Average the loss per column first, then across columns.
• Mark original missing cells. Provide a missing flag in the input and exclude them from the loss.
• Prefer distribution heads. A predicted mean and spread allows uncertainty estimates.
• Remove target leaks. If a column is a copy or deterministic function of another, consider dropping one or masking both.
• Validate by table. Hold out whole tables to measure transfer.
• Evaluate with realistic missingness. Test with patterns that look like real gaps.
• Compare against simple imputers. Mean, median and iterative imputers set a floor.

Choosing a mask ratio

Loss weighting across types

1. Normalize each numeric loss by the column variance.
2. Average cross-entropy over categorical cells.
3. Combine numeric and categorical terms with a weight chosen on validation data.
4. Watch each term separately, since one can hide the failure of another.

How to tell a shortcut from real learning

A model can lower its loss by a shortcut, such as reading a duplicate column. To catch this:

• Drop one of two highly correlated columns and see if the error jumps.
• Shuffle one column across rows and see if the error on other columns rises. If not, that column was not used.
• Check performance on tables the model has never seen.
• Compare with a model trained on shuffled labels or noise columns to see the floor.

Common mistakes

• Computing loss on all cells. The model then learns to copy visible cells and learns nothing.
• Using a fixed mask. The network memorizes the answers for those cells.
• Zero as a mask value. Zero is a legitimate value in many columns. Use a separate marker.
• Ignoring original missingness. Treating real gaps as zeros corrupts the signal.
• One loss for all types. Mixing categories and numbers under squared error gives poor results.
• Unscaled columns. A column measured in millions overwhelms the loss.
• Random masks only. Real gaps are often structured, so test on structured masks too.
• Evaluating on training tables. This hides overfitting. Evaluate on unseen tables.
• Leaking through duplicates. Duplicate columns or rows let the model cheat.
• Expecting accuracy on unpredictable cells. Random noise columns cannot be predicted. The right answer is the average, with a wide interval.

Frequently asked questions

What is masked cell modeling in simple terms?

Masked cell modeling is a training game for tables. You cover some cells, ask the model to guess the covered values from the rest of the table, and reward it when the guess is close. By playing this game on many tables, the model learns how columns relate to each other and what typical values look like, with no human labels at all.

How is it different from masked language modeling?

Masked language modeling hides words in an ordered sentence and the model uses word order and grammar. Masked cell modeling hides cells in a grid with no natural order and mixed types, so the model must use column identity, row context and column statistics. The training idea is the same, but the data structure and the losses are different.

What mask ratio should I use?

A ratio of fifteen to thirty percent is a common starting point. Lower ratios make the task easy and slow to teach, while very high ratios make it hard but can give stronger representations. Many practitioners vary the ratio across batches so the model works well whether a few or many cells are missing at test time.

Can masked cell modeling handle categorical and numeric cells together?

Yes. Each column type gets its own input encoder and its own output head. Numeric cells are trained with a regression loss or a distribution likelihood, and categorical cells with cross-entropy. The shared body of the network processes all cells in the same vector space, so one model handles both kinds.

How does it relate to imputation?

Imputation is filling real missing values, and masked cell modeling is practice for exactly that. During training, artificial gaps are created and the model learns to fill them. At test time the real gaps are handled the same way. This also lets the model give a confidence along with each fill when its head predicts a distribution.

Can it be used for classification and regression?

Yes. A label is just another column. If the label column is masked for query rows while labeled rows stay visible, predicting it is the same task as masked cell modeling. A model pre-trained this way can be fine-tuned or used in context for classification and regression on a new table.

Does it need column names?

No. The objective works from values and their relations. Column names can help as extra input when they exist, but the model learns to identify columns by their content and their dependencies on other columns, which matters for tables with cryptic or missing headers.

Does it work if real missingness is not random?

Training with random masks may not match real gaps, which are often tied to values or to the process that generated the data. A good practice is to also train and test with structured masks, such as whole blocks or value-dependent gaps, and to give the model a missing flag so it can learn that missingness itself carries information.

How is it used for anomaly detection?

A model trained to reconstruct cells assigns small errors to ordinary cells and larger errors to cells that do not fit their context. The reconstruction error of a cell, or the average error of a row, can be used as an anomaly score. It is often combined with classic detectors for extra robustness.

Is masked cell modeling enough to build a tabular foundation model?

It is a strong core objective but usually not the only one. Foundation models often add in-context label prediction, contrastive objectives and column type tasks. Masked cell modeling supplies the general understanding of table structure, and the other signals specialize it for particular uses.

Glossary

• Anomaly score: a number that says how unusual a cell or row is.
• Cell: the value at the crossing of one row and one column.
• Column mask: hiding every cell in a chosen column.
• Contrastive learning: training that pulls similar items together and pushes different items apart.
• Cross-entropy: a loss for predicting categories.
• Denoising: repairing a corrupted input.
• Fine-tuning: continuing training on a specific task.
• Imputation: filling missing values.
• In-context learning: adapting to a task from examples placed in the input.
• Likelihood: the probability a model assigns to the observed value.
• Mask ratio: the fraction of cells hidden in a pass.
• Mask token: a learned vector that stands for a hidden cell.
• MCAR, MAR, MNAR: three types of missingness, described by what the gap depends on.
• Pre-training: training on broad data before specializing.
• Reconstruction: predicting a hidden value from context.
• Self-supervised learning: learning from signals that the data supplies itself.
• Tabular foundation model: a model pre-trained on many tables that serves new tables with little or no training.

Summary

Masked cell modeling hides some cells of a table and trains a model to predict them from the visible cells. It needs no labels, works for any table shape, handles mixed types through type-specific heads and losses, and treats real missing cells as information.

It is a central pre-training recipe for tabular foundation models. It teaches column relations and value distributions, and it gives direct support for imputation networks, anomaly scoring, in-context prediction and semantic classification. Best results come from good mask design, per-type losses, scaled inputs and evaluation on unseen tables. It pairs naturally with architectures like axial attention and Perceiver IO. For more, see /model/architecture, /model/research and /model/benchmarks.

Related

It belongs to Tabular foundation models in my stack, under Foundation models and applied ML. Others in the same group: