MAE

Overview

MAE, short for masked autoencoder, is a self-supervised learning method. It hides a large share of the input, shows the rest to an encoder, and trains a small decoder to rebuild what was hidden. The model needs no labels. The only teacher is the data itself. Because the task is hard, the encoder has to learn real structure in the data to solve it.

The best known version is the masked autoencoder for images. It hides about three quarters of the patches of a picture and still learns strong features. The same idea applies to text, audio, video and tables. For tables, it is the intuition behind masked cell modeling: hide some cells, predict them from the others. This article explains the method step by step, shows a small worked example, and then connects it to models that learn from raw tables.

What it is

An autoencoder is a network that compresses an input and then rebuilds it. A masked autoencoder adds one twist. Before compression, a large part of the input is removed. The network must fill in the gap.

Think of a jigsaw puzzle with most pieces missing. If you can still describe the whole picture from a few pieces, you must know what pictures usually look like. A masked autoencoder is trained to do exactly that, millions of times.

The two halves

• Encoder. It sees only the visible pieces. It turns them into a set of feature vectors.
• Decoder. It receives the encoded visible pieces plus placeholders for the missing ones. It outputs a guess for every missing piece.

After pre-training, the decoder is usually thrown away. The encoder is kept and reused for other tasks such as classification or detection.

Why mask so much

In text, a single missing word is often easy to guess from its neighbours. Language is dense with meaning. Images are different. A missing pixel is almost always predictable from the pixels around it, because nearby pixels are very similar. If only a few patches are hidden, the model can cheat by smoothing over the gap. When most patches are hidden, smoothing is not enough. The model has to understand objects and layout.

What the output looks like

The decoder predicts raw values for each hidden patch, such as pixel intensities. The loss is the mean squared error between the prediction and the truth, measured only on hidden patches. Visible patches are not scored, since the model already saw them.

Where it came from

The idea has a long family tree.

• Autoencoders date back to the 1980s. They were used to compress data and to pre-train networks layer by layer.
• Denoising autoencoders came in the late 2000s. They corrupt the input with noise and train the network to recover the clean version. See denoising autoencoders.
• Context prediction and inpainting appeared in the mid 2010s. Networks were trained to fill in missing regions of images.
• BERT (2018) showed that masking words and predicting them gives excellent language representations. This made masked prediction the dominant pre-training recipe for text.
• Vision transformers (2020) made it natural to treat an image as a sequence of patches, much like words in a sentence. See ViT.
• Masked Autoencoders Are Scalable Vision Learners (2021) put the pieces together. It used a vision transformer, a very high mask ratio, and an asymmetric design where the encoder only sees visible patches. This made training fast and results strong.

Later work extended the same idea to video, audio, point clouds and many other kinds of data. Related methods predict features instead of pixels, or combine masking with contrastive learning.

How it works

Here is the full pipeline, step by step.

Step 1: Split the input into patches

An image of 224 by 224 pixels is cut into a grid of small squares, for example 16 by 16 pixels each. That gives 196 patches. Each patch is flattened into a vector and projected to a fixed width. Position information is added so the model knows where each patch came from.

Everyday analogy: cut a photo into a grid of tiles and number each tile by its place on the wall.

Step 2: Randomly mask most patches

A random subset of patches, often 75 percent, is removed. Only the remaining 25 percent go forward. Random choice matters. If you always hid the same region, the model could learn a shortcut.

Step 3: Encode only the visible patches

A transformer encoder processes the visible patches. Because it ignores the masked ones, the sequence is four times shorter. Attention cost grows with the square of sequence length, so this saves a lot of compute and memory. See transformers for how attention works.

Step 4: Add mask tokens and decode

The encoded vectors are put back in their original positions. At every missing position, a shared learned placeholder called a mask token is inserted. Position information is added again. A small transformer decoder reads this full sequence and predicts the content of each patch.

The decoder can be light because it is only used during pre-training. This asymmetric design is one of the main reasons the method is cheap.

Step 5: Compute the loss on hidden patches

The model compares its predictions with the real values of the hidden patches. The mean squared error is averaged over hidden patches only. Some versions normalise the target values inside each patch so the model focuses on structure rather than overall brightness.

Step 6: Update and repeat

Gradients flow through decoder and encoder. The optimiser updates the weights. A new random mask is drawn for the next sample, so the same picture poses a different puzzle each time.

Step 7: Reuse the encoder

After pre-training, the decoder is removed. The encoder is fine-tuned on a labelled task, or used as a frozen feature extractor. See transfer learning for the general pattern.

A tiny pseudocode view

def mae_step(batch, encoder, decoder, mask_ratio=0.75):
    patches = patchify(batch)                  # (B, N, D)
    keep, hidden, order = random_mask(patches, mask_ratio)
    visible = gather(patches, keep)
    latent = encoder(visible + pos_embed[keep])
    full = insert_mask_tokens(latent, order, mask_token)
    pred = decoder(full + dec_pos_embed)
    loss = ((pred[hidden] - patches[hidden]) ** 2).mean()
    return loss

Variants and extensions

Several families of change have been tried on the basic recipe.

• Different targets. Instead of raw pixels, the decoder can predict features from another network, discrete codes, or histograms of local gradients. These targets carry more meaning and can improve the encoder.
• Different masking. Guided masks hide the most informative patches, which makes the puzzle harder and the lesson richer.
• Different data. Video versions hide whole space-time tubes. Audio versions hide spans of a spectrogram. Point cloud versions hide groups of points.
• Hybrid objectives. Some work adds a contrastive term on top of reconstruction, so the features are both descriptive and discriminative.
• Hierarchical encoders. Windowed models such as Swin need extra care, because masking breaks the regular grid they expect.

Each variant answers one question: what should the model be forced to predict so that the features become useful.

Why the design is efficient

Count the cost of a naive version. If every token, visible or not, went through the encoder, the cost would be the same as for ordinary training. Dropping 75 percent of tokens cuts attention work to roughly one sixteenth and the linear layers to one quarter. The decoder adds cost, but it is small. The net result is a pre-training run that is several times faster than the same encoder trained on full inputs. This is the reason the approach scaled so quickly.

An everyday picture: a proofreader who is only handed the quarter of the page that survived a coffee spill will finish quickly, and a second, cheaper assistant then guesses the stained words. The expert does less work, and the cheap assistant takes the rest.

Key concepts and terms

A worked example

Let us walk through a small numeric case, then a code case.

Numeric walk-through

Suppose an input has 8 tokens. We use a mask ratio of 75 percent.

1. Tokens are numbered 0 to 7.
2. We hide 6 of them at random, for example 0, 1, 3, 4, 6 and 7.
3. Only tokens 2 and 5 are visible. The encoder processes a sequence of length 2.
4. The decoder receives a sequence of length 8: real encodings at positions 2 and 5, mask tokens at all other positions.
5. The decoder outputs 8 predictions. The loss uses only the 6 hidden positions.

If the true value of token 0 is 0.8 and the prediction is 0.5, its squared error is 0.09. The same is done for the other five hidden tokens and the results are averaged. That average is the training loss.

Notice what the encoder cost. Attention over 2 tokens is far cheaper than over 8. At image scale, 49 visible tokens instead of 196 gives roughly a sixteenth of the attention cost.

A small PyTorch sketch

This example is a toy masked autoencoder on random vectors. It shows the mechanics, not a production recipe.

import torch
import torch.nn as nn

class TinyMAE(nn.Module):
    def __init__(self, n_tokens=16, dim=32, mask_ratio=0.75):
        super().__init__()
        self.n, self.ratio = n_tokens, mask_ratio
        self.embed = nn.Linear(1, dim)
        self.pos = nn.Parameter(torch.randn(1, n_tokens, dim) * 0.02)
        enc_layer = nn.TransformerEncoderLayer(dim, 4, 64, batch_first=True)
        self.encoder = nn.TransformerEncoder(enc_layer, 2)
        dec_layer = nn.TransformerEncoderLayer(dim, 4, 64, batch_first=True)
        self.decoder = nn.TransformerEncoder(dec_layer, 1)
        self.mask_token = nn.Parameter(torch.zeros(1, 1, dim))
        self.head = nn.Linear(dim, 1)

    def forward(self, x):                      # x: (B, N)
        B, N = x.shape
        tok = self.embed(x.unsqueeze(-1)) + self.pos
        n_keep = int(N * (1 - self.ratio))
        noise = torch.rand(B, N)
        order = noise.argsort(dim=1)
        keep = order[:, :n_keep]
        idx = keep.unsqueeze(-1).expand(-1, -1, tok.size(-1))
        latent = self.encoder(torch.gather(tok, 1, idx))
        full = self.mask_token.expand(B, N, -1).clone()
        full.scatter_(1, idx, latent)
        pred = self.head(self.decoder(full + self.pos)).squeeze(-1)
        mask = torch.ones(B, N)
        mask.scatter_(1, keep, 0.0)
        loss = (((pred - x) ** 2) * mask).sum() / mask.sum()
        return loss

model = TinyMAE()
x = torch.sin(torch.linspace(0, 6.28, 16)).repeat(8, 1)
print(model(x).item())

On smooth signals like a sine wave, the loss drops quickly because hidden points are easy to infer from visible ones. On noisy data, the loss stays higher. This mirrors why masking works well where the data has structure.

Reading the numbers

Imagine a toy run on a signal with 16 points. At the start, the average error on hidden points is about 1.0, because the model guesses near zero. After some training it falls to 0.1 on smooth data. If you then double the noise in the data, the floor rises, since noise cannot be predicted. This shows two lessons. A lower loss does not always mean a better model, since part of the error is irreducible. And comparing losses across datasets is meaningless unless the data have the same noise level.

Checking the learned features

A quick test is the linear probe. Freeze the encoder, run the labelled data through it, and train one linear layer. If accuracy is far above a randomly initialised encoder, the pre-training taught something. A second test is nearest-neighbour retrieval: for a query sample, find the closest samples in feature space and check whether they look related.

import torch
from sklearn.linear_model import LogisticRegression

@torch.no_grad()
def extract(encoder, loader):
    feats, ys = [], []
    for x, y in loader:
        feats.append(encoder(x).mean(dim=1))   # average over tokens
        ys.append(y)
    return torch.cat(feats).numpy(), torch.cat(ys).numpy()

Xtr, ytr = extract(encoder, train_loader)
Xte, yte = extract(encoder, test_loader)
probe = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
print("probe accuracy:", probe.score(Xte, yte))

A table-shaped version of the example

Here is how the same mechanics look on a small table. Each row is a customer. Columns are age, income, city code and a flag. A masked cell model would:

1. Embed each cell with a column-specific embedding plus a value embedding.
2. Hide a random 30 percent of cells in each row, replacing them with a mask token.
3. Run attention across the cells of the row.
4. Predict the hidden numbers with a regression head and the hidden categories with a classification head.
5. Add the losses, weighting each column so no single scale dominates.

import torch
import torch.nn as nn

class CellMasker:
    def __init__(self, ratio=0.3):
        self.ratio = ratio

    def __call__(self, values, is_missing):
        # values: (B, C) floats, is_missing: (B, C) bool for true gaps
        hide = (torch.rand_like(values) < self.ratio) & ~is_missing
        return hide

def table_loss(pred_num, true_num, hide, num_cols):
    # squared error on hidden numeric cells only
    m = hide[:, num_cols].float()
    err = (pred_num - true_num[:, num_cols]) ** 2
    return (err * m).sum() / m.sum().clamp(min=1.0)

Note the detail that real gaps are excluded from the mask draw. A cell that is truly missing has no target, so it cannot be scored. It can still be fed in as a mask token so the model learns to cope.

Strengths and limitations

Strengths

• No labels needed. Large unlabelled collections become useful.
• Efficient. The encoder skips masked tokens, so training is fast and uses less memory.
• Strong features. Fine-tuned encoders reach high accuracy on downstream tasks.
• Scales well. Bigger models and longer training keep helping.
• Simple. The loss is a plain regression. There are no negative pairs and no heavy augmentation.
• General. The recipe carries over to many data types.

Limitations

• Frozen features are weaker than fine-tuned ones. A linear probe on MAE features often lags methods built on contrastive objectives. Fine-tuning closes the gap.
• Pixel targets focus on low-level detail. The model may spend capacity on texture rather than meaning.
• Mask ratio and mask style are sensitive choices. The best ratio depends on how redundant the data is.
• Needs long training. Hard puzzles take many epochs.
• Reconstruction is not the end goal. A model can rebuild well and still give mediocre features.
• Does not give uncertainty by default. The decoder outputs a point guess.

How it compares with alternatives

MAE and contrastive methods are complementary. Reconstruction teaches the model to model the data. Contrastive objectives teach it to tell samples apart. Some recent systems combine both.

Role in tabular foundation models

This is where the masked autoencoder idea matters most for table learning. A table is a grid of cells. Each cell sits in a row and a column. Masking cells and predicting them is the direct analogue of masking patches and predicting them.

Masked cell modeling

Masked cell modeling hides a random share of cells in a row, or whole columns, and asks the model to restore them from the cells that remain. It is the table version of MAE. The model has to learn how columns relate: age with income, city with postal code, temperature with pressure.

Two differences from images matter.

• Tables are less redundant. A neighbouring pixel is nearly the same as the missing one. A neighbouring column is often unrelated. So lower mask ratios, such as 15 to 40 percent, usually work better than 75 percent.
• Cells are mixed in type. Some are numbers, some are categories, some are dates or free text. The reconstruction loss has to fit each type: squared error or a quantile loss for numbers, cross entropy for categories.

Row and column structure

Images have a fixed grid. Tables have rows that are exchangeable and columns that may come in any order. A table model should not care about the order of rows, and ideally not about the order of columns either. Masked pre-training helps here. The model sees many random subsets of columns and learns to use whatever is visible. Attention across the columns of one row, and across rows for one column, is a common design. See axial attention and FT-Transformer.

Missing values and imputation

Real tables are full of gaps. A model pre-trained with masking is trained on missing values by construction. Hiding a cell during training is the same event as meeting a missing cell at run time. This makes the model a natural imputer. Filling in a gap is just running the decoder at that position. See imputation networks and MIDAS.

No column names

Many real tables have cryptic or absent headers: c1, c2, x_17. A masked objective does not need names. The model learns what a column is from the values and from how it co-varies with other columns. A foundation model built this way can read tables where the names carry no meaning. When names exist, they can be added as extra tokens as a bonus.

Pre-training on synthetic or real tables

Masked pre-training works on any table, with no labels. That allows two sources.

• Real tables. Large collections of public or licensed tables give realistic structure.
• Synthetic tables. Generators sample random relationships between columns, with noise, missing values and mixed types. Prior-fitted models such as TabPFN use this idea, although they train on a supervised in-context objective rather than reconstruction.

A mix of both is common. Synthetic data covers rare structures. Real data adds realistic marginals.

In-context learning

Masked prediction can be used at inference time as well. Put labelled rows in the context, put an unlabelled row at the end with its target cell masked, and read the prediction at that cell. The target column is treated as one more column to fill in. This turns classification and regression into a special case of cell completion. See TabICL and XTab.

Continual learning

When new data arrives, a model that reconstructs cells can keep learning with the same loss, with no label work. To avoid forgetting, replay and regularisation help. See experience replay and EWC.

Anomaly detection

A model that rebuilds cells well on normal rows will rebuild abnormal rows badly. The reconstruction error per cell, or per row, becomes an anomaly score. Large error on one cell can also point to the likely faulty field. See anomaly detection.

Sector and semantic classification

The features learned by masked pre-training can feed a light classification head. For example, a head can decide whether a column holds dates, amounts, identifiers or codes, or which industry a table belongs to. The encoder has already learned the statistics that separate these types. See semantic sector heads.

For wider context on how these pieces fit together, see /model/architecture, /model/research and /model/benchmarks.

Choosing between reconstruction and prediction of features

Pixel or value targets are easy to set up and work well on dense data. Feature targets, produced by another network, are harder to set up but often give more semantic encoders. For tables, value targets are natural, because the values are the meaning. Feature targets become useful when a column is free text or a high-cardinality code, where raw reconstruction is awkward.

Relationship to generative models

A masked model is not a full generative model by default, since it predicts hidden parts given visible ones, not the whole joint distribution. Yet by masking in a sequence of rounds, it can sample whole rows: fill a few cells, add them to the visible set, and continue. This gives a way to create synthetic tables for testing, privacy-aware sharing or class balancing. Compare with diffusion models and GANs, which model the whole distribution directly.

Practical guidance and best practices

Choose the mask ratio for your data

• Images and other dense signals: start near 75 percent.
• Text: 15 to 30 percent is common.
• Tables: start near 15 to 40 percent and tune. Try different ratios per column group.
• Time series: mask contiguous blocks as well as scattered points.

Match the loss to the data type

Use squared error or a robust variant for numbers, cross entropy for categories, and consider normalising numeric targets by column. Without this, large-scale columns dominate the loss.

Keep the decoder small

The decoder only helps during training. A shallow decoder keeps compute low and pushes knowledge into the encoder, which is what you keep.

Vary the masking pattern

Mix random, block and column-wise masks. This prepares the model for the different kinds of missingness it will meet later.

Evaluate with downstream tasks

Reconstruction loss is a training signal, not a goal. Track a linear probe and a fine-tuned score on a real task.

Normalise inputs

Scale numeric features, handle outliers, and encode categories consistently. Masked training is sensitive to scale because the loss is a distance.

Use mixed precision and checkpointing

Large encoders benefit from mixed precision and gradient checkpointing. Both reduce memory without changing the method.

Start from a framework you know

PyTorch makes a masked autoencoder short to write, as in the example above.

Setting up a training run

A sensible recipe has a few moving parts.

Log the reconstruction loss, but also log a downstream probe every few epochs. If the probe stops improving while the loss keeps falling, the model may be fitting detail that does not help the real task.

When to choose this method

• You have much unlabelled data and few labels.
• You plan to fine-tune, not only to extract frozen features.
• Your data has structure that can be inferred from context.
• You want one pre-trained encoder for many downstream tasks.

When labels are plentiful and the task is narrow, plain supervised training may be simpler and just as good.

Debugging a masked training run

When results look poor, check these in order.

1. Is the loss only on hidden positions? Print the count of scored positions. It should match the mask ratio.
2. Does the loss beat a trivial baseline? Predict the column mean for every hidden cell. If the model cannot beat that, something is broken.
3. Is the mask really random and fresh? Fix a seed, draw two masks, and confirm they differ.
4. Do visible values leak into the target? Make sure the input at hidden positions is replaced by the mask token and not by the true value.
5. Are scales sensible? Print per-column loss. One column with a huge scale will dominate.
6. Is the decoder too strong? If it can rebuild well from almost nothing, the encoder is not being pushed.
7. Does a probe beat a random encoder? If not, the features are not useful yet.

Scaling up step by step

Start small. Train a tiny model on a small slice of data until the loss clearly beats the baseline. Then widen the model, then lengthen training, then enlarge the data. Change one thing at a time. This makes it easy to see which change helped.

Data hygiene for table pre-training

• Remove exact duplicate rows, or they will be memorised.
• Keep a held-out set of whole tables, not only held-out rows, to measure generalisation to new schemas.
• Record how each missing value was produced. Some are truly absent, some are encoded as sentinel numbers such as -999. Treat sentinels as missing, not as real values.
• Check for columns that are exact copies or simple transforms of each other. They make the masked task trivial.

Common mistakes

• Masking too little. If hidden parts are easy to guess, the task teaches nothing new.
• Masking too much on tables. Hiding most columns of a weakly related table leaves nothing to predict from.
• Scoring visible patches. Including them in the loss rewards copying and weakens learning.
• Leaking the mask. If the model can tell which positions are hidden from side information, it can shortcut the task.
• Ignoring data types. Using one loss for numbers and categories produces poor results on mixed tables.
• Judging by reconstruction alone. A pretty reconstruction does not prove good features.
• Keeping a huge decoder. It wastes compute and may hold knowledge that is later discarded.
• Fitting the scaler on test data. This hides leakage in downstream evaluation.
• Forgetting that the mask changes each step. A fixed mask turns the task into memorisation.

Frequently asked questions

What does MAE stand for?

In this context MAE stands for masked autoencoder. It is a self-supervised method that hides part of the input and trains a network to rebuild it. The same letters also appear as the abbreviation for mean absolute error, a regression metric, so check the context before assuming which one is meant.

Why does MAE hide so much of the input?

Natural data is highly redundant, so hiding a little is easy to undo by simple interpolation. Hiding a large share forces the model to learn the structure of objects and scenes rather than local smoothness. The high ratio also makes training cheaper, because the encoder only processes the small visible part.

Does MAE need labels?

No. The training signal comes from the input itself. The hidden parts are the targets. Labels are only needed later, if you fine-tune the encoder for a supervised task such as classification.

Is the decoder used after pre-training?

Usually not. The decoder exists to give the encoder a learning task. Once pre-training ends, it is discarded and only the encoder is kept for downstream use. For imputation tasks you may keep it, since it is the part that fills gaps.

How is MAE different from BERT?

Both predict hidden parts from visible parts. BERT works on word tokens with a modest mask share and feeds mask tokens into the encoder. MAE removes masked tokens before the encoder and uses a separate light decoder to rebuild them. This makes MAE cheaper for data where the mask ratio is high.

Can MAE work on tabular data?

Yes, in a modified form. Cells replace patches, and the loss is adapted to numeric and categorical columns. This is called masked cell modeling. Lower mask ratios work better because columns are less redundant than neighbouring pixels.

How does it help with missing values?

Training with hidden cells exposes the model to the same situation it faces when real values are missing. At run time the missing cell plays the role of a masked cell, and the decoder produces an estimate. This is why masked models make natural imputers.

What is a good mask ratio?

It depends on how redundant the data is. About 75 percent suits images, 15 to 30 percent suits text, and 15 to 40 percent is a reasonable start for tables. Treat the number as a setting to tune on a downstream task, not a fixed rule.

Is MAE better than contrastive learning?

Neither is better in every case. MAE is fast and gives strong results after fine-tuning. Contrastive and self-distillation methods often give better frozen features. The choice depends on whether you will fine-tune, how much compute you have, and what kind of data you hold.

Does MAE work with convolutional networks?

It was designed around transformers, because they can take a sparse set of visible tokens naturally. Convolutions assume a dense grid, so masked inputs need special handling such as sparse convolutions. Later work has made masked pre-training practical for ConvNeXt and other convolutional backbones.

Glossary

• Autoencoder. A network that compresses an input and rebuilds it.
• Attention. A mechanism that lets each token weigh all other tokens.
• Backbone. The main feature-extracting part of a model.
• Block masking. Hiding a contiguous region.
• Decoder. The part that turns encoded features back into the input space.
• Encoder. The part that turns inputs into feature vectors.
• Fine-tuning. Continuing training on a labelled task.
• Imputation. Filling in missing values.
• Linear probe. A linear classifier trained on frozen features.
• Mask token. A learned vector standing in for a missing element.
• Masked cell modeling. Masked prediction applied to cells of a table.
• Mask ratio. The share of the input that is hidden.
• Patch. A small tile of an image treated as a token.
• Pre-training. Learning general features before a specific task.
• Reconstruction loss. The error between rebuilt and true values.
• Self-supervised learning. Learning from signals built from the data.
• Token. One unit of input to a transformer.

Summary

A masked autoencoder hides most of its input, encodes what remains, and trains a light decoder to rebuild the rest. The task is simple to state and hard to solve, so the encoder learns rich structure without labels. Skipping masked tokens in the encoder makes training cheap.

For tables, the same idea becomes masked cell modeling. It teaches a model how columns relate, how to cope with missing values, and how to work without column names. It supports imputation, anomaly scoring and in-context prediction. Choose the mask ratio for the data, match the loss to each column type, judge the result on downstream tasks, and the method becomes a solid base for models that learn from raw tables. Related pages include self-supervised learning, autoencoders and transformers.

Related

It belongs to Self-supervised methods in my stack, under AI and machine learning. Others in the same group: