Self-supervised learning

Overview

Self-supervised learning is a way of training models on unlabelled data by hiding or changing part of the input and asking the model to recover or relate it. The data supplies its own answers. A sentence with a missing word has the word as its label. A photo with a masked patch has the patch as its label. No human has to annotate anything.

This matters because unlabelled data is plentiful and labelled data is costly. Self-supervised pre-training lets a model learn general structure from huge collections, then adapt to a specific task with few labels. It is the main recipe behind large language models and many vision models. For tables, the same idea means masking cells, corrupting values or comparing views of a row, so that a model learns how columns relate before it ever sees a target. See transfer learning for how the learned skills are reused.

What it is

Self-supervised learning sits between two older settings.

• Supervised learning uses inputs with human labels.
• Unsupervised learning looks for structure with no targets, such as clusters.

Self-supervised learning looks unsupervised from the outside, because no labels are provided. Inside, it behaves like supervised learning, because a target is built automatically from the data. The task that produces this target is called a pretext task.

A daily-life analogy

Imagine learning a language by reading books with some words blacked out. You guess each missing word from the context. Nobody grades you with a teacher's labels, yet the book itself tells you whether you were right. After thousands of pages you understand grammar, topics and style. That is self-supervision.

The two stages

1. Pre-training: learn general representations on a large unlabelled set with a pretext task.
2. Downstream use: reuse the learned model for a real task by adding a small head, fine-tuning, or using it as a feature extractor.

Why the learned features are useful

To solve the pretext task well, the model has to capture structure that is also useful elsewhere. To fill in a missing word, it must know grammar and meaning. To restore a masked patch, it must know shapes and textures. To recover a masked cell, it must know how columns depend on each other.

Families of pretext tasks

• Masked prediction: hide parts, recover them.
• Next-element prediction: predict what follows.
• Denoising: corrupt the input, restore the clean version.
• Contrastive: learn that two views of one item are similar and different items are not. See contrastive learning.
• Self-distillation: a student matches a slowly updated teacher, as in BYOL and DINO.
• Transformation prediction: guess which rotation or shuffle was applied.

Where it came from

The idea of learning without labels is old. Early neural work used autoencoders that compress and rebuild their input. Word embedding methods in the early 2010s learned word meaning by predicting a word from its neighbours, a clear self-supervised signal.

Language

In 2018 and 2019, large language models showed that predicting masked or next words on massive text produced features that transferred to nearly every language task. This turned self-supervision into the dominant pre-training recipe. Scaling the same objective, with more data and parameters, produced the generative models used today.

Vision

Vision followed a different path at first, with pretext tasks such as predicting image rotation or solving jigsaw puzzles. Later, contrastive methods such as SimCLR and MoCo reached strong results. Self-distillation methods such as BYOL and DINO removed the need for negative examples. Masked image modeling, as in MAE, brought the language-style masking idea to patches.

Tables

Tabular self-supervision arrived later, because tables lack the natural neighbourhood structure of text and images. Work around 2020 and 2021 introduced masked feature prediction, corruption-based objectives and contrastive row views for tabular networks. Later, prior-fitted models took a related route, learning from many synthetic tables. See masked cell modeling and denoising autoencoders.

How it works

This section follows a typical self-supervised run from data to downstream use.

Step 1: Collect a large unlabelled set

Gather as many examples as is practical: web text, images, sensor logs, or many tables. Quantity and diversity matter more than annotation quality, because nothing is annotated.

Step 2: Choose a pretext task

The choice defines what the model must learn. A good pretext task:

• Is solvable only by understanding meaningful structure.
• Is not solvable by a trivial shortcut.
• Is cheap to generate from raw data.

Step 3: Build the input and the target from the same example

For masked prediction on a row with five fields, the pipeline picks some fields at random, replaces them with a mask marker, and keeps the original values as targets.

Analogy: a teacher cuts a few words out of a paragraph and keeps the cut-out pieces as the answer key.

Step 4: Encode

An encoder, usually a transformer, turns the corrupted input into vectors. The encoder is the part you hope to reuse later.

Step 5: Predict and compare

A small decoder or prediction head predicts the hidden pieces. A loss compares the prediction with the true values. For categories this is cross-entropy. For numbers it is squared error or a similar loss.

Step 6: Update weights

Gradient descent adjusts the encoder and the head to reduce the loss. Over millions of examples, the encoder improves at modeling the data.

Step 7: Move to the real task

Typical options:

• Linear probe: freeze the encoder and train a simple classifier on its features. This tests how good the features are.
• Fine-tune: continue training all or some weights on labelled data. See full fine-tuning and LoRA.
• Prompt or in-context use: show examples to the pre-trained model without any weight update.

Variants of the signal

Masking

Hide a fraction of tokens. Language models often hide around fifteen percent. Image models can hide much more, since neighbouring patches are highly redundant. Tabular models choose a rate that balances difficulty against how much context remains.

Corruption

Instead of hiding, replace a value with another sampled value from the same column. The model must detect which cells were swapped and restore them. This works well for tables because swapping within a column keeps the marginal distribution realistic.

Contrastive views

Make two altered views of one item. Train the encoder so the views land close and other items land far. The loss often used is InfoNCE. For tables, a view can be a row with some features masked or noised.

Distillation without negatives

A student network predicts the output of a teacher network, where the teacher is a slow average of the student. This avoids collapse through careful design, and needs no labelled negatives.

Avoiding collapse and shortcuts

• Collapse: the model outputs the same vector for everything, which trivially lowers some losses. Contrastive negatives, stop-gradient tricks, centering and variance terms all prevent it.
• Shortcuts: the model solves the task using an artefact, such as an ID column that reveals order. Remove such columns and randomize ordering.

Key concepts and terms

A worked example

The example below builds masked-cell pre-training for a small numeric table in PyTorch. The model learns to restore hidden cells, then its encoder is reused for a labelled task.

Pre-training with masked cells

import torch
import torch.nn as nn

torch.manual_seed(0)

n_rows, n_cols = 2000, 6
# a toy table: columns are linearly related with noise
base = torch.randn(n_rows, 2)
X = torch.cat([
    base,
    base[:, :1] * 2 + 0.1 * torch.randn(n_rows, 1),
    base[:, 1:] * -1 + 0.1 * torch.randn(n_rows, 1),
    base.sum(1, keepdim=True) + 0.1 * torch.randn(n_rows, 1),
    torch.randn(n_rows, 1),
], dim=1)

encoder = nn.Sequential(nn.Linear(n_cols * 2, 64), nn.ReLU(), nn.Linear(64, 32))
decoder = nn.Linear(32, n_cols)
opt = torch.optim.Adam(list(encoder.parameters()) + list(decoder.parameters()), lr=1e-3)

for step in range(500):
    idx = torch.randint(0, n_rows, (128,))
    x = X[idx]
    mask = (torch.rand_like(x) < 0.3).float()      # 1 where hidden
    x_in = torch.cat([x * (1 - mask), mask], dim=1) # hide values, pass mask flags
    pred = decoder(encoder(x_in))
    loss = (((pred - x) ** 2) * mask).sum() / mask.sum()
    opt.zero_grad(); loss.backward(); opt.step()

print("masked-cell loss:", float(loss))

What happens here:

• Thirty percent of cells are hidden at random in every batch.
• The model sees the visible cells and a mask flag, so it knows which values are missing.
• The loss counts only the hidden cells.
• Because column three is roughly twice column one, the model learns to fill it from column one. That is exactly the dependency structure we want it to learn.
• The sixth column is pure noise, so the model cannot predict it, and its error stays high. That is a useful signal about the data itself.

Reusing the encoder

# small labelled set (200 rows) for a downstream target
y = (X[:, 0] + X[:, 1] > 0).long()
idx = torch.arange(200)

for p in encoder.parameters():
    p.requires_grad = False                       # linear probe: freeze encoder

head = nn.Linear(32, 2)
opt = torch.optim.Adam(head.parameters(), lr=1e-2)

feats = encoder(torch.cat([X[idx], torch.zeros(200, n_cols)], dim=1))
for _ in range(200):
    loss = nn.functional.cross_entropy(head(feats), y[idx])
    opt.zero_grad(); loss.backward(); opt.step()
print("probe loss:", float(loss))

The probe uses only 200 labels. A good encoder makes such a small head enough. If the probe works as well as a model trained on all rows, the pre-training captured the structure that matters.

Evaluating a self-supervised model

Evaluation is harder than in supervised learning, because the pre-training loss is not the thing you care about. A sound evaluation looks at several angles.

Linear probe

Freeze the encoder, train a single linear layer on a labelled task, and report the score. It isolates feature quality from fine-tuning power. A low-capacity probe cannot hide weak features.

Label-efficiency curve

Train the downstream model with 1, 5, 10, 50 and 100 percent of the labels, and plot the score. Pre-training should help most at the low end. If the curve is flat, the features add little.

Fine-tuning gain

Fine-tune the whole model and compare against training the same architecture from random weights. The gap is the value of pre-training.

Robustness checks

• Add noise or drop columns at test time. Good features degrade gently.
• Shift the distribution, for example train on one region and test on another.
• Shuffle rows and columns. A tabular model should be stable to order.

Probing for structure

For tables, you can test whether the learned embedding of a column predicts its type, its range or its semantic role. These probes show that the model captured meaning without labels for the target task.

Pitfalls in evaluation

• Using test rows in pre-training.
• Tuning the pretext design on the test score.
• Reporting only one downstream task.
• Comparing to a baseline that had less tuning effort.

Designing masks for tables

The mask is the heart of tabular self-supervision, so it deserves care.

Random cell masking

Each cell is hidden with a fixed probability. It is simple and teaches local dependencies. It can be too easy when a column has an almost identical twin.

Column masking

A whole column is hidden. The model must rebuild it from the rest. This is closer to real tasks such as predicting a target field and teaches broad dependencies.

Row-block masking

Hide a group of cells in the same row. This mimics real records where a whole section, such as an address block, is missing.

Structured missingness

In real data, missingness depends on other fields. A survey respondent who skips one question may skip the next. Sampling masks that follow such patterns makes the model robust to real gaps.

Mask rate schedule

Start with a modest rate and raise it as training goes on, or sample a different rate for every batch. A varied rate forces the model to work at many difficulty levels.

Type-aware targets

Leakage inside the table

Two columns may be derived from each other, such as a date and its year. Masking one while leaving the other makes the task trivial. Detect near-duplicate columns first and mask them together.

Strengths and limitations

Strengths

• No labels needed: uses data that already exists.
• Scales with data: more unlabelled examples keep improving the features.
• Label efficiency: downstream tasks need fewer annotated examples.
• General features: one pre-trained model supports many tasks.
• Robustness: models often handle shifts and noise better than those trained on small labelled sets.
• Fits tables: masking and corruption map naturally to cells, rows and columns.

Limitations

• Compute cost: pre-training large models is expensive.
• Pretext mismatch: a pretext task that does not align with the target task can give features that help little.
• Shortcut risk: models can solve the pretext task by exploiting artefacts.
• Hard to evaluate: pre-training loss does not always predict downstream performance.
• Design sensitivity: masking rates, augmentations and temperatures can swing results.
• Bias inheritance: the model absorbs biases in the raw data.
• Tables are heterogeneous: no single pretext task suits every schema, so careful design is needed.

How it compares with alternatives

Self-supervised learning and transfer learning are partners. One produces the pre-trained model. The other describes reusing it.

Role in tabular foundation models

Tabular foundation models need a way to learn from tables at scale without a label for every table. Self-supervision provides several routes, and each one connects to a concrete table feature.

Row and column structure

A table has two axes. Self-supervised objectives can target either.

• Column prediction: mask a column and predict it from the others. This teaches column dependencies.
• Row comparison: build two noisy versions of a row and teach the encoder that they belong together. This teaches what a row looks like.
• Set structure: because row order is arbitrary, objectives are built to be invariant to shuffling.

Mixed types

A single objective must cover numbers, categories and text.

• Numeric cells use a regression loss, or are binned and predicted as classes.
• Categorical cells use cross-entropy over the column's values.
• Text cells can be embedded by a language encoder and reconstructed in embedding space.

Missing values and imputation

Masked-cell pre-training is imputation training. The model is repeatedly asked to fill hidden cells, which is the same skill needed to fill real gaps. This is why masked cell modeling and imputation networks are so closely tied to tabular pre-training. Real missingness is not random, so good pipelines also simulate structured missingness during pre-training.

No column names

Many datasets have unhelpful headers. Self-supervised objectives that rely only on values and relationships, not on names, produce encoders that work when names are missing. When names exist, they can be added as extra tokens for a boost.

Pre-training on synthetic or real tables

• Real tables supply true correlations but limited volume and legal limits.
• Synthetic tables can be sampled without limit from random generators, giving diverse structure and exact ground truth. Prior-fitted models such as TabPFN and TabICL lean on this idea, although they train with a supervised loss on generated tasks rather than masking.
• Both can be combined with self-supervised objectives to widen coverage.

In-context learning

After pre-training, a transformer can take labelled rows and unlabelled rows together and predict without gradient updates. The representation quality built during pre-training decides how well attention finds relevant rows. See transformers.

Continual learning

When new tables arrive, naive fine-tuning can erase earlier skills. Self-supervised objectives can be mixed with replay of older data, using experience replay, or with weight constraints such as EWC, so that the model keeps old structure while it adds new.

Anomaly detection

A model that predicts cells well on normal data will predict poorly on abnormal data. The reconstruction error of a masked cell or row is therefore a natural anomaly score. See anomaly detection.

Sector and semantic classification

Representations learned without labels cluster by meaning. A small head trained on a few labelled tables can then name the sector or semantic type of a column or dataset. See semantic sector heads.

Further reading on how these parts fit into a full system is on the research, architecture and benchmarks pages.

Practical guidance and best practices

• Pick a pretext task that matches the end goal. For imputation, use masked cells. For retrieval, use contrastive views.
• Tune the mask rate. Too low is trivial, too high leaves no context.
• Mask whole columns sometimes, not only random cells, so the model learns to rebuild a field from the others.
• Normalize numeric columns before pre-training so no column dominates the loss.
• Use column-specific losses so rare categories and wide numeric ranges are treated fairly.
• Remove leaking columns, such as IDs and timestamps that reveal row order.
• Shuffle rows and columns during training to enforce invariance.
• Evaluate with a linear probe on several downstream tasks, not only on pre-training loss.
• Keep a supervised baseline for comparison. Self-supervision should beat it when labels are scarce.
• Combine objectives when helpful, for example masked prediction plus contrastive loss.
• Record the pretext design in the model documentation so users know what the model has seen.
• For continual updates, mix new data with a sample of old data to limit forgetting.

Choosing between methods

Scale, compute and data planning

How much data to gather

Aim for breadth first. Many varied sources beat one huge uniform source, because the goal is general features. For tables, breadth means many schemas, sizes, domains and missingness patterns.

Deduplicate

Repeated rows or near-copies make the pretext task easy and waste compute. Remove exact duplicates and, where possible, near duplicates. See deduplication for ideas.

Curriculum

Easy examples first and harder ones later can stabilize early training. This connects to curriculum learning.

Budgeting compute

Split the budget between model size, data size and training steps. A larger model trained for too few steps underperforms a smaller one trained to convergence. Monitor validation reconstruction loss and stop when it flattens.

Keep a held-out set

Hold back a slice of tables, not only a slice of rows, so you can test how the model behaves on schemas it has never seen.

Reuse, do not retrain

Once pre-trained, store the checkpoint and reuse it. Adapt with light methods such as LoRA or a linear head. Retraining from scratch should be rare.

Common mistakes

• Judging success by pre-training loss alone. A low loss does not guarantee useful features.
• Allowing shortcuts, such as a sorted ID column that reveals the answer.
• Masking in a way that leaves duplicate columns visible, so the target is a copy of an input.
• Using a single loss for every column type, which lets large-scale numeric columns dominate.
• Forgetting to mask consistently between training and use, so the model sees a distribution at inference that it never saw in pre-training.
• Ignoring structured missingness. Real gaps cluster by row and by column, and random masking understates this.
• Over-training the pretext task until features specialize and transfer worse.
• Treating self-supervision as a replacement for evaluation on real labelled data.
• Skipping a baseline. Without one, you cannot tell whether pre-training helped.
• Mixing train and test rows during pre-training, which leaks information into the evaluation.

Frequently asked questions

What is self-supervised learning in simple terms?

It is learning from data where the labels are created from the data itself. You hide or alter part of an example and train the model to recover or relate it. Because the answers come from the data, no human annotation is needed.

How is it different from unsupervised learning?

Unsupervised learning usually looks for patterns such as clusters or components without any prediction target. Self-supervised learning builds a prediction target automatically and trains with the same machinery as supervised learning. The result is a model that produces reusable representations rather than just a grouping.

What is a pretext task?

A pretext task is the artificial problem the model solves during pre-training, such as filling a masked word, restoring a masked cell or matching two views of an image. Solving it well forces the model to learn structure that also helps on real tasks. The pretext task itself is usually not the goal.

Why does masking work so well?

To recover what was hidden, the model has to understand how the visible parts relate to it. That requires learning grammar in text, shape in images and column dependencies in tables. The more context the model can use, the better it gets, so masking turns raw data into a rich teacher.

Can self-supervised learning work on tabular data?

Yes. Typical objectives mask cells, swap values within a column, or compare noisy views of a row. They teach a network how columns relate and how rows are distributed. The gain is usually largest when labels are scarce or when many unlabelled tables are available.

Is contrastive learning a kind of self-supervised learning?

Yes. Contrastive learning creates its signal by treating two views of the same item as positives and other items as negatives. It is one of several self-supervised families, alongside masked prediction, denoising and self-distillation.

How do I know if the pre-training helped?

Compare against a supervised baseline on the same downstream task with the same labelled data. A linear probe on frozen features is a quick test of feature quality. If the pre-trained model wins, especially with few labels, pre-training helped.

Does self-supervised pre-training replace fine-tuning?

Not always. Pre-training gives general features, and some tasks work with a frozen encoder and a small head. Others need fine-tuning to reach the best results. In-context models may need neither, as they adapt by reading examples in the input.

What are the main risks?

The main risks are shortcuts, where the model solves the pretext task using an artefact, and collapse, where all inputs map to the same output. Another risk is bias carried over from the raw data. Careful design, shuffling and evaluation on varied downstream tasks reduce these risks.

How much data does it need?

There is no fixed number. Language and vision models use very large sets because their parameter counts are large. For tables, a smaller encoder can gain from far less data, and synthetic generators can add more variety. What matters most is diversity that matches the target use.

Glossary

• Augmentation: a label-preserving change to an input, such as noise or cropping.
• Autoencoder: a network that compresses an input and rebuilds it.
• Collapse: a failure in which every input gets the same representation.
• Contrastive loss: a loss that pulls positives together and pushes negatives apart.
• Corruption: a deliberate change to the input, such as swapping a value.
• Downstream task: the real task that uses the pre-trained model.
• Embedding: a vector representation of an item.
• Encoder: the network that produces representations.
• Fine-tuning: continued training on a labelled task.
• Imputation: filling in missing values.
• Linear probe: a simple classifier used to test frozen features.
• Mask: a marker that hides part of the input.
• Negative pair: two items that should be far apart.
• Positive pair: two views of the same item that should be close.
• Pre-training: the first, general learning stage.
• Pretext task: the artificial task that yields a training signal.
• Representation learning: learning useful vector summaries of data.
• Shortcut: an unintended way to solve a task that does not need understanding.
• Teacher and student: two networks in self-distillation, one guiding the other.

Summary

Self-supervised learning trains models by creating labels from the data itself, through masking, corruption, comparison of views or self-distillation. It lets large unlabelled collections teach a model general structure, which it can then reuse with few labels. It is the base of modern language, vision and increasingly tabular models.

For tables, masked-cell prediction doubles as imputation training, corruption objectives support anomaly detection, and contrastive views build row embeddings. These feed in-context learning, continual learning and semantic classification. Pick a pretext task that matches the goal, avoid shortcuts, and judge results on downstream tasks. Related reading: contrastive learning, transfer learning and transformers.

Related

It belongs to Learning paradigms in my stack, under AI and machine learning. Others in the same group: