Contrastive learning

Overview

Contrastive learning trains an encoder so that two views of the same item end up close together in embedding space, while views of different items end up far apart. The model is never told what an item is. It only learns which pairs belong together. The result is a map where distance means similarity.

This approach underlies many image encoders, text and image matching models, and retrieval systems. For tables, it gives row and column embeddings that can be used for similarity search, clustering, classification with few labels and anomaly scoring. It is a member of the self-supervised learning family and a common partner of transfer learning.

What it is

A contrastive method has four ingredients.

1. Items: images, sentences, table rows, or anything else.
2. A way to make views: two altered copies of one item, or two matched items from different sources.
3. An encoder: a neural network that maps each view to a vector.
4. A contrastive loss: a score that rewards closeness for positive pairs and distance for negative pairs.

An everyday analogy

Picture a party where each guest has a twin who wears a different outfit. You teach a helper to place people in a room so that twins stand together and everyone else spreads out. Nobody tells the helper names or jobs. After many rounds, people with similar traits also tend to stand near each other, because the helper had to find stable features that survive a change of clothes.

Positive and negative pairs

• Positive pair: two views that should match, such as two crops of one photo, a caption and its image, or two noisy versions of one row.
• Negative pair: views of different items that should be apart.

Supervised flavor

When labels exist, items with the same class can be treated as positives. This is called supervised contrastive learning. It makes class-aware embeddings and is often more stable than plain cross-entropy for small datasets.

What the embedding space looks like

After training, the space has two properties often named in the literature.

• Alignment: positive pairs are near each other.
• Uniformity: all embeddings spread evenly over the space, so information is kept rather than squashed together.

A good method balances the two. Without uniformity, everything collapses to one point.

Where it came from

The roots go back to metric learning, where networks were trained so that distances reflected similarity. Early siamese networks used pairs of inputs with a shared encoder and a margin-based loss. Triplet losses followed, using an anchor, a positive and a negative.

Noise-contrastive ideas

Estimating a model by comparing data against noise samples appeared in statistics before it reached deep learning. The same idea gave rise to losses such as InfoNCE, which treats picking the right partner among many candidates as a classification problem.

Visual representation boom

Around 2019 and 2020, methods such as MoCo and SimCLR showed that contrastive pre-training on images, using strong augmentations, could match or beat supervised pre-training on several transfer tasks. MoCo introduced a queue and a slowly updated encoder to supply many negatives. SimCLR relied on large batches and a projection head.

Beyond negatives

Later methods such as BYOL and DINO dropped explicit negatives and relied on self-distillation, which showed that careful design can avoid collapse without them. These methods are close cousins and are often discussed together with contrastive approaches.

Image and text

Matching images with their captions at large scale produced models such as CLIP, which embed pictures and text in one shared space. This enabled zero-shot classification, where class names are turned into text prompts. See zero-shot learning.

Tables

Contrastive ideas reached tabular data by creating views through feature masking, value swapping and noise. They also appear in models that align a table cell with its column header or with a textual description.

How it works

The following steps describe the usual training loop.

Step 1: Sample a batch of items

Take N items from the dataset, for example N table rows.

Step 2: Make two views of each item

Apply random changes that keep meaning but alter details.

• Images: crop, flip, change color, blur.
• Text: drop words, swap near synonyms, use different spans.
• Table rows: mask some cells, add small noise to numbers, replace a few values with samples from the same column.

This gives 2N views. The pair from the same item is positive. The other 2N minus 2 views are negatives.

Analogy: make two photocopies of a page, one smudged and one slightly rotated. A good reader should still know they come from the same page.

Step 3: Encode

Pass each view through the encoder to get a vector. Many methods then pass the vector through a small projection head, which is a short network used only during training. The loss is computed on the projection. The encoder output before the head is kept for downstream use.

Step 4: Compare vectors

Use cosine similarity, which measures the angle between vectors. Vectors are normalized to unit length, so only direction matters.

Step 5: Apply the loss

The widely used InfoNCE loss for one anchor is:

loss = -log( exp(sim(a, p) / T) / sum_over_all_candidates exp(sim(a, c) / T) )

Here a is the anchor, p its positive, c runs over the positive and all negatives, and T is a temperature. The loss is low when the positive scores higher than every negative.

Step 6: Update

Gradient descent changes the encoder so positives become closer and negatives more distant.

Step 7: Use the embeddings

After training, the encoder is reused for tasks such as nearest-neighbour search, clustering, or training a small classifier on top. See full fine-tuning and LoRA for adapting further.

The role of temperature

The temperature T controls how sharply the loss focuses on hard negatives.

• Low T: the loss concentrates on the closest negatives, which gives fine detail but can overfit to noise or false negatives.
• High T: the loss treats all negatives about equally, which is smoother but less discriminative.

Where negatives come from

False negatives

An item treated as a negative may in fact belong to the same class as the anchor. In a table of customers, two different rows may be near-duplicates. Treating them as negatives pushes similar items apart and hurts the embedding. Methods reduce this by filtering close negatives, using labels where available, or lowering the weight of very similar pairs.

Key concepts and terms

A worked example

This example trains a contrastive encoder on table rows with PyTorch. The views are made by masking cells and adding noise. It then uses the embeddings for nearest-neighbour lookup.

Building the training loop

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# toy table with three hidden groups, six numeric columns
centers = torch.randn(3, 6) * 3
labels = torch.randint(0, 3, (1500,))
X = centers[labels] + torch.randn(1500, 6)

def make_view(x, p_mask=0.3, sigma=0.2):
    mask = (torch.rand_like(x) < p_mask).float()
    noisy = x + sigma * torch.randn_like(x)
    return noisy * (1 - mask)

encoder = nn.Sequential(nn.Linear(6, 64), nn.ReLU(), nn.Linear(64, 32))
proj = nn.Sequential(nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 16))
opt = torch.optim.Adam(list(encoder.parameters()) + list(proj.parameters()), lr=1e-3)

def info_nce(z1, z2, T=0.2):
    z1, z2 = F.normalize(z1, dim=1), F.normalize(z2, dim=1)
    z = torch.cat([z1, z2], dim=0)                 # 2N x d
    sim = z @ z.T / T
    n = z1.size(0)
    sim.fill_diagonal_(float("-inf"))              # remove self-similarity
    targets = torch.cat([torch.arange(n, 2*n), torch.arange(0, n)])
    return F.cross_entropy(sim, targets)

for step in range(600):
    idx = torch.randint(0, X.size(0), (256,))
    x = X[idx]
    z1 = proj(encoder(make_view(x)))
    z2 = proj(encoder(make_view(x)))
    loss = info_nce(z1, z2)
    opt.zero_grad(); loss.backward(); opt.step()

print("final loss:", float(loss))

Notes on the code:

• Two independent views of each row form the positive pair.
• The other rows in the batch act as negatives.
• The diagonal of the similarity matrix is blocked so a view cannot match itself.
• The projection head is discarded after training. The encoder output is kept.

Using the embeddings

with torch.no_grad():
    E = F.normalize(encoder(X), dim=1)

query = E[0]
scores = E @ query
top = scores.topk(6).indices
print("labels of nearest rows:", labels[top].tolist(), "query label:", int(labels[0]))

If training worked, the nearest neighbours of a row mostly share its hidden group, even though no group label was used in training. That is the practical benefit: a label-free embedding that reflects structure in the data.

What to check next

• Plot the embeddings with a 2-D reduction such as UMAP and see whether groups separate.
• Train a linear classifier on 50 labelled rows and compare against a model on raw features.
• Vary the temperature and the mask rate to see how sensitive the result is.

Well-known contrastive methods

SimCLR in plain words

Take an image, make two heavy random edits, encode both, and require them to match while all other images in the batch do not. Large batches provide many negatives. A projection head helps the encoder keep richer features.

MoCo in plain words

Instead of needing a huge batch, keep a long queue of recent embeddings as negatives. The encoder that produces queue entries moves slowly, so the entries stay consistent. This gives many negatives with modest hardware.

Self-distillation relatives

BYOL and DINO have a student network predict a teacher network's output for another view. The teacher is a slow average of the student. They avoid negatives and rely on design choices such as centering and stop-gradient to stay away from collapse.

Applications beyond pre-training

Search and retrieval

Embed every item once, store the vectors in an index, and answer a query by finding its nearest neighbours. This is the base of semantic search, duplicate detection and recommendation. See pgvector and FAISS.

Deduplication and entity matching

Two records that describe the same real-world entity should embed close together. Contrastive training on matched pairs gives a matcher that tolerates typos, missing fields and reordered words.

Few-label classification

Embed a handful of labelled examples. Classify a new item by the nearest class centre or by a small classifier. This works well when labels are scarce.

Cross-modal matching

Train two encoders, one per modality, so that matched pairs land together. This enables searching images with text, or tables with a text description.

Clustering and exploration

Embeddings give clustering methods a better starting point than raw columns. Reduce them with a method such as UMAP to see groups.

Debugging a contrastive run

Symptom: loss stays high

• Check that positives are really matched. A bug in pairing is the most common cause.
• Lower the temperature slightly, or raise the learning rate.
• Make sure views are not so strong that they erase the item.

Symptom: loss falls quickly to near zero

• The task may be trivial. Views may share an artefact such as the same noise pattern.
• Strengthen augmentations or remove the shortcut.

Symptom: embeddings look identical

• This is collapse. Check embedding variance across the batch.
• Add more negatives, a projection head, or a variance penalty.

Symptom: good loss, poor downstream score

• Invariances may be wrong. The views might remove information the task needs.
• Try a different view design, then test with a linear probe.

Symptom: unstable training

• Reduce the learning rate and use warm-up.
• Normalize embeddings and clip gradients.

Metrics worth logging

• Mean cosine similarity of positive pairs. It should rise during training.
• Mean cosine similarity of negative pairs. It should stay low.
• Standard deviation of each embedding dimension across a batch. It should not shrink toward zero.
• Top-1 retrieval accuracy of a view among the batch. It shows how well positives are found.
• Nearest-neighbour accuracy on a small labelled validation set. It tracks downstream value.
• Gradient norm. Spikes point to a learning rate that is too high.

A note on batch composition

For tables, avoid putting many rows from the same entity in one batch, because they will be treated as negatives of each other. Sample rows from different groups, or group-aware sample so that same-entity rows are positives.

Strengths and limitations

Strengths

• Learns from unlabelled data, with labels optional.
• Produces embeddings useful for search, clustering and few-label classification.
• Works across data types: images, text, audio, tables.
• Encourages invariance to nuisance changes that the augmentations mimic.
• Scales well with more data and larger batches or queues.
• Gives a shared space for two modalities when pairs are available, as in image and text matching.

Limitations

• Depends on good views. Poor augmentations teach the wrong invariances.
• Needs many negatives for strong results with some losses, which costs memory.
• False negatives can harm quality.
• Sensitive to temperature, batch size and learning rate.
• Collapse is possible when the setup is wrong.
• Embedding quality is hard to measure without downstream tasks.
• For tables, there is no standard set of augmentations, so views must be designed with care. A wrong choice can destroy meaning, for example adding noise to an ID or flipping an ordered category.

How it compares with alternatives

In practice many systems mix several of these. For example, an encoder can be trained with masked prediction and a contrastive term together.

Role in tabular foundation models

Tabular foundation models aim to build useful representations from raw tables. Contrastive learning supplies tools for this at several levels.

Row and column structure

A table has rows (records) and columns (fields). Contrastive objectives can work on either.

• Row-level: two corrupted views of the same row are positives. Other rows are negatives. The encoder learns to describe a row in a way that survives missing cells and noise.
• Column-level: a column viewed from two random subsets of its values is a positive pair. The encoder learns a signature of what the column holds, such as range, shape and category set. This helps when headers are missing.
• Cell-to-context: a cell and its row are positives. This teaches how a value fits into its record.

Mixed types

Views must respect types. Numbers can receive small noise or be re-binned. Categories can be swapped with another value from the same column. Text can be dropped or paraphrased. Applying numeric noise to a category would break meaning, so views must be type-aware.

Missing values

Masking cells is a natural augmentation. The encoder is trained to give similar embeddings with or without a few cells, which makes it robust to real missing data. It also supports imputation networks: the embedding of a partially observed row can be used to find similar complete rows and borrow their values.

No column names

Because the loss uses only values and relations, a contrastive encoder works when headers are meaningless. Column signatures learned from values can even recover the likely role of a column, for example a date, an amount or a postal code.

Pre-training on synthetic or real tables

Synthetic tables are well suited. A generator can create many tables with known structure, and views of a row are easy to make. Real tables add realistic distributions. Mixing both gives coverage and realism. Prior-fitted models such as TabPFN learn differently, with a supervised objective on generated tasks, but contrastive training can supply row embeddings for retrieval steps in larger systems such as TabICL.

In-context learning

In an in-context tabular model, a test row must find relevant labelled rows. A contrastive embedding makes this retrieval easy, because rows that are alike sit near each other. It can also be used to choose which training rows to place in a limited context window. See transformers.

Class-aware embeddings

With labels, supervised contrastive losses pull rows of the same class together. This yields embeddings in which classes are tight clusters, which helps few-shot classification. A related idea appears in models that align text and tabular features, such as TabSTAR.

Continual learning

When new tables arrive, an embedding space can drift. Keeping a small replay buffer of old rows, as in experience replay, and including them as extra positives and negatives helps the space stay stable. Constraints such as EWC can also limit change.

Anomaly detection

Normal rows form dense regions in the embedding space. A row far from all others, or whose views do not agree, is suspicious. Distance to nearest neighbours or to a class centre gives an anomaly score. See anomaly detection.

Sector and semantic classification

Embeddings of columns and tables cluster by meaning. A few labelled examples are enough to train a head that names a column's semantic type or a dataset's sector. See semantic sector heads.

For how these ideas fit a complete system, see the research, architecture and benchmarks pages.

Practical guidance and best practices

• Design views that keep meaning. Ask: would a domain expert say these two views describe the same record?
• Use type-aware augmentations for tables. Different handling for numbers, categories, dates and text.
• Normalize numeric columns before adding noise, so noise scale is comparable across columns.
• Start with a temperature around the range that papers report for your kind of data, then tune on a validation task.
• Increase the number of negatives with a larger batch or a memory queue when results are weak.
• Watch for collapse. Track the standard deviation of embeddings across a batch. It should stay well above zero.
• Use a projection head and discard it at the end.
• Remove or down-weight near-duplicate rows, to limit false negatives.
• Normalize embeddings to unit length before cosine search.
• Evaluate with a linear probe, nearest-neighbour accuracy and a label-efficiency curve.
• Consider a supervised contrastive loss when you have labels, since it gives class-aware clusters.
• For retrieval, index embeddings with an approximate nearest-neighbour library; see FAISS and HNSW.
• Version your encoder with the index. Embeddings from different encoders cannot be compared.

A simple decision guide

Common mistakes

• Using augmentations that change meaning, such as adding noise to an identifier.
• Treating all other rows as negatives when many are near-duplicates.
• Setting temperature too low, which makes training unstable.
• Keeping the projection head for downstream use, which often lowers quality.
• Forgetting to normalize embeddings, so similarity depends on vector length.
• Training with too small a batch and no queue, giving too few negatives.
• Evaluating only on the pretext loss, which says little about downstream value.
• Comparing embeddings from different training runs or encoders as if they shared a space.
• Ignoring class imbalance, which makes frequent items dominate the negatives.
• Not testing the invariances. If the model ignores a feature you need, check that your views did not erase it.

Frequently asked questions

What is contrastive learning in one sentence?

It is a way of training an encoder so that matching views of an item are close in embedding space and views of different items are far apart. The matching is defined by how pairs are built, not by human labels. The result is a space where distance reflects similarity.

What is the difference between a positive and a negative pair?

A positive pair is two views that should be treated as the same thing, such as two crops of one photo or two noisy versions of one table row. A negative pair is two views that come from different items and should be pushed apart. The loss rewards high similarity for positives and low similarity for negatives.

What is InfoNCE?

InfoNCE is a loss that asks the model to pick the true partner of an anchor out of a set of candidates. It uses a softmax over similarity scores, scaled by a temperature. Minimizing it pulls positives together and pushes negatives away.

Do I need labels for contrastive learning?

No. The standard form uses augmentations of the same item to make positives, so no labels are needed. If you do have labels, a supervised variant treats same-class items as positives and gives class-aware embeddings.

Can contrastive learning be used on tables?

Yes. You make views of a row by masking cells, adding noise to numbers or swapping values within a column. The encoder then learns row embeddings that survive missing and noisy data. These embeddings can support search, clustering and few-label classification.

What is collapse and how do I avoid it?

Collapse is when the encoder maps every input to nearly the same vector, which makes the loss easy but the embedding useless. Negatives, a projection head, normalization, variance terms and stop-gradient tricks all help. Monitoring the spread of embeddings in a batch lets you spot it early.

Why does temperature matter?

Temperature scales the similarity scores before the softmax. A low value makes the loss focus on the hardest negatives, which can sharpen detail but amplify noise. A high value spreads attention across all negatives, which is smoother but less precise. It is an important setting to tune.

How is contrastive learning different from an autoencoder?

An autoencoder is trained to rebuild its input, so it must keep details needed for reconstruction. A contrastive encoder is trained to tell items apart and to be stable under changes, so it keeps what identifies an item and discards what the views vary. This often produces embeddings that are better for comparison and retrieval.

What are false negatives?

A false negative is a pair treated as different when the items actually belong together, such as two near-identical rows. Pushing them apart harms the embedding. Filtering, using labels, or reducing the weight of very close pairs limits the damage.

When should I not use contrastive learning?

Avoid it when you cannot design views that preserve meaning, when the dataset is tiny and a simple model suffices, or when you only need a direct prediction with plenty of labels. In those cases supervised learning or reuse of an existing pre-trained encoder is usually simpler.

Glossary

• Alignment: positives being close in the embedding space.
• Anchor: the reference item in a comparison.
• Augmentation: a change that alters an item without changing its meaning.
• Cosine similarity: the cosine of the angle between two vectors.
• Embedding: a vector representation of an item.
• Encoder: the network that computes embeddings.
• False negative: a negative pair that is actually a match.
• Hard negative: a negative that is close to the anchor.
• InfoNCE: a contrastive loss based on picking the true partner among candidates.
• Memory queue: a store of past embeddings used as negatives.
• Metric learning: learning a distance that reflects similarity.
• Projection head: a short network used only in training before the loss.
• Siamese network: two branches that share weights.
• Temperature: a scale in the softmax of the loss.
• Triplet loss: a margin loss on an anchor, a positive and a negative.
• Uniformity: embeddings spreading evenly over the space.
• View: an altered copy of an item.

Summary

Contrastive learning teaches an encoder to place matching views of an item near each other and different items far apart. Losses such as InfoNCE, careful views, enough negatives and a sensible temperature are the main ingredients. The learned embeddings support search, clustering, few-label classification and anomaly scoring.

For tabular foundation models, contrastive objectives give robust row and column embeddings that handle mixed types, missing values and absent headers. They help in-context models find relevant rows, support continual updates with replay, and connect to semantic classification. Design views with care, guard against collapse and false negatives, and evaluate on downstream tasks. Related reading: self-supervised learning, transfer learning and transformers.

Related

It belongs to Learning paradigms in my stack, under AI and machine learning. Others in the same group: