SimCLR

Overview

SimCLR is a self-supervised framework that teaches a neural network to produce useful image features without any labels. It takes one image, makes two randomly changed copies of it, and trains the network so that the two copies end up close together in feature space while copies of different images end up far apart. The name stands for "a simple framework for contrastive learning of visual representations," and the word "simple" is the point: it needs no special architecture, no memory bank and no custom layers.

After this pre-training, the learned encoder can be reused. A small classifier placed on top of frozen features, or a short fine-tuning run, often reaches strong accuracy with far fewer labeled examples. This article explains how SimCLR works, shows a small code walk-through, compares it with relatives such as MoCo, BYOL and DINO, and then explains how the same contrastive idea carries over to models that learn from raw tables.

What it is

SimCLR is a recipe for self-supervised learning. In self-supervised learning the training signal comes from the data itself. There is no human tagger. The task is made up from the data so that solving it forces the network to learn something meaningful.

The made-up task in SimCLR is an agreement game. Two views of the same image should agree. Views of different images should disagree. This family of ideas is called contrastive learning.

The four main parts

What you get at the end

• A pre-trained encoder that maps an image to a feature vector.
• The projection head is thrown away after training.
• The feature vector is used for classification, retrieval, clustering or transfer to other tasks.

Where it came from

Contrastive ideas are old. Early work on metric learning trained pairs of networks with shared weights, called Siamese networks, to tell whether two inputs matched. Later work introduced the idea of treating every instance as its own class and learning to tell instances apart. Noise-contrastive estimation gave a way to train with a few negative samples instead of all of them. The InfoNCE loss tied these threads to mutual information between views.

Several papers in the late 2010s built on these ideas for images. Some used memory banks of stored features. Some used specially designed pretext tasks, such as predicting image rotation or solving jigsaw puzzles. Others predicted future parts of a sequence.

The SimCLR paper, published in 2020 under the title "A Simple Framework for Contrastive Learning of Visual Representations," simplified the picture. It showed that three choices mattered most:

• Strong and well-chosen data augmentation, especially the pairing of random cropping with color distortion.
• A nonlinear projection head between the encoder and the loss.
• Large batch sizes and long training, which supply many negative examples.

A follow-up, often called SimCLR v2, explored bigger models, a deeper projection head and a semi-supervised pipeline in which a large pre-trained model is fine-tuned on a few labels and then distilled into a smaller one. Related approaches that appeared around the same time include MoCo, which solved the large-batch cost with a queue, and BYOL, which dropped negative pairs altogether.

How it works

Imagine a teacher who shows a student two blurry, recolored photos of the same dog and asks, "same animal?" The student must say yes. The teacher then shows a photo of a cat and asks, "same as the dog?" The student must say no. Repeat this millions of times and the student learns what makes a dog a dog, without anyone ever saying the word "dog."

Step 1: Sample a batch

A batch of N images is drawn from the unlabeled dataset. No labels are used.

Step 2: Create two views of each image

Each image goes through a random augmentation pipeline twice, producing two different views. The standard pipeline includes:

• Random crop followed by resize to a fixed size.
• Random horizontal flip.
• Color distortion, meaning random changes of brightness, contrast, saturation and hue, and sometimes conversion to gray.
• Gaussian blur.

The batch now holds 2N views. For each view there is exactly one positive partner, which is the other view of the same image.

Step 3: Encode the views

Each view goes through the base encoder, producing a representation vector. This vector is the thing you will keep and use later.

Step 4: Project

Each representation goes through the projection head, a small network with one or more hidden layers and a nonlinearity. The output is a smaller vector. The loss is applied here, not on the encoder output.

Why a head? The loss asks for invariance to the augmentations, such as color. That can throw away information that is useful elsewhere. The head absorbs that loss of information, so the encoder output keeps richer detail. This is why the head is discarded after training.

Step 5: Compute the contrastive loss

SimCLR uses a loss called NT-Xent, short for normalized temperature-scaled cross entropy. For one positive pair (i, j), the loss is:

loss(i, j) = -log( exp(sim(z_i, z_j) / T) /
                   sum over k != i of exp(sim(z_i, z_k) / T) )

Here z is a projected vector, sim is cosine similarity, and T is the temperature. The sum in the denominator runs over all other views in the batch, so each view is compared with 2N - 2 negatives. The loss is averaged over all positive pairs, in both directions.

In plain words: for each view, the model must pick out its true partner from a crowd of 2N - 1 candidates. The loss is a classification loss over that crowd.

Step 6: Update

The gradient flows through the head and the encoder. Both are updated. Because many negatives help, SimCLR was originally trained with very large batches, thousands of images at a time, with optimizers designed for large-batch training.

Step 7: Evaluate

The common test is linear evaluation. Freeze the encoder, put a single linear layer on top, train it with labels, and measure accuracy. A good encoder gives a high score with this tiny classifier. Other tests include fine-tuning with few labels and transfer to new datasets.

The role of temperature

Temperature controls how sharply the loss focuses on hard negatives.

• A low temperature makes the loss concentrate on the most confusable negatives.
• A high temperature spreads the weight more evenly.
• Too low can cause unstable training. Too high can make the signal weak.
• Good values are found by tuning.

Why augmentation is the real teacher

The network only learns what the augmentations ask it to ignore. If you crop and recolor, the encoder learns to ignore position and color shifts and to keep shape and texture. Change the augmentations and you change what the model learns. This is the most important design lever in the method.

Everyday analogies

• Two views of one image are like two photos of the same friend taken in different light. You still know it is the same friend.
• Negatives are like strangers in a crowd. The model learns to tell your friend from them.
• The projection head is like a translator who handles a narrow job, so the main thinker keeps a full vocabulary.
• Temperature is like how strict a judge is. A strict judge punishes close mistakes harder.
• Augmentation is like practicing a song in different keys. You learn the tune, not the key.

Why it learns useful features

To pick the right partner out of a crowd, the network cannot rely on color or exact position, because the augmentations scramble those. It has to notice stable things such as object parts, shapes and textures. Those are the same cues that help with classification, detection and retrieval later. That is why a task with no labels still produces features that carry meaning.

A short timeline of ideas

• Siamese networks compare two inputs with shared weights.
• Instance discrimination treats each image as its own class.
• Noise-contrastive estimation allows training with sampled negatives.
• Memory banks and queues store many negatives cheaply.
• SimCLR shows that a plain in-batch version with strong augmentation is enough.
• Later work removes negatives altogether, as in BYOL and DINO.

Key concepts and terms

A worked example

A numeric walk-through with a tiny batch

Take a batch of 2 images, A and B. After augmentation there are 4 views: A1, A2, B1, B2. Suppose the projected, normalized vectors have these cosine similarities with A1:

Use temperature T = 0.5. Then the scaled scores are 1.6, 0.2 and 0.4.

• exp(1.6) = about 4.95
• exp(0.2) = about 1.22
• exp(0.4) = about 1.49
• Sum = about 7.66
• Probability of picking the positive = 4.95 / 7.66 = about 0.646
• Loss for this view = -log(0.646) = about 0.437

If training pushes A2 closer to A1 (similarity 0.95) and B views further away (0.0 and 0.05), the positive score becomes 1.9 and the others 0.0 and 0.1. The probability rises to about 0.80 and the loss falls to about 0.22. The loss shrinks as the model learns to separate the pairs.

Code sketch in PyTorch style

The following sketch shows the loss and one training step. It is a minimal outline, not a tuned recipe.

import torch
import torch.nn as nn
import torch.nn.functional as F

def nt_xent(z1, z2, temperature=0.5):
    # z1, z2: [N, D] projections of two views
    z = torch.cat([z1, z2], dim=0)           # [2N, D]
    z = F.normalize(z, dim=1)
    sim = z @ z.t() / temperature            # [2N, 2N]
    n = z1.size(0)
    # remove self-similarity from the softmax
    mask = torch.eye(2 * n, dtype=torch.bool, device=z.device)
    sim = sim.masked_fill(mask, float("-inf"))
    # the positive for index i is i+n, and for i+n is i
    targets = torch.cat([torch.arange(n, 2 * n), torch.arange(0, n)]).to(z.device)
    return F.cross_entropy(sim, targets)

class SimCLRModel(nn.Module):
    def __init__(self, encoder, feat_dim, proj_dim=128):
        super().__init__()
        self.encoder = encoder
        self.head = nn.Sequential(
            nn.Linear(feat_dim, feat_dim),
            nn.ReLU(),
            nn.Linear(feat_dim, proj_dim),
        )

    def forward(self, x):
        h = self.encoder(x)      # keep this for downstream use
        z = self.head(h)         # use this for the loss
        return h, z

def train_step(model, optimizer, view1, view2):
    _, z1 = model(view1)
    _, z2 = model(view2)
    loss = nt_xent(z1, z2)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    return loss.item()

Using the result

After pre-training, drop the head and use the encoder:

model.eval()
with torch.no_grad():
    features, _ = model(batch_of_images)   # use `features` for a linear classifier

Variants and extensions

• SimCLR v2 uses a deeper projection head and larger encoders, and keeps part of the head when fine-tuning.
• Semi-supervised pipelines fine-tune the big pre-trained model on a few labels, then distill it into a smaller student.
• Some work swaps the ResNet for a vision transformer, which changes the best augmentation and optimizer settings.
• Others add a momentum encoder, which moves the idea toward MoCo.
• Supervised contrastive variants use labels to define positives, so that all images of one class attract each other.
• Multimodal versions pair images with text, as in CLIP.

Typical hyperparameters

These ranges are general guidance, not exact recipes. Always check them on a small run.

A tabular analog in code

This sketch shows how a "view" of a table row can be made by resampling cells from other rows. It pairs with the same NT-Xent loss shown above.

import numpy as np

def corrupt_rows(X, rate=0.3, rng=None):
    """Return a corrupted copy of X (rows x columns).
    Each chosen cell is replaced by a value from the same column
    in a random other row, so types stay valid."""
    rng = rng or np.random.default_rng()
    X = np.asarray(X, dtype=object).copy()
    n, d = X.shape
    mask = rng.random((n, d)) < rate
    donors = rng.integers(0, n, size=(n, d))
    for j in range(d):
        col = X[:, j].copy()
        X[mask[:, j], j] = col[donors[mask[:, j], j]]
    return X

rows = [[1.2, "red", 3], [0.7, "blue", 5], [2.4, "red", 4], [1.9, "green", 2]]
view_a = corrupt_rows(rows, rate=0.3)
view_b = corrupt_rows(rows, rate=0.3)
# Encode both views with a row encoder, project, and apply nt_xent(z_a, z_b).

Because donors come from the same column, a numeric column stays numeric and a category column stays categorical. This is the simplest way to get mixed types without extra rules.

Strengths and limitations

Strengths

• No labels needed for pre-training.
• Very simple design. There is no memory bank, queue or momentum network.
• Strong features: linear probes on frozen features can approach supervised results.
• Features transfer well to many downstream tasks.
• Works with common encoders such as ResNet.
• Helps in low-label settings, where it can cut labeling needs sharply.

Limitations

• Needs many negatives, which in the original recipe meant very large batches and a lot of compute.
• Sensitive to the choice of augmentations. Poor choices teach the wrong invariances.
• False negatives: two different images of the same class are treated as negatives, which pushes them apart for no good reason.
• Temperature and learning rate need careful tuning.
• Long training schedules are usually needed.
• Augmentations designed for natural images do not always make sense in other domains, such as tables or medical scans.

How it compares with alternatives

A quick rule: if you have large compute and want a simple method, SimCLR is a clear baseline. If memory is tight, consider MoCo. If you want to avoid negatives, look at BYOL or DINO.

Role in tabular foundation models

Tabular foundation models learn from raw tables. Each row is an example, each column is a feature, types are mixed (numbers, categories, dates, text), some cells are missing, and column names may be absent or unhelpful. SimCLR was built for images, but its core idea applies to tables, and many tabular methods are direct cousins of it. See /model/research for the broader landscape.

Contrastive pre-training on rows

The tabular analog of an augmented view is a corrupted row. Instead of cropping and recoloring, you can:

• Replace a random subset of cells with values drawn from the same column in other rows.
• Mask a random subset of cells.
• Add small noise to numeric columns.
• Drop a random subset of columns.

Two corrupted copies of one row form a positive pair. Other rows form negatives. The contrastive loss is the same NT-Xent idea. Published tabular methods such as SCARF and SubTab follow this pattern, and SAINT pairs a similar contrastive objective with attention across rows and columns.

Why this suits mixed types and missing values

Column corruption by resampling from the column's own distribution works for numbers and categories alike, so it does not need a separate recipe per type. Missing values are natural: a masked cell is just an artificial gap. A model trained to be stable under such gaps becomes robust to the real gaps found in operational data. This ties straight to masked cell modeling and imputation networks, which learn to fill the gaps instead of ignoring them.

Handling tables without column names

Image augmentations rely on spatial structure. Tables have none by default: column order is often arbitrary. Contrastive tabular methods therefore lean on column-wise statistics and on attention that treats columns as a set. A model trained this way can build row representations even when headers are missing, because it learns from value patterns and co-occurrence.

Synthetic pre-training and in-context learning

A different path to tabular foundation models is to pre-train on a huge number of synthetic tables and then solve new tasks by in-context learning, as in TabPFN. Here the model reads labeled rows as a prompt and predicts the labels of new rows in one forward pass. That path is not contrastive. But the two ideas can complement each other: a contrastively trained row encoder can supply embeddings that make the in-context step easier, especially for wide or noisy tables.

Anomaly detection and clustering

Contrastive row embeddings place normal rows in tight neighborhoods. Rows that land far from every neighborhood can be flagged as unusual, linking to anomaly detection. The same embeddings feed clustering for segmentation and for sector or semantic grouping of records.

Continual learning

A contrastive encoder can be updated on new batches of rows. The risk is drift, where new data pulls representations away from old ones. Replay of a few old rows, or regularizing against the earlier encoder, helps. This matters in operational settings where tables grow every day.

A cautious note

Contrastive pre-training for tables is an active research area. Results depend heavily on which corruptions are chosen, and the gains over a well-tuned gradient boosted tree are not guaranteed. See /model/benchmarks and /model/architecture for how such models are measured and built.

Practical guidance and best practices

Choosing augmentations

• Start with the standard crop plus color distortion combination for natural images.
• Remove an augmentation if it destroys the signal you care about. For example, strong color changes hurt when color is the label.
• Ablate augmentations one at a time to see which carry most of the value.
• For tables, pick corruptions that keep a row plausible, and tune the corruption rate.

Batch size and negatives

• Larger batches give more negatives and usually better features, up to a point.
• If memory is limited, use gradient checkpointing, mixed precision or a queue-based method.
• Do not shrink the batch without retuning the temperature and learning rate.

Optimization

• Use a warmup phase, then a cosine decay schedule.
• Use a large-batch optimizer when the batch is in the thousands.
• Use weight decay, but do not apply it to normalization and bias parameters if your framework makes that easy.

The projection head

• Use at least one hidden layer with a nonlinearity.
• Measure downstream accuracy on the encoder output, not the head output.
• Try different head widths; the best size is task dependent.

Evaluation protocol

• Keep a clean linear probe as a fixed yardstick.
• Also run a few-label fine-tuning test, since it is closer to real use.
• Report results over several seeds.
• Check for collapse by looking at the spread of features across a batch. If all features are nearly the same, training has failed.

Data hygiene

• Remove exact duplicates, since a duplicate of the same image in a batch becomes a harmful false negative.
• Shuffle well so that batches are diverse.
• For tables, deduplicate rows and treat groups of related rows carefully.

Scaling advice

• Start with a small dataset and a small encoder to verify the pipeline.
• Watch the loss and the linear probe score together.
• Scale up only after the small run behaves sensibly.

Debugging guide

Questions to ask before you adopt it

• Do I have a large pool of unlabeled data and few labels?
• Do I know augmentations that keep the meaning of an example?
• Can I afford the batch size and training time?
• Is a simpler baseline, such as supervised training on the labels I have, already good enough?
• How will I test that the features are useful, and on which downstream tasks?

Common mistakes

• Using weak augmentation, which makes the task too easy. The network can then solve it by matching low-level cues, such as color histograms.
• Evaluating the projection head output instead of the encoder output.
• Using a tiny batch with no other source of negatives.
• Forgetting to normalize vectors before computing cosine similarity.
• Letting the same image appear twice in a batch as separate items.
• Using a temperature copied from another setup without tuning.
• Treating the loss value as a quality score. A lower loss does not always mean better features. Use a probe.
• Skipping data checks, then wondering why the model learned shortcuts such as image borders or watermarks.
• Applying image-style augmentations to tabular data without thinking about column types.
• Judging the method on a single run.

Frequently asked questions

What does SimCLR stand for?

SimCLR stands for "a simple framework for contrastive learning of visual representations." The name reflects its aim: a minimal recipe that gets strong image features from unlabeled data using only augmentations, an encoder, a small projection head and a contrastive loss.

Is SimCLR supervised or unsupervised?

It is self-supervised, which is a form of unsupervised learning where the training signal is built from the data. The two augmented views of each image act as free labels telling the model which items belong together. Human labels are only needed later, for evaluation or fine-tuning.

Why does SimCLR need a projection head?

The loss asks the representation to ignore augmentations such as color shifts, which can discard information that is useful for other tasks. The projection head takes on that loss of information, so the encoder output stays richer. After training the head is removed and the encoder output is used.

Why does SimCLR need large batches?

Every other image in the batch serves as a negative example. More negatives give a harder and more informative contrast, which tends to produce better features. This is also the main cost of the method, and it motivated alternatives such as MoCo, which keeps negatives in a queue.

What is the NT-Xent loss?

NT-Xent is a normalized temperature-scaled cross entropy. For each view, the model scores its similarity to every other view in the batch and must give the highest probability to its true partner. The temperature controls how sharply the scores are turned into probabilities.

How is SimCLR different from BYOL?

SimCLR learns by contrasting positive pairs against many negatives. BYOL uses no negatives at all. It trains an online network to predict the output of a slowly updated target network. BYOL avoids false negatives but needs design care to prevent collapse.

Can SimCLR be used on tabular data?

Yes, in spirit. The augmentation step is replaced by row corruption, such as masking cells or swapping values from other rows, and the same contrastive loss is applied. Methods in this family exist for tables, though they must handle mixed types, missing values and the lack of spatial structure.

What encoder does SimCLR use?

The original work used convolutional ResNet encoders, but the framework is not tied to them. Any encoder that maps an input to a feature vector can be used, including vision transformers and tabular transformers.

How do I know if my SimCLR model is any good?

Freeze the encoder, train a linear classifier on top with labels, and measure accuracy. Also test with few labels and on a different dataset. Loss values alone are not reliable, because a low loss can come from shortcuts that do not help real tasks.

Does SimCLR work with small datasets?

It can, but the benefit is smaller and results are sensitive to augmentation and temperature. With little data, transfer from a model pre-trained on a larger set, or a method with fewer demands on batch size, is often a better choice.

Glossary

• Augmentation: a random change applied to an input to create a new view.
• Backbone: another word for the base encoder.
• Collapse: a failure where the model outputs the same vector for every input.
• Contrastive loss: a loss that rewards similarity for positives and dissimilarity for negatives.
• Cosine similarity: the cosine of the angle between two vectors.
• Embedding: a vector that represents an input.
• Encoder: the network that produces the representation.
• False negative: a negative pair that actually shares the same class.
• InfoNCE: a loss related to mutual information that inspired NT-Xent.
• Linear probe: a linear classifier trained on frozen features.
• Metric learning: learning a distance that reflects similarity.
• NT-Xent: normalized temperature-scaled cross entropy.
• Positive pair: two views of the same example.
• Projection head: a small network used only in pre-training.
• Pretext task: an artificial task used to learn features.
• Temperature: a scale applied to similarities before the softmax.
• View: one augmented version of an input.

Summary

SimCLR is a compact and influential recipe for learning image features without labels. It creates two augmented views of each image, encodes them, projects them through a small head, and applies the NT-Xent loss so that matching views agree and different images disagree. Its success rests on strong augmentation, a nonlinear projection head, and many negatives from large batches.

Its ideas travel well beyond images. For tables, corrupted rows play the role of augmented views, and the same contrastive objective can build row embeddings that cope with mixed types and missing cells. These embeddings support imputation, anomaly detection, clustering and few-label learning, and they complement in-context approaches. For related methods, read about MoCo, BYOL and DINO, and see /model/research, /model/architecture and /model/benchmarks for how tabular models are built and measured.

Related

It belongs to Self-supervised methods in my stack, under AI and machine learning. Others in the same group: