InfoNCE

Overview

InfoNCE is a loss function for contrastive learning. Given one anchor example, one positive example that truly belongs with it, and a set of negative examples that do not, InfoNCE trains a model so that the positive scores higher than every negative. It does this with a softmax over the scores, so the task becomes a multiple-choice question: which of these candidates is the true partner of the anchor? The loss is the cross-entropy of getting that question right.

The loss matters for two reasons. First, it works. It sits inside many of the best known methods for learning embeddings without labels, in vision, language, audio and tables. Second, it has a clean information-theoretic meaning: minimizing it raises a lower bound on the mutual information between the two views of the data. That makes it a principled tool, not just a trick. For tabular models, InfoNCE is a standard way to pre-train row and column embeddings from raw tables, without labels. See contrastive learning for the wider family and /model/research for how these ideas relate to tabular foundation models.

What it is

InfoNCE stands for information noise-contrastive estimation. The name joins two ideas: noise-contrastive estimation, a way of learning by telling data apart from noise, and the information view, in which the loss bounds mutual information.

The loss in one line

Let q be an anchor embedding, k+ the embedding of its positive partner, and k_1 to k_N the embeddings of N candidates, one of which is the positive. Let s(q, k) be a score, often a dot product or cosine similarity. With a temperature t, the loss for this anchor is:

L = - log [ exp(s(q, k+) / t) / sum over i of exp(s(q, k_i) / t) ]

The denominator sums over all candidates, including the positive. This is exactly the cross-entropy loss of a softmax classifier with N classes, where the correct class is the positive.

An everyday analogy

Picture a matching game at a party. You are shown a photo of one person (the anchor) and a lineup of ten photos, of which exactly one is a different photo of the same person. You must pick the right one. If you pick it with high confidence, you score well. To do well, you must learn what stays the same about a person across photos (face shape, build) and ignore what changes (lighting, background). InfoNCE is the scoring rule of this game, and the model is the player who gets better with practice.

Positives and negatives

• A positive pair is two views of the same underlying thing: two augmentations of one image, an image and its caption, two nearby frames, two corrupted versions of one table row.
• A negative is any other example, assumed to be unrelated to the anchor. Negatives usually come from the other items in the same batch, or from a memory queue.

The model learns an embedding space where positives are close and negatives are far apart.

Why a softmax

A softmax turns scores into probabilities that sum to one. This makes every negative compete with the positive at once, and harder negatives (those with scores close to the positive) get more weight in the gradient. That focus on hard negatives is a main reason the loss is effective.

Where it came from

The roots go back to noise-contrastive estimation, introduced around 2010 as a way to estimate unnormalized statistical models by training a classifier to tell data from noise. Related ideas appeared in language modelling, where negative sampling made word embeddings cheap to train (2013).

The InfoNCE name and its information-theoretic reading come from the 2018 paper "Representation Learning with Contrastive Predictive Coding". In that work, a model predicted future latent states in a sequence and used a softmax over one true future and several false ones. The paper showed that minimizing this loss maximizes a lower bound on the mutual information between context and future.

Soon after, the loss spread widely.

• Deep InfoMax and related methods (2018 and 2019) applied mutual information ideas to images.
• The N-pair loss (2016) was an earlier, closely related form for metric learning.
• MoCo (2019 and 2020) used InfoNCE with a large queue of negatives and a slowly moving key encoder.
• SimCLR (2020) used InfoNCE with large batches and strong augmentations, showing how far simple contrastive learning could go.
• CLIP (2021) used a symmetric InfoNCE loss between images and text, trained on a very large set of image-text pairs, and became a base for zero-shot classification.
• Supervised contrastive learning (2020) extended the loss to use class labels to define many positives.
• Later work examined what the loss really learns, including its alignment and uniformity properties (2020), and its relation to other mutual information bounds.
• For tables, methods such as SCARF (2021) and contrastive objectives inside models such as SAINT used the loss with corrupted copies of rows as positives.

Not every method in the family uses negatives. BYOL and DINO learn without explicit negatives, which is a useful contrast to keep in mind.

How it works

Step 1: build positive pairs

Take each example in a batch and make two views of it. In images, the views are two random crops, colour jitters or flips. In text, they may be a sentence and a dropout-perturbed copy of itself. In tables, they may be a row and a corrupted copy of the same row, or two different subsets of columns of the same row.

Step 2: encode

Pass both views through an encoder network. A small projection head often maps the output to the space where the loss is computed. Normalize each vector to unit length so that the dot product equals cosine similarity.

Step 3: build the similarity matrix

For a batch of B examples with two views each, compute all pairwise similarities between views. The result is a B by B matrix when comparing view A with view B. Diagonal entries are the positive pairs. Off-diagonal entries are the negatives.

Step 4: divide by temperature and apply softmax

Divide every similarity by the temperature t, then apply a softmax along each row. The row now holds a probability distribution over candidates.

Step 5: take the cross-entropy

For each row, the correct class is the diagonal entry. The loss is the negative log of its probability. Average over the batch. Often the loss is computed in both directions (A to B and B to A) and averaged, which is the symmetric form used in CLIP.

Step 6: update

Backpropagate and update the encoder. Gradients push positives together and push the hardest negatives away.

import torch
import torch.nn.functional as F

def info_nce(a, b, temperature=0.1):
    """a, b: (B, D) embeddings of two views. Row i of a pairs with row i of b."""
    a = F.normalize(a, dim=1)
    b = F.normalize(b, dim=1)
    logits = a @ b.t() / temperature          # (B, B) similarity matrix
    targets = torch.arange(a.size(0), device=a.device)
    loss_ab = F.cross_entropy(logits, targets)       # a -> b
    loss_ba = F.cross_entropy(logits.t(), targets)   # b -> a
    return 0.5 * (loss_ab + loss_ba)

The role of temperature

The temperature t controls how sharp the softmax is.

• A small t makes the distribution peaky. The loss concentrates on the hardest negatives and pushes strongly on them. Too small a value can make training unstable and can punish false negatives hard.
• A large t makes the distribution flat. All negatives matter about equally and the gradient is gentle. Too large a value gives weak separation.

Think of the temperature as the strictness of a judge. A strict judge (low t) penalizes near misses sharply. A lenient judge (high t) hardly cares which wrong answer was close. Many setups use a value around 0.05 to 0.5, and some learn it as a parameter.

The mutual information reading

The bound can be stated simply. If the candidate set has N items (one positive, N - 1 negatives), then:

I(x; y) >= log(N) - L

where I is the mutual information between the two views and L is the InfoNCE loss. Two consequences follow.

1. Lowering the loss raises a lower bound on mutual information, so the encoder is pushed to keep information shared between the two views.
2. The bound can never exceed log(N). With few negatives, the bound is loose and cannot measure large mutual information. This is one argument for using many negatives.

A caution: the practical success of the loss is not fully explained by the bound. Research has shown that the details of the encoder, the augmentations and the negatives matter as much as the bound itself. The mutual information view is a helpful guide and not the whole story.

Alignment and uniformity

A second reading says the loss does two jobs.

• Alignment: positive pairs end up close.
• Uniformity: all embeddings spread out over the unit sphere, so the space is used well and information is kept.

If only alignment were rewarded, all points would collapse to one place. The negatives in the denominator prevent that collapse by pushing points apart.

Where the negatives come from

The false negative problem

If two examples in a batch are actually about the same thing (two rows of the same class, two photos of the same dog), treating one as a negative pushes similar things apart. This hurts quality. Remedies include larger sets of diverse examples, using labels to remove same-class negatives, debiased versions of the loss, and softer targets.

Common variants of the loss

• Symmetric InfoNCE. Average the loss over both directions of the pair. Used when the two views play equal roles.
• Multi-positive forms. When several positives exist for one anchor, average the loss over them or place all positives in the numerator.
• Debiased forms. Correct the denominator for the chance that a sampled negative is really a positive.
• Hard negative reweighting. Raise the weight of negatives that are close to the anchor.
• Learned temperature. Treat the temperature as a trainable scalar, often stored as a log value so it stays positive.
• Local and global forms. Compare patches or cells with a global summary of the same example, as in some information maximization methods.

What the gradient does

For one anchor, the gradient on the positive embedding points toward the anchor, scaled by the chance the model gave to the wrong answers. The gradient on each negative points away from the anchor, scaled by the chance the model gave to that negative. So the update has three useful properties.

• When the model is already confident and right, the gradient is small and learning slows.
• When the model puts high probability on a wrong candidate, the push on that candidate is strong.
• Negatives with tiny scores have almost no effect, which is why easy negatives add little.

This is the mechanical reason that hard negatives matter and that a very large pool of easy negatives gives diminishing returns.

Batch size and the loss floor

The smallest possible loss depends on the number of candidates. With a perfect model and a very large margin, the loss approaches zero. With a modest margin, the loss floor rises as more candidates are added, because there are more chances for a wrong candidate to score high. Do not compare raw loss values between runs with different batch sizes or different temperatures. Compare downstream probe scores instead.

Contrastive learning for sets of columns

Tables allow a view that images and text do not: the same row seen through different subsets of columns. Two disjoint sets of columns from one row form a natural positive pair, because both describe the same entity. A model trained this way learns that the weight and height columns of a person say something about the age and income columns, which is precisely the kind of cross-column knowledge that prediction needs.

Key concepts and terms

A worked example

A numeric walk-through

Take one anchor and four candidates, where the first is the positive. Suppose the cosine similarities are:

Use a temperature of t = 0.5. Divide each similarity by t:

• Candidate 1: 0.80 / 0.5 = 1.6
• Candidate 2: 0.30 / 0.5 = 0.6
• Candidate 3: 0.10 / 0.5 = 0.2
• Candidate 4: 0.50 / 0.5 = 1.0

Exponentiate: exp(1.6) is about 4.953, exp(0.6) is about 1.822, exp(0.2) is about 1.221, exp(1.0) is about 2.718.

The sum is about 4.953 + 1.822 + 1.221 + 2.718 = 10.714.

The probability of the positive is 4.953 / 10.714, which is about 0.462. The loss is minus the natural log of 0.462, which is about 0.772.

Now compare with a smaller temperature, t = 0.1. The scaled scores become 8, 3, 1 and 5. Exponentials: about 2981, 20.1, 2.72 and 148.4. The sum is about 3152. The probability of the positive is about 0.946, and the loss is about 0.055. The same similarities give a much smaller loss, because a sharper softmax rewards a clear margin more.

Finally, consider a bad case where the positive has similarity 0.30 and the negative candidate 4 has 0.80. At t = 0.5 the scaled scores of the positive and candidate 4 swap roles. The positive probability becomes about 0.19 and the loss rises to about 1.7. The gradient is then large and pushes the encoder to fix this ordering.

The upper bound from the mutual information view with N = 4 candidates is log(4), about 1.386. Even a perfect model cannot certify more than 1.386 nats of information with these four candidates, which is the reason bigger candidate sets are used.

Pre-training a tabular encoder with InfoNCE

The following sketch trains a row encoder by treating two corrupted copies of the same row as a positive pair and other rows in the batch as negatives.

import torch
import torch.nn as nn
import torch.nn.functional as F

class RowEncoder(nn.Module):
    def __init__(self, d_in, d_hidden=128, d_out=64):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(d_in, d_hidden), nn.ReLU(),
            nn.Linear(d_hidden, d_out),
        )

    def forward(self, x):
        return self.net(x)

def corrupt_row(x, p=0.4):
    """Replace a random share of cells with values from other rows (swap noise)."""
    B, D = x.shape
    mask = torch.rand(B, D, device=x.device) < p
    donors = x[torch.randint(0, B, (B,), device=x.device)]
    return torch.where(mask, donors, x)

enc = RowEncoder(d_in=30)
opt = torch.optim.Adam(enc.parameters(), lr=1e-3)

for epoch in range(100):
    for x in loader:                       # x: (B, 30) scaled table rows
        a = enc(corrupt_row(x))
        b = enc(corrupt_row(x))
        loss = info_nce(a, b, temperature=0.2)   # from the sketch above
        opt.zero_grad()
        loss.backward()
        opt.step()

After training, the encoder maps a row to an embedding. Rows that are alike end up close. The embeddings can feed a classifier, a clustering step or a nearest neighbour search.

Checking the result

A fair check is a linear probe. Freeze the encoder, train a simple logistic regression on the embeddings with the labels, and compare with the same regression on raw features. Also look at alignment (average distance between positive pairs) and uniformity (how spread out the embeddings are). If alignment is good but uniformity is poor, the embeddings may be nearly collapsed.

Strengths and limitations

Strengths

• Needs no labels. Positive pairs come from augmentations or natural pairings.
• Principled. It has a clear meaning as a bound on mutual information and as a classification task.
• Strong in practice. It sits inside leading self-supervised and multimodal methods.
• Easy to implement. It is a cross-entropy over a similarity matrix.
• Scales with negatives. More negatives usually give better embeddings.
• Focuses on hard cases. The softmax weights close negatives more.
• Flexible. Any pair of views can be used, including different modalities.
• Works with any encoder. Convolutional nets, transformers and plain networks all fit.

Limitations

• Needs many negatives. Large batches or queues add memory and engineering cost.
• Sensitive to the temperature. A poor choice harms quality.
• Sensitive to augmentations. The positives define what the model treats as irrelevant. Bad augmentations teach bad invariances.
• False negatives. Related examples treated as unrelated hurt the result.
• Loose bound. The mutual information bound saturates at log N, and is not a precise measure.
• Collapse of dimensions. Without care, embeddings may use only a few dimensions.
• Task mismatch. An embedding good at separating instances may be less good for a specific downstream task.
• Hard for tables. Choosing label-preserving augmentations for tables is less obvious than for images. A change in a cell can change the meaning of the row.
• Cost. All pairwise similarities in a batch grow with the square of the batch size.

How it compares with alternatives

Contrastive and reconstructive methods are complements. Reconstruction keeps fine detail of the input. Contrastive learning keeps what is shared between views and discards the rest. Several tabular methods combine both. For the broader context, see self-supervised learning and information theory.

Role in tabular foundation models

A tabular foundation model works on raw tables: rows and columns, mixed types, missing cells and, often, no helpful column names. It is pre-trained on many tables, either synthetic ones from a generator of data-producing processes, real ones, or both, and it predicts through in-context learning by reading labelled rows. For examples of such models see TabPFN, TabICL, TabSTAR and XTab. See /model/architecture for how the parts fit.

InfoNCE enters this picture in several concrete ways.

Row embeddings from corrupted views

A row is a natural object for a contrastive pair. Two corrupted copies of the same row (masked cells, swapped cells, dropped columns) should map to nearby embeddings, and different rows should map to distant ones. The model must then learn which cells carry the identity of a row and which are incidental. This gives a label-free signal that works on any table, whatever its types or names. Models such as SAINT combine this idea with a reconstruction loss.

Column embeddings without names

If columns have no useful names, the model must learn what a column is from its values. A contrastive objective can treat two random subsets of values from the same column as a positive pair and subsets from different columns as negatives. The resulting column embedding captures the distribution and type of the column, such as whether it behaves like an age, an identifier or a price, without reading its name. These embeddings are useful for schema matching and for sharing knowledge between tables with different layouts.

Aligning columns with text

When column names or descriptions do exist, a symmetric InfoNCE loss can align column values with their text, in the way CLIP aligns images with captions. This helps models such as TabSTAR connect semantic information with numbers, and supports the semantic and sector classification of columns and tables. See semantic sector heads.

Mixed types

Numeric columns are scaled or binned. Categorical columns use embeddings. Text columns pass through a language encoder. InfoNCE does not care about the input type because it works on the output embeddings. This makes it a convenient common loss across types. The augmentations, however, must be type aware: noise suits numbers, category swaps suit categories, and word dropout suits text.

Missing values

Missingness can itself be the augmentation. Randomly hiding cells produces views of the same row with different visible parts. A contrastive model then learns embeddings that are stable under missing cells. That stability is helpful for imputation and for downstream prediction on incomplete data. It complements reconstruction objectives such as masked cell modeling and denoising autoencoders.

In-context learning and retrieval

Some tabular models retrieve similar rows before predicting, or attend to the most relevant labelled rows in the context. The quality of such retrieval depends on a good similarity measure between rows. A contrastively trained embedding provides one, since similar rows are close by construction. Nearest neighbour style behaviour inside attention can be seen as a soft version of the same idea.

Synthetic pre-training

When tables are drawn from a generator, the generator gives free positive pairs: two samples of rows from the same generated table share structure, and rows from different generated tables do not. That supplies labels for a table-level contrastive objective, for example to learn embeddings of whole datasets. Such dataset embeddings help with choosing a model, a sector or a prior to apply.

Anomaly detection

A contrastive encoder trained on normal rows gives an embedding space in which typical rows cluster. Distance from the cluster, or low similarity to the nearest normal neighbours, is an anomaly score. See anomaly detection, isolation forest and one-class SVM for related approaches. A caution: if the augmentations hide the very features that mark an anomaly, the encoder will ignore them.

Continual learning

When a tabular model is updated with new data, the contrastive objective can drift, because the set of negatives changes. Mixing in stored or regenerated old tables, through experience replay, keeps the embedding space consistent. Distillation of old embeddings, in the spirit of LwF, is another option.

For measured results of such models, see /model/benchmarks. For research context, see /model/research.

Practical guidance and best practices

Start with a clean pair definition

Write down exactly what makes two views a positive pair, and check that the pair really shares the property you care about. The loss only teaches what the pairs define.

Pick augmentations with care

For tables:

• Use masking and swap noise at moderate rates.
• Avoid changes that flip the meaning of a row, such as altering a key identifier.
• Mix several augmentations instead of one.
• Respect types. Do not add Gaussian noise to a category code.

Use enough negatives

Use large batches, or a queue. If memory is limited, gradient caching or a memory queue helps. Check performance as the number of negatives grows and stop when gains flatten.

Tune the temperature

Try a small grid, such as 0.05, 0.1, 0.2 and 0.5. Watch both loss and the downstream probe. Consider making the temperature a learned parameter.

Normalize embeddings

Use cosine similarity with unit-length vectors. This keeps scores bounded and makes the temperature meaningful.

Add a projection head

Compute the loss on the output of a small projection head and use the layer before it for downstream tasks. The head absorbs details that are useful for the loss but not for transfer.

Guard against false negatives

If labels or groupings are available, exclude same-group examples from the negatives. For tables with duplicate or near-duplicate rows, remove duplicates first. See deduplication.

Monitor collapse

Track the standard deviation of embeddings across the batch. If it drops towards zero, the model is collapsing. Also track the rank of the embedding matrix.

Evaluate downstream

Use a linear probe or a small head on held-out labelled data, and compare against raw features and against a reconstruction-only baseline. Use scikit-learn for fast probes.

Keep splits clean

Fit scalers, encoders and vocabularies on training data only, and make sure that views of the same row never straddle the train and test split.

Use standard tooling

PyTorch and JAX make it easy to write the loss and to scale it across devices. When using several devices, gather embeddings from all devices so that every device sees the full set of negatives.

Common mistakes

• Forgetting to normalize embeddings, so that scores grow without bound and the temperature stops meaning anything.
• Putting the positive outside the denominator, which changes the loss and breaks its bound.
• Using too few negatives and expecting strong embeddings.
• Using a temperature that is far too small, which makes training noisy.
• Treating duplicates or same-class rows as negatives.
• Using augmentations that destroy the signal, such as masking nearly every cell.
• Choosing augmentations that leave a shortcut, so the model matches views by an irrelevant cue, such as a scaling artefact or a row order.
• Computing the loss in only one direction when the data are symmetric, losing half the signal.
• Evaluating with leaked data, where views of one row appear in both train and test.
• Not gathering negatives across devices in distributed training.
• Reading the value of the loss as a quality measure across different batch sizes. The loss floor depends on the number of candidates.
• Assuming the mutual information bound is tight and quoting it as a measurement.
• Using the projection head output for downstream tasks when the layer below it transfers better.

A short debugging routine

1. Overfit a tiny batch first. The loss should fall close to zero. If not, the pairing or the code is wrong.
2. Print the similarity matrix for a small batch. The diagonal should grow larger than the rest.
3. Shuffle the pairing on purpose. The loss should stay near log of the number of candidates. If it does not, there is leakage between views.
4. Check embedding spread. The per-dimension standard deviation should not shrink to zero.
5. Run a linear probe every few epochs, not only at the end.

Frequently asked questions

What is InfoNCE?

InfoNCE is a loss function for contrastive learning. It takes an anchor, one true positive partner and several negatives, and trains a model so that the positive scores higher than all negatives under a softmax. In effect it is a multiple-choice classification loss in which the correct answer is the true partner.

What does InfoNCE stand for?

It stands for information noise-contrastive estimation. The name combines noise-contrastive estimation, a method that learns by distinguishing real data from noise, with the information view in which minimizing the loss maximizes a lower bound on mutual information between two views of the data.

How is InfoNCE related to mutual information?

Minimizing the loss raises a lower bound on the mutual information between the anchor view and the positive view. The bound is the logarithm of the number of candidates minus the loss, so it can never exceed that logarithm. This gives the loss a theoretical meaning, though practical success depends on other details too.

What does the temperature do?

The temperature divides the similarity scores before the softmax. A small value makes the distribution sharp and focuses the loss on the hardest negatives, while a large value makes it flat and treats all negatives more equally. The best value depends on the data and is usually found by trying a small range of values.

How many negatives do I need?

More negatives usually give better embeddings, because the task becomes harder and the bound looser at fewer candidates. In practice, people use large batches or memory queues to reach hundreds or thousands of negatives. The gain flattens at some point, so test a few sizes and stop when improvement stops.

What is a false negative and why does it matter?

A false negative is an example treated as unrelated to the anchor that is actually related, such as another row of the same class. The loss then pushes similar things apart, which hurts the embedding. Removing duplicates, using labels to exclude same-class examples and using debiased losses can reduce the problem.

What is the difference between InfoNCE and triplet loss?

Triplet loss compares an anchor with one positive and one negative and requires a margin between them. InfoNCE compares the anchor with one positive and many negatives at once through a softmax, so each update uses far more information. This usually gives stronger and more stable training.

Can InfoNCE be used on tabular data?

Yes. A common design treats two corrupted copies of the same row as a positive pair and other rows in the batch as negatives. The corruptions can be masking or swapping cell values. The augmentations must be chosen with care so they do not change the meaning of the row.

Does InfoNCE need negatives?

By definition yes, because the denominator sums over candidates. Some other methods, such as BYOL and DINO, avoid negatives and use other tricks to prevent collapse. InfoNCE uses its negatives both to define the task and to spread embeddings apart so they do not collapse.

How does InfoNCE relate to tabular foundation models?

It offers a label-free way to learn row, column and table embeddings from raw tables, with corrupted copies of rows or subsets of column values as positives. It can also align column values with text. Such embeddings support retrieval, imputation, anomaly detection and semantic classification, and they are complementary to reconstruction objectives.

Glossary

• InfoNCE: a contrastive loss that scores a true pair against negatives with a softmax and bounds mutual information.
• Contrastive learning: learning by pulling related examples together and pushing unrelated ones apart.
• Anchor: the reference example in a contrastive pair.
• Positive: a true partner of the anchor.
• Negative: an example treated as unrelated to the anchor.
• Temperature: a scalar controlling the sharpness of the softmax.
• Cosine similarity: the dot product of two unit-length vectors.
• Embedding: a vector that represents an example.
• Projection head: a small network between the encoder and the loss.
• Mutual information: a measure of the shared information between two variables.
• Noise-contrastive estimation: a method that learns by separating data from noise samples.
• Collapse: a failure in which all inputs receive the same embedding.
• Hard negative: a negative that looks similar to the anchor.
• False negative: a negative that is actually related to the anchor.
• Linear probe: a simple linear classifier trained on frozen embeddings to test their quality.
• Augmentation: a random transform that keeps the meaning of an example.

Summary

InfoNCE turns representation learning into a multiple-choice game: find the true partner of the anchor among many candidates. Its softmax cross-entropy form is simple, its link to mutual information gives it a principled reading, and its focus on hard negatives makes it effective. The key choices are the definition of positives, the source and number of negatives, the temperature, the augmentations and the normalization of embeddings.

For tabular foundation models, InfoNCE offers a label-free way to learn row, column and table embeddings from raw tables with mixed types, missing cells and no column names. It pairs well with reconstruction objectives such as denoising autoencoders and masked cell modeling, and supports retrieval, imputation, anomaly detection and semantic classification. Guard against false negatives and collapse, tune the temperature, and always check results with a downstream probe. For related reading, see contrastive learning, SimCLR, CLIP, /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Self-supervised for tables in my stack, under Foundation models and applied ML. Others in the same group: