BYOL

Overview

BYOL, short for Bootstrap Your Own Latent, is a self-supervised method that learns image features without using negative examples. It runs two networks side by side. The online network is trained by gradient descent. The target network is a slowly updated moving average of the online network. Both see different augmented views of the same image, and the online network learns to predict what the target network outputs for the other view.

The surprise of BYOL is that this works at all. Without negatives, one might expect the model to cheat by outputting the same vector for every input. In practice, the extra predictor on the online side, the stop-gradient on the target side and the slow moving average keep the system away from that trivial answer. This article explains the mechanism with plain analogies, shows a numeric update and a code sketch, compares BYOL with SimCLR, MoCo and DINO, and finishes with how the same ideas help models that learn from raw tables.

What it is

BYOL is a member of the self-supervised learning family. It is sometimes grouped with contrastive methods because it uses two views of the same image, but it does not contrast anything with anything. There are no negative pairs. The loss only asks one prediction to match one target.

The two networks

The predictor exists only on the online side. This asymmetry between the two branches is a key ingredient.

What you keep

After training, you keep the online encoder. The projector and predictor are dropped. The encoder maps an image to a feature vector that can be used for classification, retrieval or transfer.

Where it came from

Early self-supervised vision methods used pretext tasks such as predicting rotations or solving jigsaw puzzles. Then contrastive methods became dominant. They learn by pulling views of the same image together and pushing views of other images apart, as in SimCLR and MoCo. These methods work well but need many negatives, and that often means large batches, queues or memory banks.

The BYOL paper, "Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning," appeared in 2020. It showed that a model could reach strong representations without negatives, by using a target network that is a slow average of the online network. The word "bootstrap" refers to the fact that the target is built from the model's own earlier state, not from labels or other images.

The result prompted a lot of follow-up analysis. Researchers asked why BYOL does not collapse. Several lines of explanation arose:

• The predictor and the stop-gradient create an update that is not the gradient of any single loss, which changes the dynamics.
• The moving average target moves slowly, which gives the predictor a stable goal.
• Batch normalization may play a role by implicitly comparing each sample to batch statistics. Later work showed that BYOL can work with other normalization schemes when the setup is chosen with care, so the role of batch normalization is not as simple as first thought.

Related negative-free methods followed, such as SimSiam, which drops even the moving average and relies on stop-gradient and the predictor, and DINO, which applies self-distillation to vision transformers.

How it works

Picture an apprentice and a master craftsperson. The apprentice looks at a rough sketch of a chair and tries to guess what the master would say about a refined photo of the same chair. The master changes their opinions only a little each day, based on what the apprentice has learned. Over time both improve, and neither needs to compare against a different chair to stay grounded.

Step 1: Two augmented views

An image is augmented twice to produce view v and view v'. The augmentations are the familiar ones: random crop and resize, flip, color jitter, gray conversion, blur and sometimes solarization. BYOL is known to depend less on color augmentation than contrastive methods, though augmentation still matters.

Step 2: Online branch

View v goes through the online encoder, then the online projector, then the online predictor. The result is a prediction vector p.

Step 3: Target branch

View v' goes through the target encoder and the target projector. The result is a target vector z'. No gradient flows through the target branch.

Step 4: Normalize and compare

Both p and z' are normalized to unit length. The loss is the mean squared error between them, which equals two minus twice their cosine similarity:

loss = || p/|p| - z'/|z'| ||^2  =  2 - 2 * cos(p, z')

Step 5: Symmetrize

The roles of the two views are swapped. View v' goes through the online branch, and v goes through the target branch. The two losses are added. This uses each pair of views twice.

Step 6: Update the online network

Only the online parameters get gradient updates. The stop-gradient on the target means the target is treated as a fixed label for that step.

Step 7: Update the target network

The target parameters follow the online parameters through a moving average:

theta_target  <-  tau * theta_target + (1 - tau) * theta_online

The coefficient tau starts around 0.99 and is gradually increased toward 1 during training, which makes the target slower and slower over time.

Step 8: Repeat

After many epochs the online encoder produces features that are consistent across augmentations and that carry useful structure.

Why does it not collapse?

If the online network ignored its input and always returned the same vector, the target would also learn to return that vector, and the loss would be zero. That is the collapse. BYOL avoids it in practice through a combination of factors:

• The predictor is an extra trainable module that must keep up with a moving target.
• The target is not updated by the loss, so there is no direct incentive for both sides to agree on a constant.
• The slow average makes the target change gradually, so the predictor can chase it.
• Normalization layers and the loss normalization add further implicit effects.

No single factor gives a formal guarantee. Collapse is avoided by the overall design and by tuning. If you remove the predictor or use a target equal to the online network, training can collapse.

Everyday analogies

• Online and target networks are like a student and a teacher who changes standards slowly.
• The predictor is like a note-taker who translates the student's view into the teacher's language.
• Stop-gradient is like treating the teacher's answer key as fixed while grading one test.
• No negatives means no need for a lineup of look-alikes. The student studies one thing at a time.
• Collapse is like both agreeing to say "I don't know" on every question. The setup makes that answer unstable.

BYOL and its relatives

The last two rows belong to a wider family of non-contrastive methods. They prevent collapse by adding explicit terms about the spread of embeddings, not by relying on a predictor.

A mental model of the training loop

• Pull a batch of images or rows.
• Make two corrupted views of each.
• Run view one through the online encoder, projector and predictor.
• Run view two through the target encoder and projector with no gradient.
• Compare the two normalized vectors with a squared error.
• Swap the views and add the second loss.
• Update the online network.
• Move the target a tiny step toward the online network.

Where the idea is used

Key concepts and terms

A worked example

A numeric walk-through of the loss

Suppose the normalized prediction p and normalized target z' are two-dimensional vectors:

• p = (0.6, 0.8)
• z' = (0.8, 0.6)

Their cosine similarity is 0.6 * 0.8 + 0.8 * 0.6 = 0.96. The loss is 2 - 2 * 0.96 = 0.08. If the predictor improves and p becomes (0.7, 0.714) after normalization, the cosine with z' rises to about 0.99 and the loss falls to about 0.02. If p pointed in the opposite direction, the cosine would be -1 and the loss would be 4, the largest possible value.

A numeric walk-through of the target update

Let tau = 0.996. An online weight is 0.50 and the target weight is 0.40.

• New target weight = 0.996 * 0.40 + 0.004 * 0.50 = 0.3984 + 0.0020 = 0.4004

The target moved by 0.0004. This tiny step is what makes the target stable.

Code sketch

import copy
import torch
import torch.nn as nn
import torch.nn.functional as F

def mlp(in_dim, hidden, out_dim):
    return nn.Sequential(
        nn.Linear(in_dim, hidden),
        nn.BatchNorm1d(hidden),
        nn.ReLU(),
        nn.Linear(hidden, out_dim),
    )

class BYOL(nn.Module):
    def __init__(self, encoder, feat_dim, proj_dim=256, hidden=1024, tau=0.996):
        super().__init__()
        self.tau = tau
        self.online_enc = encoder
        self.online_proj = mlp(feat_dim, hidden, proj_dim)
        self.predictor = mlp(proj_dim, hidden, proj_dim)
        self.target_enc = copy.deepcopy(encoder)
        self.target_proj = copy.deepcopy(self.online_proj)
        for p in list(self.target_enc.parameters()) + list(self.target_proj.parameters()):
            p.requires_grad = False

    @torch.no_grad()
    def update_target(self):
        pairs = [(self.online_enc, self.target_enc), (self.online_proj, self.target_proj)]
        for online, target in pairs:
            for po, pt in zip(online.parameters(), target.parameters()):
                pt.data = self.tau * pt.data + (1 - self.tau) * po.data

    def loss_one_side(self, x_online, x_target):
        p = self.predictor(self.online_proj(self.online_enc(x_online)))
        with torch.no_grad():
            z = self.target_proj(self.target_enc(x_target))
        p = F.normalize(p, dim=1)
        z = F.normalize(z, dim=1)
        return (2 - 2 * (p * z).sum(dim=1)).mean()

    def forward(self, v1, v2):
        loss = self.loss_one_side(v1, v2) + self.loss_one_side(v2, v1)
        return loss

A training step calls the model, back-propagates, steps the optimizer, and then calls update_target.

What to monitor

• The loss should fall and then settle at a small value.
• The standard deviation of normalized outputs across a batch should stay away from zero. Values near zero hint at collapse.
• A linear probe on frozen encoder features should improve with training.
• The cosine similarity between online and target outputs should rise but not hit exactly one for all pairs.

A tabular analog in code

The sketch below builds two views of a row by masking cells and then applies a BYOL-style step. The row encoder is a placeholder for any network that maps a numeric vector to an embedding.

import torch
import torch.nn as nn

def corrupt(x, rate=0.3):
    keep = (torch.rand_like(x) > rate).float()
    return x * keep

row_encoder = nn.Sequential(nn.Linear(40, 256), nn.ReLU(), nn.Linear(256, 128))
model = BYOL(row_encoder, feat_dim=128, proj_dim=64, hidden=256, tau=0.99)
opt = torch.optim.AdamW(
    list(model.online_enc.parameters())
    + list(model.online_proj.parameters())
    + list(model.predictor.parameters()),
    lr=1e-3,
)

for rows in loader:                     # rows: [N, 40] scaled numeric features
    loss = model(corrupt(rows), corrupt(rows))
    opt.zero_grad()
    loss.backward()
    opt.step()
    model.update_target()

Categorical columns would first be turned into embeddings or encoded as numbers. The two calls to corrupt give two independent masks over the same rows.

Strengths and limitations

Strengths

• No negative pairs, so no worry about false negatives or about how to sample negatives.
• Works with smaller batch sizes than in-batch contrastive methods.
• Strong features on standard linear evaluation and transfer tests in the original reports.
• Less sensitive to the choice of augmentation than contrastive methods, according to the original paper.
• Conceptually simple loss: a single regression between two normalized vectors.
• The target-and-predictor design inspired many later methods.

Limitations

• No formal guarantee against collapse. Stability comes from design and tuning.
• Needs the extra predictor and the moving-average target, which add memory and code.
• Sensitive to hyperparameters such as the target decay rate, learning rate and optimizer.
• Its behavior depends on normalization details, which can surprise people who change them.
• Long training is usually needed.
• Without negatives, there is no explicit push to keep different images apart, so quality depends on the implicit dynamics.

How it compares with alternatives

If you cannot afford large batches and do not want to maintain a queue, BYOL is a good option. If you work with transformers and want label-free features with clear emergent structure, look at DINO.

Role in tabular foundation models

Tabular foundation models learn from raw tables: rows as examples, columns as features, mixed numeric and categorical types, missing cells, and often no meaningful column names. See /model/research for the wider background. BYOL's setting, with no negatives, is attractive for tables for a specific reason.

Why avoiding negatives helps with tables

In a table, many rows can legitimately be near-duplicates of each other, or belong to the same group, such as one customer with many transactions or one machine with many sensor readings. A contrastive method would treat those rows as negatives when they come from different rows. That pushes similar rows apart and hurts the embedding. This is the false negative problem in a tabular form. BYOL does not need negatives, so it avoids this trap.

Views of a row

The augmentation step is replaced by row corruption. Ways to make a view of a row include:

• Masking a random set of cells.
• Replacing some cells with values from the same column in another row.
• Adding small noise to numeric cells.
• Dropping a random subset of columns.

Two corrupted views of the same row go into the online and target branches. The online branch learns to predict the target's embedding of the other view. The model has to learn which cell patterns are stable and which are noise. See masked cell modeling for a related reconstruction approach.

Mixed types and missing values

Because the target is an embedding, not a raw reconstruction, BYOL does not need to model the exact value of each numeric or categorical cell. That avoids having to balance a loss across numbers, categories and text. Missing cells are handled by masking, which is a natural tabular corruption. The learned encoder is then robust to the gaps found in operational data. This also helps imputation networks that build on row embeddings.

Tables without headers

The model learns from value patterns and co-occurrence across columns. If headers are missing or cryptic, the encoder can still produce a useful row embedding. This supports sector or semantic classification of tables, where the goal is to guess what kind of data a table holds.

The momentum target in continual learning

The slowly moving target gives a stable reference when new rows arrive. When a tabular encoder is updated on fresh data, the target changes more slowly than the online model, which can reduce abrupt drift in the embedding space. This is a helpful property when you want embeddings from last month to remain comparable to those from today. It does not remove forgetting by itself, so combine it with replay of older rows.

Anomaly detection and clustering

Row embeddings from a BYOL-style encoder can be used for anomaly detection, by flagging rows far from dense regions, and for clustering, by grouping nearby rows. Because the training does not pull different rows apart, clusters tend to preserve natural groups.

Relation to in-context learning

Models such as TabPFN learn to solve tasks by reading labeled rows as context. A row encoder trained with a BYOL-style objective could provide compact inputs for such a model, which may help with wide tables. That pairing is a design idea and not a claim about a specific released system.

Caveats

Self-supervised pre-training for tables is a research area. Gains over strong tree-based baselines are not guaranteed, and results depend on corruption design. Use the guidance in /model/benchmarks for fair evaluation, and see /model/architecture for how model components fit together.

Practical guidance and best practices

Target decay rate

• Start near 0.99 and increase toward 1 over the run with a cosine schedule.
• A rate that is too low makes the target noisy and may cause instability.
• A rate that is too high early on slows learning.

The predictor

• Keep the predictor an MLP with a hidden layer, similar to the projector.
• Do not remove it. Without it, collapse is likely.
• Some setups use a higher learning rate for the predictor.

Normalization

• The original design uses batch normalization in the projector and predictor.
• If you change normalization, test for collapse and compare with a linear probe.
• Group normalization with weight standardization has been shown to work in some setups.

Optimizer and schedule

• Use warmup then cosine decay.
• Use a large-batch optimizer when the batch is in the thousands.
• Use weight decay with care, and tune it.

Augmentations

• Start with the standard crop plus color pipeline.
• For tables, tune the corruption rate and keep rows plausible.

Monitoring for collapse

• Track the standard deviation of normalized embeddings across the batch.
• Track the effective rank of the embedding matrix.
• Run a periodic linear probe, even on a small labeled subset.

Evaluation

• Use a linear probe as the main yardstick.
• Add a few-label fine-tuning test.
• Test transfer to at least one different dataset.
• Report spread over seeds.

Keep the setup reproducible

• Record tau, learning rate, batch size and augmentation list.
• Save both online and target encoders.
• Keep the random seeds.

Debugging guide

Questions to ask before you adopt it

• Are near-duplicate or grouped items common in my data, making negatives risky?
• Can I support two networks and a moving-average update?
• Do I have a way to monitor collapse during training?
• Would a simpler contrastive baseline be enough?
• How will I test that the embeddings help my real task?

Reading the training curves

• A smoothly falling loss with a rising probe score is the healthy pattern.
• A loss that drops to near zero in the first few steps is a warning sign.
• A probe score that rises and then falls late in training may point to a target rate that is too low.
• Compare several seeds to see what normal variation looks like.

Reporting and reproducibility

• Record tau schedule, learning rate, batch size and augmentation list.
• Share the evaluation recipe, including how features were extracted.
• Keep notes on failed runs and what you changed.
• Save checkpoints from both networks.

Common mistakes

• Removing the predictor and expecting the model to still work.
• Letting gradients flow into the target network.
• Using a target decay rate that is too low.
• Forgetting to normalize the vectors before the loss.
• Changing normalization layers without checking for collapse.
• Only reading the loss, which can look fine even when features are poor.
• Applying heavy image augmentations to tabular data without checking that rows remain valid.
• Assuming that no negatives means no failure modes. Collapse and partial collapse (dimensional collapse) are still possible.
• Using the projector or predictor output as the feature instead of the encoder output.
• Not testing on a second dataset.

Frequently asked questions

What does BYOL stand for?

BYOL stands for Bootstrap Your Own Latent. The name reflects that the model builds its training targets from its own earlier state, using a slowly updated copy of itself, rather than from labels or from other images.

Does BYOL really use no negative examples?

Yes. Unlike SimCLR or MoCo, BYOL does not compare a view with views of other images. It only asks the online network to predict the target network's output for another view of the same image.

Why doesn't BYOL collapse to a constant output?

There is no single proven reason. The predictor on the online side, the stop-gradient on the target and the slow moving-average update together make the constant solution unstable in practice. Normalization layers also have an effect. Because the guarantee is empirical, monitoring for collapse is still a good habit.

What is the target network in BYOL?

It is a copy of the encoder and projector whose weights are an exponential moving average of the online network's weights. It receives no gradient updates. Its output serves as a fixed regression target for each training step.

What is the role of the predictor?

The predictor is a small network on the online branch that maps the online projection to a guess of the target projection. It makes the two branches asymmetric. Removing it generally leads to collapse, so it is a core part of the design.

How is BYOL different from DINO?

Both use a student that learns from a slowly updated teacher or target. BYOL regresses a prediction onto a normalized target with a mean squared error. DINO instead matches probability distributions, using a centering and sharpening scheme, and is usually applied to vision transformers.

Can BYOL work on tabular data?

Yes, in principle. Replace image augmentations with row corruptions such as masking cells or swapping values within a column, use a row encoder, and apply the same online and target setup. Avoiding negatives is helpful since near-duplicate rows are common in tables.

Does BYOL need large batches?

It is less dependent on batch size than in-batch contrastive methods, because it does not need many negatives. Batch size still affects optimization and batch normalization statistics, so very tiny batches can be a problem.

How do I evaluate a BYOL model?

Freeze the encoder and train a linear classifier on labeled data, then measure accuracy. Add a few-label fine-tuning test and a transfer test. Do not rely on the loss, since it can be small even when features are poor.

Is BYOL better than contrastive methods?

It depends on the setting. BYOL avoids negatives and can work with smaller batches, but it has more subtle dynamics and needs careful tuning. Contrastive methods are easier to reason about. The best choice depends on compute, data and the downstream task.

Glossary

• Augmentation: a random transformation that creates a new view.
• Bootstrap: using the model's own earlier state as the target.
• Collapse: all inputs mapped to the same output.
• Cosine similarity: the cosine of the angle between two vectors.
• Dimensional collapse: embeddings that use only a few of the available dimensions.
• Effective rank: a measure of how many dimensions an embedding set really uses.
• Encoder: the network whose output is kept.
• Exponential moving average: a running average that favors recent values.
• Online network: the branch trained by gradient descent.
• Predictor: the extra MLP that maps online output to the target space.
• Projector: the MLP that maps encoder output to the loss space.
• Stop-gradient: blocking gradient flow into a branch.
• Target network: the slowly updated branch used as a regression target.
• View: one augmented version of an input.

Summary

BYOL shows that self-supervised representation learning does not require negative pairs. An online network, extended by a predictor, learns to match the output of a slowly moving target network on another view of the same input. The loss is a simple regression between normalized vectors. Stability comes from the predictor, the stop-gradient and the moving-average target, along with careful tuning and normalization choices.

For tables, the no-negatives property is attractive because near-duplicate and grouped rows are common, and treating them as negatives would hurt. With corrupted rows as views, a BYOL-style objective can train a row encoder that handles mixed types and missing cells, and that supports imputation, anomaly detection, clustering and semantic classification. For comparison, read about SimCLR, MoCo and DINO, and see /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Self-supervised methods in my stack, under AI and machine learning. Others in the same group: