DINO

Overview

DINO is a self-supervised method in which a student network learns to match the output of a teacher network, and no labels are involved. The name comes from "self-distillation with no labels." Both networks have the same architecture. The teacher is not trained by gradient descent. Instead it is a moving average of the student. The student sees many crops of an image, including small local crops, while the teacher sees only large global crops. The student must produce the same output distribution that the teacher gives for the global view.

DINO became well known because it works especially well with vision transformers. Features learned this way support nearest-neighbor classification with no further training, and the self-attention maps of the model often outline the main objects in an image, even though no one ever told the model what an object is. This article explains the mechanism with simple analogies, walks through the loss with numbers, compares DINO with SimCLR, MoCo and BYOL, and explains how the same ideas carry over to models that learn from raw tables.

What it is

DINO belongs to the family of self-supervised learning methods, and to the subgroup that avoids negative pairs. It is a form of knowledge distillation where the teacher and student are the same size and the teacher is built from the student itself. That is why it is called self-distillation.

The pieces

What you keep

After training, you keep the teacher or the student backbone. In the original reports, the teacher backbone gave slightly better features. The projection head is dropped.

Where it came from

Knowledge distillation, in its classic form, trains a small student to copy a large teacher. The teacher is trained first, usually with labels. Self-supervised learning removed the labels, and methods such as BYOL showed that the teacher itself could be a slow average of the student, with no negatives needed.

The DINO paper, "Emerging Properties in Self-Supervised Vision Transformers," appeared in 2021. It made three main points:

• Self-supervised training of vision transformers produces features that carry explicit information about the layout of a scene, visible in the self-attention maps.
• These features are very good for k-nearest-neighbor classification, which needs no trained classifier.
• A simple setup, with a momentum teacher, centering, sharpening and multi-crop, avoids collapse without negatives or a predictor.

Later work extended the line, including a second generation that trained larger models on carefully curated image collections and produced general-purpose features for many tasks. Related ideas appeared in methods that distill, mask and predict in feature space. The common thread is that a slowly moving teacher gives a stable target.

How it works

Think of a master painter and a student. The master looks at a whole landscape and gives a rough but confident description: "mostly a mountain, a bit of sky." The student is shown only small patches, such as a corner with a rock or a cloud, and must still say, "mostly a mountain, a bit of sky." To do that, the student must learn how local parts relate to the whole scene. The master, meanwhile, slowly absorbs what the student has learned and refines the description.

Step 1: Multi-crop augmentation

Each image is cropped several times:

• Two global crops that cover a large part of the image (for example, over 40 percent of its area).
• Several local crops that cover a small part (for example, under 40 percent), usually at a lower resolution.

Each crop also gets flips, color jitter, blur and sometimes solarization.

Step 2: Teacher sees global crops

The teacher processes only the two global crops. This gives two teacher outputs.

Step 3: Student sees all crops

The student processes every crop, global and local. This gives many student outputs.

Step 4: Turn outputs into distributions

Each network ends in a projection head that outputs K numbers. A softmax with a temperature converts them into a probability distribution over K "prototype" dimensions. These are not class labels. They are just K learned directions that the model uses to describe the input.

• The teacher uses a low temperature, which makes its distribution sharper.
• The student uses a higher temperature.

Step 5: Center the teacher

Before the softmax, the teacher output has a running mean subtracted. This is called centering. The center is an exponential moving average of the teacher outputs over recent batches.

Step 6: Compute the loss

For each pair of a teacher global crop and a different student crop, the loss is the cross-entropy between the teacher distribution and the student distribution:

loss = - sum over k of  P_teacher(k) * log P_student(k)

The loss is summed over all valid pairs and averaged. Pairs where the teacher and student see the same crop are skipped. So the student must predict what the teacher says about a global view from a local view or from the other global view. This is the "local-to-global" matching that gives rise to the structure inside the model.

Step 7: Update the student

Only the student receives gradients. The teacher output is treated as a constant.

Step 8: Update the teacher

The teacher is updated by an exponential moving average of the student's weights:

theta_teacher  <-  lambda * theta_teacher  +  (1 - lambda) * theta_student

The coefficient lambda starts near 0.996 and is raised toward 1 during training with a cosine schedule.

Why centering and sharpening?

Without any negatives or predictor, there are two ways training can fail:

• One dimension dominates. The teacher always puts nearly all its probability on the same prototype, whatever the input.
• Uniform collapse. The teacher spreads probability equally over all prototypes for every input.

The two tricks pull in opposite directions:

• Centering subtracts the running mean, which discourages any single dimension from dominating. On its own, centering pushes toward the uniform distribution.
• Sharpening uses a low temperature, which pushes toward a peaked distribution. On its own, sharpening encourages a dominating dimension.

Used together, the two effects balance. This balance is what keeps the training from collapsing, in practice.

Why transformers?

A vision transformer cuts an image into patches and lets every patch attend to every other. A special class token summarizes the image. In DINO the class token output goes to the projection head. Because the student must match the teacher's whole-image description from small crops, the attention layers learn which patches belong together. This is why attention maps from the class token tend to highlight the object in the foreground.

Everyday analogies

• Student and teacher are like a junior and a senior doctor. The junior sees one test result, the senior sees the whole chart, and the junior learns to reach the same diagnosis from less.
• Multi-crop is like recognizing a friend from a photo of just their hand, then from a full-length photo.
• Centering is like grading on a curve so that no one answer is always the most popular.
• Sharpening is like asking the teacher to be confident rather than hedging.
• The moving average is a teacher who is cautious about changing their mind.

Typical hyperparameters

These values are common starting points and should be checked on a small run.

A mental model of the training loop

• Take an image and cut two big crops and several small ones.
• Send the big crops to the teacher.
• Send every crop to the student.
• Center and sharpen the teacher outputs.
• Compare each student output with each teacher output from a different crop.
• Update the student by gradient descent.
• Move the teacher a tiny step toward the student.
• Update the running center.

Where the idea is used

Key concepts and terms

A worked example

A numeric walk-through of the loss

Use K = 4 prototype dimensions to keep the numbers small. The teacher raw outputs for a global crop are (2.0, 1.0, 0.0, -1.0). The running center is (0.5, 0.5, 0.5, 0.5). The teacher temperature is 0.5.

Step 1: center. Subtract the center to get (1.5, 0.5, -0.5, -1.5).

Step 2: sharpen. Divide by 0.5 to get (3.0, 1.0, -1.0, -3.0).

Step 3: softmax. The exponentials are about 20.1, 2.72, 0.37 and 0.05. The sum is about 23.2. The teacher distribution is about (0.866, 0.117, 0.016, 0.002).

Suppose the student raw outputs for a local crop are (1.0, 1.0, 0.0, 0.0), with student temperature 1.0. The exponentials are 2.72, 2.72, 1.0 and 1.0, with sum 7.44. The student distribution is about (0.366, 0.366, 0.134, 0.134).

Cross-entropy:

• 0.866 * -log(0.366) = 0.866 * 1.005 = 0.870
• 0.117 * -log(0.366) = 0.117 * 1.005 = 0.118
• 0.016 * -log(0.134) = 0.016 * 2.01 = 0.032
• 0.002 * -log(0.134) = 0.002 * 2.01 = 0.004
• Total loss is about 1.02

The student is far from the teacher, since it spreads probability over two prototypes while the teacher is confident in the first. Gradient steps push the student's first output up. The loss shrinks as the student learns to see "mostly prototype 1" from the local crop.

Code sketch

import torch
import torch.nn as nn
import torch.nn.functional as F

class DINOLoss(nn.Module):
    def __init__(self, out_dim, t_student=0.1, t_teacher=0.04, center_m=0.9):
        super().__init__()
        self.ts, self.tt, self.cm = t_student, t_teacher, center_m
        self.register_buffer("center", torch.zeros(1, out_dim))

    def forward(self, student_out, teacher_out, n_global=2):
        # student_out: list of [N, K] tensors, one per crop (global crops first)
        # teacher_out: list of [N, K] tensors, one per global crop
        total, count = 0.0, 0
        t_probs = [F.softmax((t - self.center) / self.tt, dim=-1).detach()
                   for t in teacher_out]
        for i, tp in enumerate(t_probs):
            for j, s in enumerate(student_out):
                if i == j:
                    continue                       # skip same-crop pairs
                logp = F.log_softmax(s / self.ts, dim=-1)
                total = total + (-tp * logp).sum(dim=-1).mean()
                count += 1
        self._update_center(torch.cat(teacher_out))
        return total / count

    @torch.no_grad()
    def _update_center(self, teacher_out):
        batch_center = teacher_out.mean(dim=0, keepdim=True)
        self.center = self.center * self.cm + batch_center * (1 - self.cm)

@torch.no_grad()
def update_teacher(student, teacher, lam):
    for ps, pt in zip(student.parameters(), teacher.parameters()):
        pt.data = lam * pt.data + (1 - lam) * ps.data

The temperatures shown are typical orders of magnitude, with the teacher lower than the student. Treat them as starting points to tune.

A k-NN evaluation sketch

# features: [N, D] normalized embeddings, labels: [N]
sims = test_feats @ train_feats.t()               # cosine similarity
topk = sims.topk(k=20, dim=1)
votes = torch.zeros(test_feats.size(0), num_classes)
weights = (topk.values / 0.07).exp()
votes.scatter_add_(1, train_labels[topk.indices], weights)
pred = votes.argmax(dim=1)

No classifier is trained. The embeddings alone carry enough structure for a nearest-neighbor vote to work well.

A tabular analog in code

The sketch below shows a row version of the idea. The teacher sees a lightly masked row. The student sees a heavily masked one. It reuses the DINOLoss class from above.

import copy
import torch
import torch.nn as nn

def mask(x, rate):
    return x * (torch.rand_like(x) > rate).float()

class RowDINO(nn.Module):
    def __init__(self, in_dim=40, hidden=256, out_dim=512):
        super().__init__()
        self.student = nn.Sequential(
            nn.Linear(in_dim, hidden), nn.GELU(), nn.Linear(hidden, out_dim))
        self.teacher = copy.deepcopy(self.student)
        for p in self.teacher.parameters():
            p.requires_grad = False

model = RowDINO()
criterion = DINOLoss(out_dim=512)
opt = torch.optim.AdamW(model.student.parameters(), lr=1e-3)

for rows in loader:                         # rows: [N, 40] scaled numeric features
    t_views = [model.teacher(mask(rows, 0.1)) for _ in range(2)]
    s_views = [model.student(mask(rows, 0.1)) for _ in range(2)] + \
              [model.student(mask(rows, 0.6)) for _ in range(4)]
    loss = criterion(s_views, t_views)
    opt.zero_grad(); loss.backward(); opt.step()
    update_teacher(model.student, model.teacher, lam=0.99)

Here the first two student views match the teacher views in how they are made, and the extra four student views are heavy-mask "local" versions. Same-index pairs are skipped by the loss, as in the image version.

Strengths and limitations

Strengths

• No labels and no negatives needed.
• Strong features that work with plain k-NN classification.
• Self-attention maps that often show object boundaries.
• Works with both convolutional networks and transformers, with an especially good fit for vision transformers.
• Multi-crop teaches local-to-global reasoning efficiently.
• Features transfer to retrieval, copy detection and segmentation-style tasks.

Limitations

• Several delicate hyperparameters: teacher temperature, center momentum, teacher decay schedule and number of prototypes.
• Collapse is avoided by balance, not by guarantee.
• Multi-crop raises compute, since the student processes many crops per image.
• Long training and large data are typically needed for the best results.
• Augmentations designed for images do not transfer directly to other data types.
• Less interpretable loss than a pure contrastive objective.

How it compares with alternatives

A rule of thumb: DINO is a strong choice when you train a transformer and want features usable with a simple k-NN, and when you can afford multi-crop. BYOL and SimCLR are simpler to state. MAE learns different features that keep more local detail.

Role in tabular foundation models

Tabular foundation models learn from raw tables: each row is an example, columns hold numbers, categories, dates or text, cells may be missing, and column names may be absent or unhelpful. See /model/research for context. DINO was designed for images, but the self-distillation pattern maps onto tables with interesting parallels.

Local and global views of a row

In an image, a local crop is a small piece and a global crop is a large piece. In a table, an analogous pair is a row with many cells hidden (the local view) and a row with few cells hidden (the global view). The teacher sees the nearly complete row. The student sees a heavily masked version, and must produce the same distribution. To do so, the student has to learn how columns relate to each other, so that a few visible cells predict the character of the whole row. That is a direct match with the tabular goal of modeling dependencies across columns.

Mixed types and missing values

Masking cells covers missing data naturally. Because the target is a distribution over K prototypes, not the raw cell values, there is no need to design separate losses for numeric and categorical columns. The learned prototypes can capture row types such as "typical transaction" or "unusual reading" without any label. This links to masked cell modeling, where the target is the cell value itself, and to imputation networks.

Attention across columns

Transformers for tables, such as FT-Transformer and SAINT, treat each column as a token. The class token of such a model summarizes a row, in the same way as in a vision transformer. A DINO-style objective on that token can train the model without labels. The attention maps over columns can then be inspected to see which columns the model relies on, which is a useful interpretability aid for tables.

Tables without headers

Self-distillation relies on value patterns and co-occurrence, not on header text. A model trained this way can build row embeddings even if column names are missing, and its prototypes can be used for sector or semantic classification of tables, where the question is what kind of data a table holds.

Continual learning

The momentum teacher gives a stable reference. When a tabular encoder is updated on new data, the teacher moves slowly, so embeddings remain comparable over time. This helps with continual learning, where new rows arrive every day. Replaying a sample of older rows is still wise, since a moving average alone does not prevent forgetting.

Anomaly detection and clustering

The prototype assignments of rows act like soft cluster labels. Rows that match no prototype well, or whose assignment is unstable across corruptions, are candidates for anomaly detection. The embeddings feed clustering directly, and a k-NN check gives a quick sanity test of quality.

Relation to in-context learning

Models such as TabPFN are pre-trained on many synthetic tables to learn from labeled rows given in the prompt. A self-distilled row encoder is a different route. It could provide compact row summaries as inputs to an in-context learner, or serve as a pre-training stage for real tables. This is a design idea, not a claim about any specific released system.

Caveats

Self-supervised pre-training for tables is an active research topic. Results depend heavily on the corruption design, and gains over a well-tuned tree-based model are not guaranteed. Use the practices in /model/benchmarks for fair evaluation and see /model/architecture for how components fit together.

Practical guidance and best practices

Multi-crop settings

• Use two global crops and several local crops, as in the original recipe.
• Keep local crops small and low resolution to save compute.
• Ablate the number of local crops. Returns diminish after a handful.

Temperatures

• Keep the teacher temperature lower than the student temperature.
• Consider warming up the teacher temperature from a slightly higher value at the start.
• Too low a teacher temperature can cause instability.

Centering

• Use a center momentum near 0.9.
• Update the center from teacher outputs only.
• Check that the center is neither exploding nor stuck.

Teacher decay

• Start near 0.996 and raise toward 1 with a cosine schedule.
• If the teacher lags too much, features improve slowly.
• If it follows the student too closely, training may become unstable.

Number of prototypes

• A large K, such as tens of thousands, is typical for images.
• For smaller datasets or tables, a smaller K can be enough.
• Tune K by looking at how the prototypes are used. If only a few get all the mass, reduce K or adjust the temperatures.

Optimization

• Use warmup, then cosine decay for the learning rate and weight decay.
• Clip gradients to keep early training stable.
• Freeze the last layer of the head for the first epoch if you see early instability.

Evaluation

• Use a k-NN classifier as a fast, training-free check.
• Add a linear probe.
• Inspect attention maps from the class token.
• Test on a different dataset to check transfer.
• Report the spread over seeds.

Monitoring for collapse

• Track the entropy of the teacher output. Near zero entropy for all inputs suggests a dominating dimension. Entropy near the maximum for all inputs suggests uniform collapse.
• Track the spread of embeddings across a batch.
• Track the k-NN score during training.

Tabular tips

• Choose the corruption rates for global and local views separately, for example a low rate for the teacher view and a high rate for the student view.
• Keep rows plausible: swap values within the same column rather than inventing new ones.
• Deduplicate rows and handle grouped rows with care.

Debugging guide

Questions to ask before you adopt it

• Am I training a transformer or another encoder that can take a class token?
• Can I afford the extra compute for multiple crops?
• Do I know a sensible way to make local and global views of my data?
• Do I have a way to monitor collapse during training?
• How will I check that the features help my real task?

Common mistakes

• Letting gradients flow into the teacher.
• Forgetting to center, then seeing one prototype dominate.
• Using the same temperature for teacher and student.
• Setting the teacher decay rate too low.
• Including same-crop pairs in the loss.
• Using too few prototypes or too many for the data size.
• Judging success by the loss, which is not a quality score.
• Skipping the k-NN or linear probe check.
• Applying image-style multi-crop to tables without redefining what a local view means.
• Using only one seed.

Frequently asked questions

What does DINO stand for?

DINO stands for self-distillation with no labels. The name describes the setup: a student network is trained to match a teacher network, the teacher is built from the student itself, and no human labels are used at any point.

Is DINO a contrastive method?

No. Contrastive methods such as SimCLR and MoCo compare a view against negative examples from other images. DINO uses no negatives. It matches the student's output distribution to the teacher's, and avoids collapse with centering and sharpening.

How does DINO avoid collapse?

It combines two opposing effects. Centering subtracts a running mean from the teacher output, which stops a single dimension from dominating. Sharpening lowers the teacher temperature, which stops the output from becoming uniform. The momentum teacher adds further stability.

What is multi-crop in DINO?

Multi-crop means each image is cut into two large global crops and several small local crops. The teacher sees only the global crops, while the student sees all of them. The student must predict the teacher's view of the whole from small parts, which teaches local-to-global reasoning.

Why does DINO work so well with vision transformers?

Attention lets every patch relate to every other patch, and the class token gives a whole-image summary that can be matched across crops. This setup tends to produce attention maps that highlight objects and features that are strong enough for k-NN classification.

How is DINO different from BYOL?

Both use a momentum teacher or target and no negatives. BYOL regresses a normalized prediction onto a normalized target and relies on a predictor. DINO matches probability distributions over K dimensions and relies on centering and sharpening, with no predictor.

Can DINO be applied to tabular data?

Yes, in spirit. A row with few hidden cells can act as the global view and a row with many hidden cells as the local view. A student that matches the teacher from the heavily masked view learns how columns relate. Mixed types and missing values are handled by the corruption design.

What is k-NN evaluation?

It is a way to test features without training a classifier. Each test sample is compared with labeled training samples, and the nearest neighbors vote on the label. A good representation places similar items near each other, so this simple test reveals the quality of the features.

Do I need a lot of data to train DINO?

The best results come from large datasets and long training. With small datasets, results are more variable and the settings need more tuning. Starting from a model that is already pre-trained and fine-tuning it is often more practical.

What are the prototypes in DINO?

They are the K output dimensions of the projection head. After the softmax, each dimension acts like a soft cluster that the model uses to describe an input. They are not class labels. They emerge from training and are used only to compute the loss.

Glossary

• Attention map: a view of which tokens a given token attends to.
• Centering: subtracting a running mean from the teacher output.
• Class token: a learned token whose final state summarizes the input.
• Cross-entropy: a loss comparing two probability distributions.
• Distillation: training one model to copy another.
• Exponential moving average: a running average favoring recent values.
• Global crop: a large view of an image.
• k-NN: k-nearest neighbors, a classification method based on closest examples.
• Local crop: a small view of an image.
• Momentum teacher: a teacher updated as a moving average of the student.
• Multi-crop: using several crops of different sizes per example.
• Prototype: one of K output dimensions used as a soft cluster.
• Self-distillation: distillation where the teacher comes from the student.
• Sharpening: lowering the temperature to make a distribution more peaked.
• Student: the network updated by gradients.
• Temperature: a scale applied before the softmax.

Summary

DINO trains a student network to match a momentum teacher across crops of different sizes, with no labels and no negatives. Centering and sharpening keep the outputs from collapsing, and multi-crop forces the student to infer the whole from its parts. The result is a set of features that work well with simple k-NN classification and that, in vision transformers, produce attention maps that highlight objects.

For tables, the same pattern maps onto rows with different amounts of hidden cells. A student that sees a heavily masked row and matches a teacher that sees a nearly complete row must learn the dependencies between columns. That supports handling of mixed types and missing values, row embeddings without headers, anomaly detection, clustering and semantic classification, and it complements in-context learners. For comparison, read about SimCLR, MoCo and BYOL, and see /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Self-supervised methods in my stack, under AI and machine learning. Others in the same group: