ViT

Overview

ViT, short for vision transformer, is an image model that treats a picture as a sequence of small patches and processes them with the same self-attention layers used in language models. The image is cut into a grid of squares, each square is turned into a vector, position information is added, and a standard transformer encoder reads the whole sequence. A special token collects the result for classification.

ViT showed that convolutions are not required for strong image recognition, provided there is enough data or good pre-training. It became a backbone for classification, detection, segmentation and multimodal models such as CLIP. This page explains what ViT is, how it works, how it compares with CNN models, and how its core idea, turning structured inputs into sequences of tokens, relates to models that learn from raw tables.

What it is

A vision transformer is an encoder-only transformer applied to images. It has almost no image-specific machinery. Instead of convolutions that slide a small filter over the picture, it uses attention, where every patch can look at every other patch in each layer.

Patches as words

A sentence is a sequence of words. A ViT treats an image as a sequence of patches. A common setup cuts a 224 by 224 image into 16 by 16 patches, which gives 14 times 14, or 196, patches. Each patch is flattened and projected into a vector, much like a word becomes an embedding.

The jigsaw analogy

Imagine a jigsaw puzzle laid out on a table. A convolutional approach looks through a small window and slides it across the puzzle, building up understanding from local views. A vision transformer first lifts out every piece, writes a short description of each, then lets all the pieces talk to each other at once. A piece with a patch of blue sky can ask which other pieces also look like sky, and a piece with a wheel can ask which pieces look like a car body. The position label on each piece tells the group where it came from.

Main parts

• A patch embedding layer that cuts and projects patches.
• A learnable class token added at the front of the sequence.
• Position embeddings that tell the model where each patch sits.
• A stack of transformer encoder blocks with multi-head self-attention and feed-forward layers.
• A classification head on the class token or on pooled patch outputs.

Typical sizes

Patch size is written after the name, for example Base with 16 by 16 patches. Smaller patches mean more tokens, finer detail and higher cost.

Where it came from

The transformer architecture was introduced in 2017 in the paper "Attention Is All You Need", first for language translation. It was quickly adopted for language modelling, and models such as BERT and GPT grew from it. See Transformers for the general design.

For images, convolutional networks such as ResNet ruled for most of the 2010s. Researchers tried adding attention to convolutional models, or using attention in local windows. A fully attention-based image classifier was the missing step.

In 2020 the paper "An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale" introduced ViT. The key finding was that a plain transformer, trained on very large image collections, matched or beat strong convolutional models on standard benchmarks while using less compute to train. On smaller datasets, ViT was weaker than convolutional models, because it lacked built-in assumptions about images.

Soon after, DeiT showed that with strong augmentation and distillation, ViT can be trained well on a mid-sized dataset. The Swin transformer in 2021 brought shifted local windows and a hierarchy of scales, which suits dense prediction tasks. Self-supervised methods such as DINO and the masked autoencoder MAE showed that ViT can learn strong features without labels. Multimodal models like CLIP used ViT as the image encoder. Later work such as ConvNeXt modernised convolutional networks and narrowed the gap, which showed that training recipes mattered as much as architecture. For how such ideas link to table-focused work, see /model/research.

How it works

Step 1: cut the image into patches

The input image has height H, width W and C colour channels. It is divided into non-overlapping patches of size P by P. The number of patches is N = (H times W) divided by (P times P). For a 224 by 224 image with P equal to 16, N is 196.

Step 2: flatten and project

Each patch holds P times P times C numbers. For a 16 by 16 patch with three channels, that is 768 numbers. A learned linear layer maps them to the model's hidden size D. In code this is often written as a convolution with a kernel and stride equal to the patch size, which gives the same result.

Step 3: add the class token

A learned vector, called the class token, is placed at the start of the sequence. It does not come from the image. It acts as a place where the model gathers a summary of the whole picture. After the last layer, the output at this position is used for classification.

Step 4: add position embeddings

Attention by itself ignores order. If you shuffled the patches, the output for each patch would just be shuffled too. To inject location, a learned position vector is added to each patch embedding. The original ViT used one-dimensional learned embeddings and found that the model discovers two-dimensional layout by itself. Other designs use two-dimensional or relative encodings.

Step 5: run the transformer encoder

The sequence of N plus one tokens passes through L encoder blocks. Each block has two parts.

• Multi-head self-attention. Every token produces a query, a key and a value. A token compares its query with every key to get attention weights, then takes a weighted sum of the values. Multiple heads let the model look for several kinds of relationships at once.
• A feed-forward network. A two-layer network applied to each token separately.

Both parts use layer normalisation before them and a residual connection around them, so signals and gradients flow easily.

Step 6: read out

The output at the class token goes through a small head, often a single linear layer, to produce class scores. Some models average all patch outputs instead and skip the class token.

The core formula

Attention for one head can be written compactly.

Attention(Q, K, V) = softmax( Q K^T / sqrt(d) ) V

Here Q, K and V are matrices of queries, keys and values for all tokens, and d is the size of each head. The softmax turns the comparisons into weights that sum to one.

A tiny numeric walk-through

Take three patch tokens with one-dimensional keys of 1, 0 and 2, and a query of 1 for token A. The raw scores of A against each token are 1 times 1, 1 times 0, and 1 times 2, which gives 1, 0 and 2. After the softmax these become about 0.24, 0.09 and 0.67. Token A therefore takes roughly two thirds of its new content from token C, a quarter from itself, and a little from B. Different heads learn different scoring rules, so A may look at neighbours in one head and at distant look-alikes in another.

Cost of attention

Attention compares every token with every other token. The cost grows with the square of the number of tokens. With 196 tokens that is manageable. With a 1024 by 1024 image and 16 by 16 patches, there are 4096 tokens, and the cost is far higher. This is why high-resolution work often uses windows as in Swin, or larger patches, or efficient attention kernels.

Fine-tuning at a new resolution

A model pre-trained at one resolution is often fine-tuned at a higher one. The number of patches changes, so the learned position embeddings must be resized. This is done by interpolating the embeddings in two dimensions. It works well in practice and gives better accuracy for a modest extra cost.

Training recipes

ViT training depends heavily on the recipe.

• Large-scale pre-training on big labelled or weakly labelled image sets, followed by fine-tuning.
• Strong data augmentation such as random crops, mixing of images, and colour changes.
• Regularisation such as stochastic depth, label smoothing and weight decay.
• Long schedules with warm-up and cosine decay of the learning rate.
• Mixed precision to save memory, as in Mixed Precision.
• Self-supervised pre-training with MAE or DINO when labels are scarce.

Why layer normalisation and residuals matter

A stack of twelve or more attention blocks is hard to train without help. Two simple tools make it work.

• Residual connections add each block's input to its output. If a block has nothing useful to add, it can learn to output near zero, and the signal passes through untouched. Gradients also flow straight back through these shortcuts.
• Layer normalisation rescales each token's features to a steady range. Placing it before attention and before the feed-forward network, the so-called pre-norm layout, makes training more stable for deep stacks.

Think of a relay race where every runner is allowed to hand the baton straight to the next runner, skipping their own leg when it would only slow the team down.

What each layer tends to learn

Studies of trained ViTs suggest a rough pattern, though it varies by model and training recipe.

• Early layers have some heads that look at nearby patches and others that look far away. So the model can use global context early.
• Middle layers build up part-level features, such as textures and shapes.
• Late layers focus on object-level features that help the final decision.

This differs from a CNN, where the receptive field grows slowly from local to global with depth. In a ViT, the receptive field is global from the start, but the model still organises its features in stages.

Pooling choices

Efficiency ideas

Because attention cost rises with the token count, many methods reduce it.

• Windowed attention, as in Swin, limits each token to a local window and shifts windows between layers.
• Token merging combines similar tokens in deeper layers.
• Token pruning drops patches that seem unimportant.
• Efficient attention kernels compute exact attention with less memory traffic.
• Larger patch sizes at the first layer cut the token count directly.
• Low-rank or linear attention approximations trade exactness for speed.

None of these changes the main picture: patches in, tokens through attention, summary out.

Key concepts and terms

A worked example

This example builds a small vision transformer in PyTorch and trains it for image classification. Sizes are small so that it can run on a modest machine. Names are placeholders.

The model

import torch
import torch.nn as nn

class PatchEmbed(nn.Module):
    def __init__(self, img=32, patch=4, in_ch=3, dim=128):
        super().__init__()
        self.proj = nn.Conv2d(in_ch, dim, kernel_size=patch, stride=patch)
        self.n = (img // patch) ** 2

    def forward(self, x):
        x = self.proj(x)                  # (B, dim, H/p, W/p)
        return x.flatten(2).transpose(1, 2)   # (B, N, dim)

class TinyViT(nn.Module):
    def __init__(self, img=32, patch=4, dim=128, depth=6, heads=4, classes=10):
        super().__init__()
        self.embed = PatchEmbed(img, patch, 3, dim)
        self.cls = nn.Parameter(torch.zeros(1, 1, dim))
        self.pos = nn.Parameter(torch.zeros(1, self.embed.n + 1, dim))
        layer = nn.TransformerEncoderLayer(
            d_model=dim, nhead=heads, dim_feedforward=dim * 4,
            dropout=0.1, activation="gelu", batch_first=True, norm_first=True,
        )
        self.encoder = nn.TransformerEncoder(layer, num_layers=depth)
        self.norm = nn.LayerNorm(dim)
        self.head = nn.Linear(dim, classes)
        nn.init.trunc_normal_(self.pos, std=0.02)
        nn.init.trunc_normal_(self.cls, std=0.02)

    def forward(self, x):
        t = self.embed(x)
        cls = self.cls.expand(t.size(0), -1, -1)
        t = torch.cat([cls, t], dim=1) + self.pos
        t = self.encoder(t)
        return self.head(self.norm(t[:, 0]))

model = TinyViT()
print(model(torch.randn(2, 3, 32, 32)).shape)   # torch.Size([2, 10])

Counting tokens

For a 32 by 32 image and 4 by 4 patches, there are 8 times 8, or 64, patches. With the class token, the sequence has 65 tokens. Each token has 128 numbers. Attention compares 65 tokens against 65 tokens, which is cheap.

Training

opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.05)
sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=1e-3, total_steps=20000)
loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1)

for step, (xb, yb) in enumerate(loader):
    opt.zero_grad()
    loss = loss_fn(model(xb), yb)
    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step()
    sched.step()

What to expect

On a small image set trained from scratch, a tiny ViT usually does worse than a small convolutional network such as a ResNet, unless strong augmentation is used. If you start from a pre-trained ViT and fine-tune it, results are typically much better. This matches the central lesson of the original work: scale and pre-training matter.

Practical steps for fine-tuning

1. Choose a pre-trained ViT whose patch size and resolution fit your images.
2. Replace the classification head with one sized for your classes.
3. Freeze most layers for a few epochs, training only the head.
4. Unfreeze all layers with a small learning rate, often ten to a hundred times smaller than from scratch.
5. Use augmentation that matches your domain.
6. Evaluate on a held-out set that matches deployment conditions.

Looking at attention

You can visualise what the class token attends to. For a photo of a dog, the attention often highlights the head and body. This helps spot shortcut learning, such as a model that attends to the background rather than the object. Attention maps are a hint, not a full explanation, so pair them with attribution methods such as SHAP when you need stronger evidence.

Strengths and limitations

Strengths

• Global context from the first layer, since every patch can attend to every other patch.
• Strong accuracy at scale, with pre-training on large datasets.
• A simple, uniform design that reuses transformer tools and hardware optimisations.
• Scales well with data and model size.
• Works as a general backbone for classification, detection, segmentation and multimodal encoders.
• Learns well with self-supervision, which removes the need for labels.
• Easy to combine with text or other modalities, since all inputs become token sequences.

Limitations

• Data-hungry when trained from scratch, because it lacks the locality bias of convolutions.
• Attention cost grows with the square of the token count.
• Patchification can lose very fine detail within a patch.
• Needs careful training recipes and regularisation.
• Position embeddings must be adapted when resolution changes.
• Less natural for dense tasks without hierarchy, which designs like Swin address.
• Attention maps are not a faithful explanation.
• On small datasets, convolutional models often remain easier to train.

How it compares with alternatives

ViT or CNN

For small datasets, limited compute or edge devices, a convolutional model is often the first choice. For large datasets, pre-trained backbones, or systems that mix vision with text, ViT is often the better base. A good practice is to try a pre-trained model of each family on your own data, because published rankings may not match your domain.

Plain ViT or Swin

A plain ViT keeps one resolution through all layers, which is simple and works well for classification. Swin builds a pyramid of scales and limits attention to windows, which suits tasks that need feature maps at several resolutions, such as detection.

Role in tabular foundation models

Tabular foundation models learn from raw tables. Tables have rows and columns, columns of mixed types, missing cells, and sometimes no column names. See /model/architecture for how such a model is built. ViT is an image model, but its central idea carries over: turn a structured input into a set of tokens, add information about where each token belongs, and let attention relate them.

Cells as patches

In ViT, a patch is the basic unit, and attention relates patches. In a tabular model, the basic unit can be a cell, a column or a row. Each cell is encoded as a vector that depends on its type, for example a numeric value, a category or a missing marker. Attention then relates cells to each other. The jigsaw picture applies directly: a table is a grid, and each cell can ask which other cells matter for it.

Position information without names

ViT adds position embeddings so the model knows where each patch sits. A table has a different issue. Column order often carries no meaning, and column names may be absent. A tabular foundation model therefore avoids tying meaning to a fixed position. It can use column embeddings that are learned from the column's content, or attention that is invariant to order. Compare the table-oriented transformers FT-Transformer and SAINT, which tokenise each feature, and Axial Attention, which attends along rows and along columns separately, much as some vision models attend along image axes.

Class token and readout

ViT adds a class token to gather a summary. Tabular transformers use the same trick. A special token attends to all feature tokens and produces the prediction. In in-context tabular models such as TabPFN, the query rows act as the tokens that read from labelled context rows.

Pre-training

ViT gained its strength from large-scale pre-training and, later, self-supervised pre-training with masked patches. Tabular foundation models follow a similar path. They are pre-trained on a very large number of tables, often generated synthetically from random causal programs, and sometimes with masked cells to learn structure. See Masked Cell Modeling and MAE, which is the image version of the same masking idea.

Missing values and imputation

MAE hides a large share of patches and teaches the model to rebuild them. That is image imputation. A tabular model hides cells and learns to fill them. The result is an imputer that works across many tables. Related ideas appear in Imputation Networks.

In-context learning

A ViT is trained once and then applied with fixed weights. A tabular foundation model goes further and adapts to a new table through context: it reads labelled rows and predicts for new rows in a forward pass with no gradient steps. Large vision-language models show a similar behaviour with example images in the prompt.

Mixed types

Images have one type of value, pixel intensities. Tables have numbers, categories, dates and text. Where ViT uses one linear layer for patches, a tabular model uses separate encoders for each type, then projects all to a shared size so a transformer can process them together.

Continual learning

When a pre-trained ViT is fine-tuned on new data, it can forget earlier skills. Tabular models that keep adapting face the same risk. Techniques like EWC and Experience Replay apply to both.

Anomaly detection

Patch-level features from a ViT are used for visual anomaly detection, where each patch is compared with a bank of normal patches. A tabular analogue compares each cell or row with what the model expects given the rest of the table. See Anomaly Detection.

Sector and semantic classification

When a table must be classified by meaning, a model needs to read the contents and the structure of the table together. This resembles how a ViT classifies a picture by combining many patches into one decision. In a tabular setting, the tokens are cells or columns, and the decision is a sector or a type.

Practical guidance and best practices

Choosing a model

• Use a pre-trained ViT unless you have a very large dataset.
• Pick a patch size that matches the detail you need. Smaller patches help small objects but cost more.
• Use a hierarchical model such as Swin for detection and segmentation.
• For tiny datasets or edge devices, try a convolutional model first.

Data

• Use strong augmentation when training or fine-tuning.
• Match resolution between pre-training and fine-tuning, or interpolate position embeddings carefully.
• Keep a clean validation set that mirrors real use.
• Check class balance and apply weighting if needed.

Training

• Use AdamW with weight decay, warm-up and a decaying learning rate.
• Use layer-wise learning rate decay when fine-tuning: lower rates for early layers, higher for later ones.
• Use stochastic depth and label smoothing as regularisers.
• Use mixed precision and gradient checkpointing for large models.
• Track both accuracy and calibration.

Hyperparameters

Evaluation checklist

• Report top-one accuracy, and also per-class results if classes are imbalanced.
• Test under shifts such as new lighting, blur, or different cameras.
• Measure latency and memory on the target hardware, not only on a training machine.
• Check calibration, for example by comparing confidence with accuracy in bins.
• Review the worst errors by hand to find data issues.
• Compare against a simple baseline, such as a linear classifier on frozen features.

Feature extraction without fine-tuning

Sometimes full fine-tuning is not needed. A frozen pre-trained ViT can turn each image into a vector, which you can feed to a simple classifier such as logistic regression or a gradient boosting model. This is cheap and robust for small datasets. It is also a handy way to join images to a table: the image vector becomes a set of numeric columns that a tabular model can use next to the other fields.

Deployment

• Export to a portable runtime such as ONNX Runtime.
• Consider quantisation for latency.
• Cache or batch inputs, since attention cost grows with tokens.
• Monitor input drift, such as camera or lighting changes.

A short debugging checklist

If a ViT trains poorly, check these points in order.

• Confirm the patch size divides the image size, or that padding is handled on purpose.
• Confirm that position embeddings are added once and have the right length.
• Confirm that the class token is prepended and that the head reads it, or that pooling is applied as intended.
• Check the learning rate warm-up. Transformers often fail without it.
• Check weight decay and the optimiser settings. Adaptive optimisers with decoupled decay are the usual choice.
• Check augmentation strength. Too little leads to overfitting on small data, too much slows learning.
• When fine-tuning at a new resolution, interpolate the position embeddings rather than reusing them unchanged.
• Run more than one seed before drawing conclusions.

A note on evaluation

Compare a ViT and a convolutional network under the same data, augmentation, schedule and compute budget. Report accuracy and also throughput and memory use. A model that wins by a point but is far slower may be the wrong choice for your setting.

When a ViT is the right pick

A ViT fits best when you have a strong pre-trained checkpoint, a large dataset, or a multimodal system that needs a common token interface. For small datasets trained from scratch, a convolutional network is often the safer starting point.

Common mistakes

• Training a large ViT from scratch on a small dataset and blaming the architecture.
• Using weak augmentation, which leads to overfitting.
• Forgetting to resize position embeddings when changing resolution.
• Choosing a patch size too large for small objects.
• Reading attention maps as complete explanations.
• Comparing models with different training recipes and calling it an architecture comparison.
• Fine-tuning with a learning rate that is too high, which erases pre-trained features.
• Ignoring calibration when using the model for decisions.
• Skipping the class token or pooling choice without testing it.
• Using ViT on very long inputs without a plan for attention cost.
• Leaking test images into pre-training or fine-tuning data.

Frequently asked questions

What does ViT stand for?

ViT stands for vision transformer. It is an image model that splits an image into patches, turns each patch into a vector, and processes the sequence with a standard transformer encoder using self-attention.

How is ViT different from a CNN?

A CNN uses small filters that slide over the image and builds understanding from local patterns to global ones. A ViT uses attention, where every patch can interact with every other patch from the first layer. CNNs have stronger built-in assumptions about images, while ViT relies on data and pre-training to learn them.

Why does ViT need so much data?

It has weak built-in assumptions about locality and translation. A CNN gets these for free, but a ViT must learn them from examples. With large datasets or self-supervised pre-training, it learns them well. With small datasets, it can overfit unless regularisation and augmentation are strong.

What is the class token?

The class token is an extra learned vector placed at the start of the patch sequence. Through attention it gathers information from all patches, and its final output is used for classification. Some designs skip it and average the patch outputs instead.

Why does ViT use position embeddings?

Self-attention does not know the order of its inputs. Without position embeddings, shuffling the patches would give the same results as the original arrangement. Adding position vectors tells the model where each patch came from so spatial layout can be used.

What patch size should I use?

The common choice is 16 by 16 for classification at 224 pixels. Smaller patches, such as 8 or 14, capture finer detail but create more tokens and cost more. Larger patches are cheaper but may miss small objects. Pick based on object size and compute.

Can ViT work on high-resolution images?

Yes, but plain attention gets expensive as the number of tokens grows. Designs such as Swin use windows and hierarchy to keep cost manageable, and other methods use larger patches, efficient attention or token merging.

Is ViT better than ResNet?

Not always. With large-scale pre-training, ViT often reaches higher accuracy. With small datasets, limited compute or edge deployment, a ResNet or similar convolutional model can be easier to train and cheaper to run. Test both on your data.

Can ViT be used for tables?

Not directly, but its design inspired table transformers that turn each cell or column into a token. Models such as FT-Transformer apply attention across feature tokens, and tabular foundation models extend this to learning in context across many tables.

How is ViT used in multimodal models?

ViT often serves as the image encoder. It turns an image into tokens or a vector that is aligned with text, as in CLIP, or is passed to a language model. Because images and text are both token sequences, they can be processed with similar tools.

Glossary

• Attention head: one set of query, key and value projections that computes one attention pattern.
• Class token: an extra learned token that collects a global summary.
• Distillation: training a model to imitate a teacher model.
• Encoder block: one layer of attention and feed-forward network with residuals and normalisation.
• Fine-tuning: continuing training of a pre-trained model on a new task.
• Hierarchical transformer: a transformer that works at several resolutions, like a feature pyramid.
• Inductive bias: assumptions built into a model that guide learning.
• Layer normalisation: scaling activations per token to stabilise training.
• Masked autoencoder: a model that hides parts of the input and learns to rebuild them.
• Patch: a small square region of an image used as a token.
• Patch embedding: the vector produced from a patch.
• Position embedding: a vector added to a token to encode where it sits.
• Self-attention: a mechanism where tokens weigh each other to build new representations.
• Stochastic depth: randomly skipping layers during training as regularisation.
• Token: a unit of input to a transformer, such as a patch or a word piece.

Summary

ViT turns an image into a sequence of patch tokens, adds position information and a class token, and runs a standard transformer encoder over the sequence. Global attention gives it context from the first layer, and its uniform design scales well with data and compute. It needs large-scale or self-supervised pre-training to shine, and its attention cost grows with the number of tokens, which led to variants such as Swin.

For tabular work, ViT is a conceptual relative. The move from pixels to patch tokens parallels the move from raw cells to typed cell tokens, and masked pre-training, class tokens and attention readouts all carry over. Tabular foundation models add pre-training on many tables and learning in context, as described in /model/architecture, /model/research and /model/benchmarks. Use a pre-trained ViT for large-scale or multimodal vision tasks, compare it with a CNN on your own data, and keep the training recipe fixed when you compare models.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: