Swin

Overview

Swin is a hierarchical vision transformer that computes self-attention inside small local windows and shifts those windows from one layer to the next. The shift lets information flow across window borders without the cost of global attention. The result is a model whose compute grows linearly with image size, and whose feature maps come in several resolutions, like a classic convolutional network.

This design makes Swin a strong backbone for dense vision tasks such as object detection and semantic segmentation. It also teaches a broader lesson that matters for tabular work: when full attention is too expensive, restrict it to a structured neighbourhood and then change the neighbourhood from layer to layer. The same idea appears in models that attend over rows and columns of a table. See /model/architecture for how that idea is used there.

What it is

Swin stands for "Shifted windows". It is a neural network architecture built from transformer blocks, but arranged so that it behaves like a pyramid. Early stages see many small patches at high resolution. Later stages see fewer, larger patches that each cover more of the image.

Three ingredients define it:

• Window attention. The feature map is cut into fixed-size, non-overlapping windows. Attention is computed only among the tokens inside each window.

• Shifted windows. In every second block, the window grid is moved by half a window. Tokens that sat on a border in one block now sit in the middle of a window in the next.

• Patch merging. Between stages, neighbouring tokens are merged, so the number of tokens falls and the channel width grows. This builds the feature hierarchy.

What a Swin model outputs

A Swin backbone returns feature maps at several scales, typically four. A classification head can pool the last map. A detection or segmentation head can use all of them. That multi-scale output is the main practical difference from the plain ViT, which keeps one resolution from start to finish.

Where Swin is used

• Image classification, as a drop-in backbone with a pooled head.

• Object detection and instance segmentation, as the feature extractor under a pyramid network.

• Semantic segmentation, as the encoder of a decoder network.

• Video understanding, through a three-dimensional version where windows span time as well as space.

• Medical imaging, where scans are large and local detail matters.

• Self-supervised pre-training, where masked patches are reconstructed.

Why a hierarchy matters

Objects in a picture appear at many scales. A small bird and a large truck need different receptive fields. A single-resolution model must learn both scales at once. A pyramid gives each scale its own feature map, so a head can pick the best level. This is the same reason convolutional pyramids became standard in detection.

Typical sizes

Swin comes in tiny, small, base and large variants. They differ in channel width, number of blocks per stage and number of attention heads. The structure stays the same. Only the depth and width change.

Where it came from

Transformers entered vision with the Vision Transformer, which cut an image into patches and fed them to a standard transformer encoder. That model worked well after large-scale pre-training, but it had two weaknesses for dense tasks. Its attention cost grows with the square of the number of tokens, and it has no built-in multi-scale feature maps.

Convolutional networks such as ResNet had solved both problems long before. They have local operations, cost that scales linearly with image size, and a natural feature pyramid. Researchers wanted a transformer that kept those strengths.

The Swin Transformer paper, "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows", appeared in 2021. It proposed local window attention with a shift between layers, and showed strong results on classification, detection and segmentation. A follow-up, Swin Transformer V2, appeared later and addressed training stability and the move to larger images and bigger models.

The architecture then spread widely. It became a common backbone in detection and segmentation toolkits, and its windowing idea influenced later designs such as ConvNeXt, which in turn borrowed some of the training recipe and design choices of transformers to modernise convolutions.

How it works

This section walks through one forward pass. Think of reading a very large map. You do not stare at the whole map at once. You look at one tile at a time, then you slide your view so tiles overlap with their neighbours, and later you zoom out to see the bigger picture.

Step 1: Patch partition

The input image is cut into small patches, for example 4 by 4 pixels. Each patch is flattened and projected by a linear layer to a vector of fixed width. After this step the image is a grid of tokens.

Everyday analogy: cutting a photograph into small square stamps and writing a short description on the back of each stamp.

Step 2: Window partition

The token grid is divided into windows of a fixed size, for example 7 by 7 tokens. Each window is treated as its own small sequence. Attention is then computed independently inside each window.

Because every window has the same number of tokens, the cost of attention per window is constant. The total cost is that constant times the number of windows. Doubling the image area doubles the number of windows, so cost grows linearly.

Step 3: Window self-attention

Inside a window, every token is turned into a query, a key and a value. Each token compares its query to all keys in the window, takes a weighted average of the values, and gets an updated vector. This is standard multi-head self-attention, only on a small set of tokens.

Analogy: a small committee meeting where everyone hears everyone else, but only members of the same committee are in the room.

Step 4: Relative position bias

Swin does not rely only on absolute position embeddings. Inside each window it adds a learned bias to the attention scores that depends on the relative offset between two tokens. A token one step to the right gets one learned bias per head. A token two steps below gets another.

This gives the model a sense of "near" and "far" that carries across windows and transfers well when the window size changes.

Step 5: Shifted windows

The next block uses the same window size, but the grid is displaced by half a window in both directions. Tokens that were in different windows in the previous block can now attend to each other.

Analogy: after the first round of committee meetings, you reshuffle the committees so that people from the edge of one group sit together with people from the edge of another. Information starts to travel across the whole room over several rounds.

Step 6: Efficient shifting with a cyclic roll

A naive shift would create windows of different sizes at the border, and also more windows. Swin avoids this with a cyclic shift: the feature map is rolled, so the pieces from one side wrap around to the other. Windows stay the same size.

Because wrapped-around tokens were not really neighbours, an attention mask blocks attention between tokens that come from different original regions. After attention, the map is rolled back.

Step 7: Feed-forward layer and residual links

As in any transformer block, attention is followed by a small feed-forward network applied to every token. Layer normalisation comes before each sub-layer, and residual connections add the input back to the output. One Swin stage is a stack of such blocks, alternating between regular and shifted windows.

Step 8: Patch merging

At the end of a stage, groups of 2 by 2 neighbouring tokens are concatenated and projected to a new vector. The number of tokens falls to a quarter and the channel width doubles. The next stage then works at lower resolution with richer features.

Analogy: zooming out on the map. You see fewer tiles, but each tile now summarises a larger area.

Step 9: Task heads

For classification, the last feature map is averaged and passed to a linear layer. For detection or segmentation, the multi-scale feature maps feed a feature pyramid and a task-specific head. The backbone is shared. Only the heads change.

A quick cost comparison

Key concepts and terms

A worked example

A numeric walk-through of the token counts

Take an input image of 224 by 224 pixels. Use a patch size of 4 and a window size of 7.

1. After patch partition: 224 / 4 = 56 tokens per side, so a 56 by 56 grid, or 3,136 tokens.

2. Stage 1 windows: 56 / 7 = 8 windows per side, so 64 windows of 49 tokens each.

3. After patch merging: 28 by 28 tokens, so 784 tokens. Windows: 4 per side, 16 windows.

4. After the next merge: 14 by 14 tokens, so 196 tokens. Windows: 2 per side, 4 windows.

5. After the last merge: 7 by 7 tokens, so 49 tokens. This is exactly one window, so attention at the last stage is global.

Global attention on 3,136 tokens would score about 9.8 million token pairs per head. Window attention scores 64 windows times 49 by 49 pairs, which is about 154 thousand pairs per head. That is a drop of roughly sixty times in the first stage.

Using a pre-trained Swin backbone in Python

The example below uses a typical deep learning library API with a placeholder model name. Check the library you use for the exact identifiers.

import torch
import timm

# Create a small Swin backbone that returns multi-scale features.
model = timm.create_model(
    "swin_tiny_patch4_window7_224",
    pretrained=False,       # set True to load pre-trained weights
    features_only=True,     # return the feature pyramid
)
model.eval()

x = torch.randn(1, 3, 224, 224)
with torch.no_grad():
    feats = model(x)

for i, f in enumerate(feats):
    print(i, tuple(f.shape))

The printed shapes show four stages. Spatial size shrinks by two at each stage and channel width grows. This is the pyramid that a detection head would consume.

Fine-tuning for classification

import torch.nn as nn
import timm

model = timm.create_model(
    "swin_tiny_patch4_window7_224",
    pretrained=True,
    num_classes=10,          # new head for 10 classes
)

# Freeze the backbone, train only the head first.
for name, p in model.named_parameters():
    if not name.startswith("head"):
        p.requires_grad = False

opt = torch.optim.AdamW(
    [p for p in model.parameters() if p.requires_grad], lr=1e-3
)
loss_fn = nn.CrossEntropyLoss()

A common recipe is to train the head for a few epochs, then unfreeze everything with a lower learning rate. This follows the usual transfer learning pattern.

A tiny sketch of the window partition

import torch

def window_partition(x, ws):
    # x: (B, H, W, C), ws: window size
    B, H, W, C = x.shape
    x = x.view(B, H // ws, ws, W // ws, ws, C)
    x = x.permute(0, 1, 3, 2, 4, 5).contiguous()
    return x.view(-1, ws * ws, C)   # (num_windows * B, ws*ws, C)

x = torch.randn(1, 56, 56, 96)
w = window_partition(x, 7)
print(w.shape)   # torch.Size([64, 49, 96])

The shape confirms the arithmetic above: 64 windows, 49 tokens each, 96 channels.

Reading the attention mask

The mask is the least intuitive part, so here is a small picture. Suppose a row of eight tokens is split into two windows of four, and the shift is two.

original tokens :  a b c d | e f g h
after shift 2   :  c d e f | g h a b
window 1 holds  :  c d e f   (all true neighbours)
window 2 holds  :  g h a b   (g h are neighbours, a b are neighbours,
                              but h and a are not adjacent in the image)

In window 2, the pair (g, h) may attend to each other and the pair (a, b) may attend to each other. The mask blocks attention between {g, h} and {a, b}. Without it, the model would invent a link between two regions that sit on opposite edges of the picture.

Parameter and compute budget at a glance

Pre-training choices that matter

• Supervised pre-training on a large labelled image set was the first recipe.
• Self-supervised pre-training, such as masked image modeling, also works well with this backbone and reduces the need for labels. See self-supervised learning and MAE.
• Contrastive methods such as SimCLR can also be applied to the backbone.

Strengths and limitations

Strengths

• Linear cost in image size, so it scales to large inputs.

• A built-in feature pyramid, which is what dense heads expect.

• Strong results across classification, detection and segmentation when pre-trained at scale.

• Compatible with existing detection and segmentation frameworks that expect hierarchical backbones.

• Relative position bias gives a clean notion of locality.

• Works well with transfer learning from large pre-training sets.

Limitations

• Local attention means long-range links form only after several layers.

• The shift and mask logic adds implementation complexity compared with a plain transformer.

• Window size ties the model to the resolution it was trained at, unless the position bias is interpolated.

• Like other transformers, it is data hungry and benefits from large-scale pre-training or strong augmentation.

• Memory use for high-resolution inputs is still large, even if it is linear.

• For some tasks, modern convolutional networks reach similar accuracy at lower cost.

How it compares with alternatives

Swin is often used as the backbone inside other systems. For example, a detection transformer can use a Swin backbone to supply multi-scale features, and a segmentation model can use it as an encoder. See also CNN for the older convolutional family that inspired this design.

Role in tabular foundation models

Swin is a vision model, so it is not used directly on tables. Its value for tabular foundation models is conceptual. The same engineering problem appears whenever a model reads a grid of values and wants to relate many cells to each other without paying the full quadratic price.

A table is also a grid

A table has rows and columns. Cells that share a row or a column are related. A model that learns from raw tables must decide which cells attend to which. Full attention over every cell is too costly for wide, long tables. Swin shows one answer: attend inside a structured neighbourhood, then change the neighbourhood.

Alternating neighbourhoods

In Swin, regular windows alternate with shifted windows. In tabular transformers, a similar alternation exists between attention across the columns of one row and attention across the rows of one column. Each pattern is local in one direction. Together they let information reach every cell after a few layers. The axial attention page explains this row and column pattern, and TabPFN and TabICL use related two-way attention ideas. For a description of the architecture of tabular foundation models, see /model/architecture.

Hierarchy and summarising

Patch merging compresses many tokens into fewer, richer tokens. A tabular analogue is to summarise many rows into a small set of learned vectors, or many columns into a compact row embedding. The Perceiver IO approach, with a small latent array that reads from a large input, is a close cousin of this compression idea.

Mixed types, missing values and no column names

Swin assumes every pixel is a number of the same kind. Tables break that assumption. Columns can be numeric, categorical, date or text. Some cells are missing. Column names may be absent or meaningless. A tabular foundation model therefore needs:

• Per-cell encoders that handle each type, so a number and a category both become vectors.

• A mask or flag for missing cells, so absence is a signal and not a zero.

• Column identity that comes from learned position or from data statistics, not from names.

A vision model gets its structure for free from the pixel grid. A table model has to build that structure from the data. The patch embedding step of Swin is the closest analogue of the cell encoder in a table model.

Pre-training and in-context learning

Swin is pre-trained on large image sets and then fine-tuned. Many tabular foundation models take a different route. They are pre-trained on vast numbers of synthetic or real tables, and at inference they read a labelled training table and a query row together, then predict without any gradient update. This is few-shot learning by in-context learning. See /model/research for how this is studied, and /model/benchmarks for how such models are measured.

Imputation, anomaly detection and semantic labels

Dense vision models predict a value for every pixel. A tabular model can do the same for cells. Predicting a masked cell is imputation, and the idea behind masked cell modeling mirrors masked image modeling in vision. Cells that the model finds hard to predict can flag anomalies. Pooling column representations can feed semantic sector heads, which label a table by its domain.

Mapping vision ideas to table ideas

The mapping is loose, but it explains why vision architecture papers are worth reading for anyone who builds models of tables.

Continual learning

A Swin backbone is usually fine-tuned once. Tabular models often need to adapt to new data over time without forgetting old tasks. Methods such as experience replay and EWC address this. The shared lesson with Swin is to keep the backbone general and add small task-specific parts.

Practical guidance and best practices

Choosing a variant

• Start with the tiny or small variant. Move up only if accuracy is the bottleneck.

• Match the window size to the pre-training setting. Changing it requires interpolating the relative position bias.

• Prefer a model pre-trained on a large set when your own data is small.

Training tips

• Use AdamW with weight decay and a warm-up followed by cosine decay.

• Use strong augmentation, since transformers overfit small data.

• Use stochastic depth, which drops residual branches at random during training.

• Use mixed precision to save memory. See the mixed precision page.

• If memory is tight, use gradient checkpointing.

Dealing with image sizes

• Pad the input so that height and width are multiples of the window size times the total downsampling factor.

• When moving to higher resolution, scale the window size or fine-tune at the new size.

• For detection, use the multi-scale outputs through a feature pyramid.

Deployment

• Export to a runtime format for inference. See ONNX Runtime.

• Test the exported model with the same masks and shifts, since these are easy to get wrong.

• Consider quantisation style methods only after measuring accuracy loss on your own data.

A short checklist before you train

1. Confirm the input size is divisible by the window size times the total downsampling.
2. Confirm the pre-trained weights match the variant and window size you build.
3. Run one batch and print every stage output shape.
4. Freeze the backbone for a short warm-up and check the loss falls.
5. Unfreeze with a lower learning rate for the backbone than for the head.
6. Track validation accuracy at the native resolution, not only a resized one.
7. Save the exact configuration next to the weights so the run can be repeated.

When to pick something else

• If you only classify small images and have little data, a compact convolutional network is often simpler.
• If you need global context from the first layer, such as in some medical scans, a plain transformer may fit better.
• If your data is a table and not an image, use a model made for tables, such as TabPFN, FT-Transformer or XGBoost.

Common mistakes

• Treating Swin as a drop-in for ViT. The input pipeline, padding rules and heads differ.

• Forgetting the attention mask on shifted windows, which lets wrapped-around tokens attend to strangers.

• Changing the window size without adjusting the position bias table.

• Using an input size that is not divisible by the window size and downsampling factors.

• Fine-tuning with a high learning rate, which destroys pre-trained features.

• Training from scratch on a small dataset and blaming the architecture for poor results.

• Comparing models at different input resolutions and calling the result a fair test.

• Assuming local attention cannot see globally. After enough layers and merges it does.

Frequently asked questions

What is the Swin Transformer in simple terms?

Swin is a vision transformer that looks at an image through small windows instead of all at once. It moves the windows between layers so that neighbouring areas can share information. It also shrinks the feature map in stages, so early layers see fine detail and later layers see the big picture.

Why does Swin use shifted windows?

Plain window attention would keep every window isolated, so information could never cross a window border. Shifting the window grid by half a window in alternate layers connects neighbouring windows. This costs almost nothing extra and gives the model a way to build global context over several layers.

How is Swin different from ViT?

ViT computes attention among all patches at one fixed resolution, so its cost grows with the square of the token count. Swin limits attention to local windows and builds a multi-scale hierarchy by merging patches. That makes Swin cheaper on large images and a more natural backbone for detection and segmentation.

Is Swin better than a convolutional network?

It depends on the task, the data and the budget. Swin often matches or beats older convolutional networks when pre-trained at scale, and it is a strong dense-prediction backbone. Modern convolutional designs such as ConvNeXt close much of the gap, so a fair comparison needs equal training recipes and resolutions.

What is patch merging?

Patch merging joins each 2 by 2 group of neighbouring tokens into one token with a wider vector. It reduces the number of tokens by four and doubles the channels. Doing this between stages creates the feature pyramid that dense vision heads need.

What is relative position bias?

It is a learned number added to the attention score between two tokens, chosen by how far apart they are inside the window. It tells the model about nearness without needing absolute coordinates. It also helps when the window size changes, because the table of biases can be resized.

Can Swin be used on tabular data?

Not directly, since it expects an image grid of pixels with a fixed type. The ideas transfer, though. Local attention with alternating neighbourhoods, token merging and multi-scale summaries all have counterparts in models that attend over rows and columns of a table. Practitioners usually pick a model built for tables instead.

Does Swin need a lot of data?

Like most transformers, it learns best with large pre-training sets and strong augmentation. For a small dataset, start from pre-trained weights and fine-tune. Training from scratch on a few thousand images tends to give worse results than a convolutional network of similar size.

What is Swin V2?

Swin V2 is a later version of the architecture that improves training stability and makes it easier to scale to larger models and higher-resolution inputs. It keeps the core idea of shifted window attention and changes details such as how attention scores are normalised and how position bias is computed.

Which tasks is Swin best for?

It is best known for dense prediction such as object detection, instance segmentation and semantic segmentation, where a feature pyramid is needed. It also performs well for image classification and is used as an encoder in many larger systems, including video and medical imaging models.

Glossary

• Attention: A way for each token to take a weighted average of other tokens.

• Backbone: The part of a network that extracts features, shared by several heads.

• Cyclic shift: Rolling a feature map so that shifted windows keep equal size.

• Dense prediction: A task that outputs a value for every pixel or region.

• Feature pyramid: A set of feature maps at decreasing resolution.

• Head: The final layers that turn features into a task output.

• Multi-head attention: Several attention computations in parallel, each with its own projections.

• Patch: A small square region of an image.

• Patch merging: Combining neighbouring tokens to lower resolution.

• Relative position bias: A learned attention offset that depends on token distance.

• Shifted window: A window grid displaced by half a window.

• Stage: A group of blocks working at one resolution.

• Stochastic depth: Randomly skipping residual branches during training.

• Token: A vector representing a patch or a group of patches.

• Window attention: Attention restricted to a fixed-size local group of tokens.

Summary

Swin is a hierarchical vision transformer that restricts attention to local windows and shifts the windows between layers. This keeps cost linear in image size, links neighbouring regions, and produces a multi-scale feature pyramid suited to detection and segmentation. Compared with ViT it is cheaper on large images, and compared with ResNet it brings attention and strong pre-training benefits. For tabular foundation models, it is a conceptual relative rather than a direct tool: structured local attention, alternating neighbourhoods, token compression and masked prediction all have close counterparts in models that learn from raw tables. Read more at /model/architecture, /model/research and /model/benchmarks.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: