ResNet

Overview

ResNet, short for residual network, is a deep convolutional network built from blocks that add their input back to their output. This shortcut, called a skip connection or residual connection, lets gradients flow through very deep stacks. Before ResNet, adding layers beyond a certain point made networks harder to train and sometimes less accurate. ResNet removed that barrier and made networks with dozens or even hundreds of layers practical.

The idea is small but its reach is wide. Residual connections now appear in transformers, diffusion models, language models and tabular networks. ResNet itself remains a standard vision backbone and a common starting point for transfer learning. This article explains how residual blocks work, shows a worked example, lists strengths and limits, and then explains how residual ideas shape models that learn from raw tables.

What it is

ResNet is a family of convolutional networks. A typical member has an initial convolution, four stages of residual blocks, a global average pooling layer and a linear classifier. The names ResNet-18, ResNet-34, ResNet-50, ResNet-101 and ResNet-152 refer to the number of weighted layers.

At its core is a simple rule. Instead of asking a block to learn the full output H(x) from input x, ask it to learn only the difference F(x) = H(x) - x. The block output is then x + F(x).

Everyday analogy: you are editing a document. It is easier to say "keep everything and change these three words" than to retype the entire document. A residual block keeps the input by default and only describes the change. If no change is needed, the block can learn to output zero and pass the input through untouched.

The residual block

            x
            |------------------+
            v                  |
     conv 3x3 + norm + ReLU    |  (identity or 1x1 projection)
            v                  |
     conv 3x3 + norm           |
            v                  |
            +<-----------------+
            v
          ReLU
            v
          output

Two block types

The bottleneck squeezes channels with a 1x1 convolution, does the costly 3x3 work on fewer channels, and expands again. This is like packing a suitcase tightly for travel and unpacking at the destination.

Where it came from

The path to ResNet runs through a puzzle.

• LeNet and AlexNet showed that convolutional networks could learn from data, especially with GPUs. See CNN.
• VGG (2014) showed that depth helps: stacks of small convolutions beat shallow nets.
• The degradation problem then appeared. Making a plain network deeper made training error go up, not down. This was not overfitting, since training error itself rose. Optimisation was failing.
• Highway networks (2015) introduced gated shortcuts as a possible fix.
• Deep Residual Learning for Image Recognition (2015) proposed ungated identity shortcuts. It won a major image recognition contest that year and made networks of 100+ layers trainable.
• Identity Mappings in Deep Residual Networks (2016) showed that placing normalisation and activation before the convolution ("pre-activation") keeps the shortcut path clean and helps very deep networks.
• Wide ResNet, ResNeXt, DenseNet and others explored width, grouped convolutions and dense connections.
• Transformers (2017) adopted residual connections in every block, around both attention and feed-forward layers. See transformers.
• ResNet strikes back style training recipes (2021) showed that ResNet-50 improves a lot with modern training tricks, which led to modernised designs such as ConvNeXt.

How it works

Step 1: The problem with depth

Training uses backpropagation. The gradient at an early layer is a product of many factors, one for each layer above it. If these factors are mostly smaller than one, the product shrinks toward zero: the vanishing gradient problem. If they are larger than one, it explodes. Either way, early layers learn slowly or erratically.

Analogy: whisper a message down a line of a hundred people. By the end, the message is faint or garbled.

Step 2: Add a shortcut

With a skip connection, the gradient has a second path. At each block the derivative of x + F(x) with respect to x is 1 + F'(x). The 1 means the gradient can flow straight back, undiminished. The message now has an express lane as well as the long way round.

Step 3: Learn the residual

Because the shortcut handles the default, the block only needs to learn a correction. If the best function is close to the identity, learning a correction near zero is easier than learning the identity from scratch with a stack of nonlinear layers.

Step 4: Match shapes

Addition needs equal shapes. When a block changes the number of channels or halves the spatial size, the shortcut uses a 1x1 convolution with stride (a projection) to match. Otherwise it is a pure identity with no parameters.

Step 5: Stack stages

ResNet-50 has four stages with 3, 4, 6 and 3 bottleneck blocks. Each stage works at a smaller spatial size and with more channels than the previous one. Typical channel widths are 256, 512, 1024 and 2048 at the block outputs.

Step 6: Normalise

Batch normalisation follows each convolution. It keeps activations well scaled and lets learning rates be larger. A common trick is to initialise the last normalisation scale in each block to zero, so that every block starts as an identity and training begins from a shallow effective network.

Step 7: Pool and classify

After the last stage, global average pooling reduces each map to one number. A linear layer maps the 2048 numbers to class scores.

Step 8: Train

The loss is cross entropy, the optimiser is stochastic gradient descent with momentum or AdamW, and the learning rate follows a schedule. Data augmentation with random crops and flips is standard.

A view through the unrolled network

A residual network can be read as an ensemble of paths of different lengths. With n blocks there are 2^n paths from input to output, since each block can be taken or skipped. Most gradient flows through the shorter paths. This partly explains why removing a single block at test time hurts a ResNet only a little, while removing a layer from a plain network can ruin it.

A link to differential equations

If you write the block as x_next = x + F(x), it looks like one step of a simple numerical method for solving an ordinary differential equation. A very deep ResNet then approximates a smooth flow of the features. This view inspired continuous-depth models. See neural ODE.

The layout of ResNet-50

The model has roughly twenty-five million parameters. The smaller ResNet-18 has roughly eleven million. These figures are approximate and depend on the head size.

Each time the spatial size halves, the channels double. That keeps the compute per stage roughly balanced. It is a pattern that most later CNNs copied.

Pre-activation variant

In the original block the order is convolution, normalisation, activation, with a final activation after the addition. In the pre-activation variant the order inside the branch becomes normalisation, activation, convolution, and nothing follows the addition. The shortcut path is then a pure identity from the first block to the last.

class PreActBlock(nn.Module):
    def __init__(self, ch):
        super().__init__()
        self.bn1 = nn.BatchNorm2d(ch)
        self.conv1 = nn.Conv2d(ch, ch, 3, 1, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(ch)
        self.conv2 = nn.Conv2d(ch, ch, 3, 1, 1, bias=False)

    def forward(self, x):
        out = self.conv1(torch.relu(self.bn1(x)))
        out = self.conv2(torch.relu(self.bn2(out)))
        return x + out          # nothing after the addition

Very deep networks, over a hundred layers, tend to train more smoothly with this layout. The same idea explains why modern transformers apply normalisation before each sub-layer.

Stochastic depth

During training, each residual branch can be dropped at random, and only the shortcut is used. At test time all branches are active, scaled accordingly. This acts as a regulariser and shortens the effective depth during training.

class DropBranch(nn.Module):
    def __init__(self, p):
        super().__init__()
        self.p = p

    def forward(self, x, branch_out):
        if self.training and torch.rand(()) < self.p:
            return x                      # skip this branch entirely
        keep = 1.0 - self.p
        return x + branch_out / keep if self.training else x + branch_out

Key concepts and terms

A worked example

A tiny numeric picture

Take a scalar input x = 2.0. Let a plain two-layer block compute F(x) = 0.1 * x, which gives 0.2. A plain network would output 0.2, losing most of the signal. A residual block outputs x + F(x) = 2.2. The signal survives.

Now stack ten such blocks. In the plain version the value is multiplied by 0.1 ten times: it becomes 2.0 * 10^-10, which is effectively zero. In the residual version each block multiplies by 1.1, so after ten blocks the value is 2.0 * 1.1^10, about 5.19. The signal is preserved, and gradients behave the same way in reverse.

Residual block in PyTorch

import torch
import torch.nn as nn

class BasicBlock(nn.Module):
    def __init__(self, in_ch, out_ch, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_ch, out_ch, 3, stride, 1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_ch)
        self.conv2 = nn.Conv2d(out_ch, out_ch, 3, 1, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_ch)
        self.relu = nn.ReLU(inplace=True)
        self.short = nn.Identity()
        if stride != 1 or in_ch != out_ch:
            self.short = nn.Sequential(
                nn.Conv2d(in_ch, out_ch, 1, stride, bias=False),
                nn.BatchNorm2d(out_ch),
            )

    def forward(self, x):
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return self.relu(out + self.short(x))

A small ResNet

class MiniResNet(nn.Module):
    def __init__(self, n_classes=10):
        super().__init__()
        self.stem = nn.Sequential(
            nn.Conv2d(3, 32, 3, 1, 1, bias=False),
            nn.BatchNorm2d(32), nn.ReLU(inplace=True))
        self.layers = nn.Sequential(
            BasicBlock(32, 32), BasicBlock(32, 32),
            BasicBlock(32, 64, stride=2), BasicBlock(64, 64),
            BasicBlock(64, 128, stride=2), BasicBlock(128, 128),
        )
        self.head = nn.Linear(128, n_classes)

    def forward(self, x):
        x = self.layers(self.stem(x))
        return self.head(x.mean(dim=(2, 3)))

net = MiniResNet()
print(net(torch.randn(4, 3, 32, 32)).shape)   # torch.Size([4, 10])

Fine-tuning a pre-trained ResNet

The usual practical use is to load a pre-trained backbone and replace the head.

import torch.nn as nn
import torchvision

model = torchvision.models.resnet50(weights="DEFAULT")
for p in model.parameters():
    p.requires_grad = False            # freeze the backbone first
model.fc = nn.Linear(model.fc.in_features, 5)   # new head for 5 classes

opt = torch.optim.AdamW(model.fc.parameters(), lr=1e-3)
# train the head for a few epochs, then unfreeze the last stage
for p in model.layer4.parameters():
    p.requires_grad = True
opt = torch.optim.AdamW(
    [{"params": model.fc.parameters(), "lr": 1e-3},
     {"params": model.layer4.parameters(), "lr": 1e-4}])

Residual MLP for plain vectors

The same idea works without convolutions. This block suits flat feature vectors.

class ResidualMLPBlock(nn.Module):
    def __init__(self, dim, hidden, p=0.1):
        super().__init__()
        self.norm = nn.LayerNorm(dim)
        self.net = nn.Sequential(
            nn.Linear(dim, hidden), nn.GELU(), nn.Dropout(p),
            nn.Linear(hidden, dim), nn.Dropout(p))

    def forward(self, x):
        return x + self.net(self.norm(x))

A full residual network for table rows

This sketch shows a complete model for a table with numeric and categorical columns, in the spirit of a ResNet baseline for tabular data.

class TabResNet(nn.Module):
    def __init__(self, n_num, cat_sizes, dim=128, depth=4, n_out=2):
        super().__init__()
        self.cat_emb = nn.ModuleList(
            [nn.Embedding(n + 1, 16) for n in cat_sizes])   # +1 for missing
        in_dim = n_num * 2 + 16 * len(cat_sizes)            # values + mask flags
        self.inp = nn.Linear(in_dim, dim)
        self.blocks = nn.Sequential(*[ResidualMLPBlock(dim, dim * 2) for _ in range(depth)])
        self.out = nn.Sequential(nn.LayerNorm(dim), nn.ReLU(), nn.Linear(dim, n_out))

    def forward(self, x_num, x_cat):
        miss = torch.isnan(x_num)
        x_num = torch.nan_to_num(x_num, nan=0.0)
        embs = [e(x_cat[:, i]) for i, e in enumerate(self.cat_emb)]
        h = torch.cat([x_num, miss.float()] + embs, dim=1)
        return self.out(self.blocks(self.inp(h)))

Three details matter. Missing numbers are replaced by zero and flagged by a mask channel, so the model can tell a true zero from a gap. Each categorical column has an extra index for missing. And the blocks use layer normalisation rather than batch normalisation, which is more robust for small or uneven batches.

How much does the shortcut matter

A simple experiment makes the point. Train two networks on the same data, identical except for the shortcut. At depth 4 they perform alike. At depth 30, the plain one often fails to train at all, while the residual one trains fine. You can print the gradient norm at the first layer to see why:

def first_layer_grad_norm(model, x, y, loss_fn):
    model.zero_grad()
    loss_fn(model(x), y).backward()
    first = next(model.parameters())
    return first.grad.norm().item()

In the plain deep model that number can be many orders of magnitude smaller than in the residual one.

Strengths and limitations

Strengths

• Trains very deep networks. The shortcut keeps gradients healthy.
• Simple and general. The idea needs only an addition.
• Strong, well-understood baseline. ResNet-50 is a standard point of comparison.
• Rich ecosystem. Pre-trained weights, recipes and conversions exist in all major frameworks such as PyTorch and TensorFlow.
• Good transfer. Features transfer to detection, segmentation and medical imaging.
• Hardware friendly. Dense convolutions run fast on GPUs and on optimised runtimes such as TensorRT.
• Influential. The residual idea spread to almost every modern architecture.

Limitations

• Not the most efficient. Newer designs reach higher accuracy at the same cost. See EfficientNet and ConvNeXt.
• Memory use. Stored activations for the shortcuts add up in deep nets. Gradient checkpointing can help.
• Local receptive field. Like other CNNs, it needs depth to see global context.
• Batch normalisation dependence. Small batches make training less stable.
• Fixed grid assumption. It is not suited to data without neighbourhood structure.
• Diminishing returns with depth. Beyond a point, extra layers add little.
• Training recipe sensitivity. Accuracy depends strongly on augmentation and schedule, not only the architecture.

How it compares with alternatives

Role in tabular foundation models

Models that learn from raw tables do not use ResNet as a vision backbone, but the residual idea sits inside almost all of them.

Residual connections as a universal part

A transformer block for tables wraps attention and feed-forward layers in residual connections. Without them, deep stacks of attention over cells, rows and columns would be hard to train. See FT-Transformer, SAINT and TabPFN. So when a tabular foundation model uses dozens of layers, it relies on the same principle ResNet proved.

A ResNet-style baseline for tables

Studies of deep learning on tabular data have found that a plain MLP is a weak baseline, while an MLP with residual blocks, normalisation and dropout is a strong and simple one. A "ResNet for tabular data" is a stack of blocks like the ResidualMLPBlock above, applied to a vector made from the row. It is a handy reference when judging whether a fancier tabular model is worth its cost. Gradient-boosted trees such as XGBoost are still a key baseline too.

Mixed types and missing values

A tabular residual network needs numbers. Categories become embeddings, numeric columns are scaled or binned, and missing values are marked with a mask flag or a learned token. Foundation models for tables push this to the cell level, so that each cell is a token. See masked cell modeling.

No column names, order and symmetry

A ResNet on images respects spatial layout. A table has no such layout. A residual MLP over a flat vector fixes an order of columns, which is a weakness if the order is arbitrary. Attention-based models can be made order-free. When there are no useful column names, the model must infer meaning from values and from interactions between columns, which depth and residual paths help to capture.

Pre-training, in-context learning and continual learning

Pre-training a deep stack on many synthetic or real tables only works if training is stable, and residual paths provide that stability. In-context learning takes labelled rows as input and predicts for new rows in a single forward pass. It depends on deep attention stacks with residual connections. For continual learning, residual structure also helps a little: new layers or adapters can be added as residual branches that start near zero, so the old function is preserved at the start. See adapters and experience replay.

Imputation, anomalies and semantics

• Imputation. A residual autoencoder for tables learns to refine a rough guess of a missing value. See imputation networks.
• Anomaly detection. Residual autoencoders reconstruct normal rows well and abnormal rows poorly. The error is the score. See anomaly detection.
• Semantic classification. A residual head over a pooled table embedding can classify columns or whole tables by sector or meaning. See semantic sector heads.

The word "residual" in other senses

In statistics, a residual is the difference between a prediction and the truth. Boosting trees fit residuals step by step. A ResNet block is a different but related notion: it models a correction on top of a default. Both share the same thought, which is to model what is left to explain.

For the broader architecture picture, see /model/architecture, /model/research and /model/benchmarks.

Practical guidance and best practices

Pick the right depth

• ResNet-18 or 34 for small data or fast iteration.
• ResNet-50 as the default balanced choice.
• ResNet-101 or deeper only when accuracy matters more than cost.

Use pre-trained weights

Fine-tuning a pre-trained model is nearly always better than training from scratch on a small set. Freeze the backbone first, train the head, then unfreeze gradually with a lower learning rate for the backbone.

Use a modern training recipe

Label smoothing, mixup or cutmix, a cosine schedule with warm-up, weight decay, exponential moving average of weights and strong augmentation can lift accuracy a great deal without changing the architecture.

Initialise the last norm scale to zero

This makes each residual branch start as an identity and often speeds up early training.

Keep shapes consistent

When the number of channels or stride changes, use a projection shortcut. Check shapes with a dummy forward pass.

Use mixed precision

Mixed precision reduces memory and increases speed with little loss of accuracy.

Watch batch size

Batch normalisation needs reasonable batches. With very small batches, use group normalisation or sync statistics across devices with data parallelism.

Deploy efficiently

Fold batch normalisation into convolutions for inference, export to a portable format such as ONNX Runtime and test latency on the target device.

Use residuals in your own designs

When a model gets deep, add a shortcut. Use pre-normalisation for very deep stacks. Scale the residual branch with a small learned factor if training is unstable.

Troubleshooting guide

Cost awareness

The cost of a convolutional layer is about the product of output height, output width, input channels, output channels and kernel area. Halving the spatial size cuts cost by four, doubling channels raises it by four. ResNet balances these two effects stage by stage, which keeps compute per stage similar. Knowing this helps when you resize inputs or change widths.

When to prefer something else

• For tight latency or memory limits, compare with EfficientNet.
• For the best accuracy at large scale, compare with ViT and ConvNeXt.
• For plain tables, start with trees and a residual MLP before reaching for anything bigger.

Common mistakes

• Adding depth to fix underfitting without checking data and recipe.
• Forgetting to match shortcut shapes. The addition fails or silently broadcasts.
• Placing activation after the addition in a pre-activation design, which breaks the clean identity path.
• Fine-tuning everything at a high learning rate. This destroys useful pre-trained features.
• Using different input normalisation from the pre-trained model.
• Evaluating in training mode. Batch normalisation and dropout behave differently.
• Ignoring the train and test resolution gap.
• Using a ResNet on shuffled tabular columns as if they had spatial structure.
• Comparing models trained with different recipes. The recipe often matters more than the architecture.
• Assuming deeper is always better. Past a point, returns shrink and cost climbs.

Frequently asked questions

What is ResNet in simple terms?

ResNet is a deep convolutional network that adds a block's input to its output. This shortcut lets very deep networks train properly, because information and gradients can travel straight through the stack. The result is that adding layers helps accuracy rather than hurting it.

What is a skip connection?

A skip connection is a path that bypasses one or more layers and joins the output further along, usually by addition. In ResNet it carries the input of a block straight to its output. It gives gradients a short route during training and makes it easy for a block to behave like an identity.

Why does ResNet solve the vanishing gradient problem?

During backpropagation the derivative of the block output with respect to its input includes a term equal to one, from the shortcut. This means the gradient can pass backward without shrinking at each layer. The long stack of layers no longer multiplies the signal into nothing.

What do the numbers in ResNet-50 mean?

The number counts the layers that have weights, mainly convolutions and the final linear layer. ResNet-50 has fifty such layers arranged in bottleneck blocks across four stages. Larger numbers mean deeper, heavier and usually slightly more accurate networks.

Is ResNet still used today?

Yes. It is a common baseline, a popular backbone for transfer learning, and a building block inside larger systems. Newer designs often beat it, but its simplicity, availability and speed keep it in wide use, and its residual idea lives on in nearly every modern model.

Can I use ResNet for tabular data?

You can use the residual idea, in the form of a residual MLP, and it makes a strong simple baseline for tables. A convolutional ResNet is not suited to tables, since neighbouring columns usually have no spatial relationship. Attention-based models and tree ensembles are other strong choices.

What is the difference between ResNet and a transformer?

ResNet builds features with local convolutions in a grid, with residual shortcuts around each block. A transformer mixes information across all tokens with attention, and also uses residual shortcuts around each attention and feed-forward layer. The shortcut is common to both.

What is a bottleneck block?

A bottleneck block uses a 1x1 convolution to reduce the number of channels, a 3x3 convolution on the reduced set, and another 1x1 convolution to restore the channels. It cuts computation compared with two full 3x3 convolutions, which is why deeper ResNets use it.

Should I freeze layers when fine-tuning?

It is usually wise to start with the backbone frozen and train only the new head, then unfreeze later stages with a smaller learning rate. This protects useful general features while the head is still random. With large target datasets, full fine-tuning from the start can also work.

How does ResNet relate to boosting residuals?

Both model what remains after a default prediction. A residual block predicts a correction added to its input. Gradient boosting fits each new tree to the errors of the current ensemble. The mechanism differs, but the thought of learning a correction rather than the whole function is shared.

Glossary

• Backpropagation. The method that passes error gradients back through a network.
• Basic block. A residual block with two 3x3 convolutions.
• Batch normalisation. A layer that standardises activations over a batch.
• Bottleneck. A narrow stage used to save compute.
• Degradation. Worse training error in deeper plain networks.
• Fine-tuning. Further training of a pre-trained model on a new task.
• Identity shortcut. A parameter-free skip path.
• Pre-activation. Normalise and activate before convolving.
• Projection shortcut. A 1x1 convolution used on the skip path to match shapes.
• Residual. The difference between a target and a default, or the learned correction in a block.
• Stage. A group of blocks at the same resolution.
• Stem. The first layers that process the raw input.
• Vanishing gradient. Gradients that shrink to nothing in deep networks.

Summary

ResNet adds the input of each block back to its output, so the block only has to learn a correction. This simple change lets gradients travel through very deep networks and ended the problem that deeper plain networks trained worse. The family from ResNet-18 to ResNet-152 became a standard backbone and a go-to starting point for transfer learning.

For tabular foundation models, the lasting value of ResNet is the residual connection itself. It stabilises deep attention stacks, supports pre-training on many tables, and gives a strong residual MLP baseline for flat features. Use pre-trained weights for images, modern training recipes for best accuracy, and residual paths in your own deep designs. Related pages include CNN, EfficientNet and ConvNeXt.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: