ConvNeXt

Overview

ConvNeXt is a convolutional neural network that was redesigned step by step using ideas that made vision transformers successful. It uses no attention. It is a pure CNN, yet with the right training recipe and design choices it matches or beats transformer models of similar size on standard image tasks. It keeps the speed, simplicity and hardware friendliness of convolutions.

The work behind it is also a lesson in method. The authors started from a plain ResNet and applied one change at a time, measuring the effect of each. This made clear which ingredients of modern vision models come from attention and which come from better training and macro design. This article explains the changes, the block, a worked example, strengths and limits, and then connects the ideas to models that learn from raw tables.

What it is

ConvNeXt is a family of hierarchical convolutional backbones. Like a ResNet it has a stem and four stages that work at falling resolution and rising width. Unlike a ResNet its blocks are built from a large depthwise convolution, layer normalisation, an inverted bottleneck of linear layers and a smooth activation.

Everyday analogy: imagine an old, reliable car being upgraded one part at a time with ideas borrowed from a new sports model. You swap the tyres, then the engine mapping, then the seats, and after each change you test the speed. At the end you have a car that looks like the old one but drives like the new. ConvNeXt is that kind of upgrade for a CNN.

The ConvNeXt block

        x
        |---------------------------+
        v                           |
  depthwise conv 7x7                |
        v                           |
  layer normalisation               |
        v                           |
  linear: C to 4C                   |
        v                           |
  GELU                              |
        v                           |
  linear: 4C to C                   |
        v                           |
  layer scale, stochastic depth     |
        v                           |
        +<--------------------------+
        v
      output

Sizes

The family comes in tiny, small, base, large and larger variants. They differ in the number of blocks per stage and in channel widths. Larger ones are usually pre-trained on bigger image sets before fine-tuning.

Where it came from

• CNNs powered vision from the late 1980s, with a breakthrough in 2012. See CNN.
• ResNet (2015) made deep CNNs trainable. See ResNet.
• ResNeXt, MobileNet and EfficientNet introduced grouped and depthwise convolutions and inverted bottlenecks. See EfficientNet.
• Transformers (2017) transformed language modelling. See transformers.
• Vision Transformer (2020) applied them to image patches and showed strong results with large pre-training. See ViT.
• Swin Transformer (2021) added windows and a hierarchy so that transformers worked as general vision backbones and not only classifiers. See Swin.
• A ConvNet for the 2020s (2022) asked how far a plain CNN can go if it is modernised in the same way. The answer was: as far as the transformer models. This is the ConvNeXt paper.
• ConvNeXt V2 (2023) added a global response normalisation layer and trained the network with masked autoencoding adapted to convolutions, making the architecture and the pre-training method work together. See MAE.

The story shows how ideas flow. Vision transformers borrowed hierarchy from CNNs. ConvNeXt borrowed training recipes and block layout from transformers. Then both camps converged.

How it works

The best way to understand ConvNeXt is to follow the upgrade path from a ResNet-50.

Step 1: Modernise the training recipe

Before touching the architecture, use a transformer-era recipe. That means a longer schedule, the AdamW optimiser, heavy data augmentation (mixing methods, random erasing, strong crops), label smoothing, stochastic depth and an exponential moving average of weights. This alone lifts a ResNet-50 a good distance. The lesson is that part of what looked like an architecture gap was a recipe gap.

Step 2: Change the macro design

• Stage ratio. Spread blocks across stages with more in the third stage, in a pattern like 3, 3, 9, 3, similar to hierarchical transformers.
• Patchify stem. Replace the first big convolution and pooling by one non-overlapping 4x4 convolution with stride 4. This is like cutting the image into patches, as in ViT.

Step 3: Use depthwise convolution and widen

Replace the 3x3 convolution of a bottleneck with a depthwise convolution. Depthwise convolution mixes information across space only, per channel. The linear layers then mix across channels. This separation resembles attention, which mixes across space, and the feed-forward layer, which mixes across channels. To compensate for lost capacity, widen the network.

Step 4: Use an inverted bottleneck

Move to a design where the hidden layer is wider than the input and output, four times wider. This mirrors the feed-forward layer of a transformer, and it saves compute compared with the old design when combined with depthwise convolution.

Step 5: Use large kernels

Move the depthwise convolution to the start of the block and enlarge its kernel to 7x7. Attention can see the whole image at once. A large kernel brings a CNN closer to that wide view. The change is paired with the depthwise form so cost stays small.

Step 6: Swap micro details

• ReLU to GELU. A smooth activation used in modern transformers.
• Fewer activations. A transformer block has only one activation. ConvNeXt keeps just one per block.
• Fewer normalisation layers. One per block, instead of one after each convolution.
• Batch normalisation to layer normalisation. Layer normalisation works per sample, so it does not depend on batch size.
• Separate downsampling layers. Between stages, use a dedicated 2x2 convolution with stride 2 and a normalisation layer.

Step 7: Add layer scale and stochastic depth

A small learned scale is multiplied into the output of each block, starting near zero. This eases the training of deep stacks. Stochastic depth randomly drops blocks during training for regularisation.

Step 8: Put it all together

Each step brought a small gain. Together they took a ResNet-50 up to the level of a similarly sized Swin transformer. The final network is simple: a stem, four stages of repeated identical blocks, and a head.

What the large depthwise kernel does

A 7x7 depthwise kernel looks at a 7 by 7 neighbourhood in each channel. Stacked with downsampling, it gives each late-stage unit a very large view of the input. Per channel, the kernel acts like a learned spatial filter, comparable to a fixed pattern of attention weights that does not depend on content. Attention weights depend on the content, so attention is more flexible. The convolution is cheaper and has strong built-in structure.

Why layer normalisation instead of batch normalisation

Batch normalisation uses statistics across the batch. This couples samples and behaves differently in training and testing. Layer normalisation uses statistics inside each sample, so training and testing match, and it works with any batch size. Earlier attempts to replace batch normalisation in CNNs hurt accuracy. With the new recipe and block layout, the swap helped.

Stage layout of a small variant

The layout mirrors a hierarchical transformer. The ninth-block middle stage carries most of the depth, since mid-level features are where most of the modelling happens.

How ConvNeXt V2 changes the block

The second version adds one new layer inside the block, after the activation. Global response normalisation looks at the energy of each channel across the whole map, compares it with the other channels, and rescales so that useful channels stand out and redundant ones are damped. It also lets the block drop the older layer scale. The motivation was that when the network was pre-trained with masking, the channels tended to collapse into similar features. The new layer keeps them diverse.

class GRN(nn.Module):
    """Global response normalisation for channels-last tensors (B, H, W, C)."""
    def __init__(self, dim):
        super().__init__()
        self.gamma = nn.Parameter(torch.zeros(1, 1, 1, dim))
        self.beta = nn.Parameter(torch.zeros(1, 1, 1, dim))

    def forward(self, x):
        gx = torch.norm(x, p=2, dim=(1, 2), keepdim=True)     # per channel energy
        nx = gx / (gx.mean(dim=-1, keepdim=True) + 1e-6)      # relative to others
        return self.gamma * (x * nx) + self.beta + x

Because gamma and beta start at zero, the layer starts as an identity and learns how much competition to add.

Why masked pre-training needs sparse convolutions

With a transformer, hidden patches can simply be removed from the token list. With a convolution, the input is a dense grid. If you put zeros in the hidden areas, the kernels still slide over them, and information about the mask shape leaks into the features. Worse, the model can learn to copy neighbours. The fix is to treat the visible patches as a sparse set and run convolutions only on them, so hidden positions never contribute. After pre-training the sparse operations can be swapped for ordinary dense ones, because the weights are the same.

Key concepts and terms

A worked example

Counting the cost of a block

Take a stage with C = 96 channels. The depthwise 7x7 convolution has 96 * 49 = 4,704 weights. The two linear layers have 96 * 384 and 384 * 96, which is 36,864 each, so 73,728 together. In total about 78,000 weights for the block, ignoring biases and norms.

A classic ResNet bottleneck at a similar width would use a full 3x3 convolution, with C_mid * C_mid * 9 weights. At a middle width of 96 that is 82,944 for the 3x3 alone, with extra weights for the two 1x1 layers. The ConvNeXt block mixes space in a very cheap step (4,704 weights) and uses the linear layers for channel mixing.

The block in PyTorch

import torch
import torch.nn as nn

class LayerNorm2d(nn.Module):
    """Layer norm over channels for tensors shaped (B, C, H, W)."""
    def __init__(self, ch, eps=1e-6):
        super().__init__()
        self.weight = nn.Parameter(torch.ones(ch))
        self.bias = nn.Parameter(torch.zeros(ch))
        self.eps = eps

    def forward(self, x):
        mean = x.mean(1, keepdim=True)
        var = (x - mean).pow(2).mean(1, keepdim=True)
        x = (x - mean) / torch.sqrt(var + self.eps)
        return self.weight[:, None, None] * x + self.bias[:, None, None]

class ConvNeXtBlock(nn.Module):
    def __init__(self, dim, layer_scale=1e-6):
        super().__init__()
        self.dw = nn.Conv2d(dim, dim, 7, padding=3, groups=dim)
        self.norm = nn.LayerNorm(dim, eps=1e-6)
        self.pw1 = nn.Linear(dim, 4 * dim)
        self.act = nn.GELU()
        self.pw2 = nn.Linear(4 * dim, dim)
        self.gamma = nn.Parameter(layer_scale * torch.ones(dim))

    def forward(self, x):                     # (B, C, H, W)
        shortcut = x
        x = self.dw(x)
        x = x.permute(0, 2, 3, 1)             # channels last for norm and linear
        x = self.pw2(self.act(self.pw1(self.norm(x))))
        x = (self.gamma * x).permute(0, 3, 1, 2)
        return shortcut + x

A compact ConvNeXt

class TinyConvNeXt(nn.Module):
    def __init__(self, n_classes=10, dims=(32, 64, 128), depths=(2, 2, 4)):
        super().__init__()
        self.stem = nn.Sequential(
            nn.Conv2d(3, dims[0], kernel_size=2, stride=2),   # patchify
            LayerNorm2d(dims[0]))
        stages = []
        for i, (d, n) in enumerate(zip(dims, depths)):
            blocks = [ConvNeXtBlock(d) for _ in range(n)]
            if i + 1 < len(dims):
                blocks += [LayerNorm2d(d), nn.Conv2d(d, dims[i + 1], 2, 2)]
            stages.append(nn.Sequential(*blocks))
        self.stages = nn.Sequential(*stages)
        self.norm = nn.LayerNorm(dims[-1])
        self.head = nn.Linear(dims[-1], n_classes)

    def forward(self, x):
        x = self.stages(self.stem(x))
        x = x.mean(dim=(2, 3))
        return self.head(self.norm(x))

net = TinyConvNeXt()
print(net(torch.randn(2, 3, 64, 64)).shape)    # torch.Size([2, 10])

Fine-tuning a pre-trained ConvNeXt

import torchvision

model = torchvision.models.convnext_tiny(weights="DEFAULT")
model.classifier[2] = nn.Linear(model.classifier[2].in_features, 12)

# lower learning rate for the backbone, higher for the new head
params = [
    {"params": model.features.parameters(), "lr": 5e-5},
    {"params": model.classifier.parameters(), "lr": 5e-4},
]
opt = torch.optim.AdamW(params, weight_decay=0.05)

Ablation thinking

The central exercise of the ConvNeXt work can be repeated on your own models. Keep a list of changes. Apply one at a time. After each, retrain with a fixed recipe and record the score. Keep what helps and drop what does not. This reveals which change gave which gain and prevents attributing gains to the wrong source.

The exact numbers vary with setup, so treat the table as a guide to relative size, not a measurement.

Channels-first versus channels-last inside the block

The convolution wants tensors shaped batch, channels, height, width. The normalisation and linear layers are easiest to write with channels last. The block therefore permutes the tensor twice. On some hardware, keeping the whole network in the channels-last memory format avoids copies and runs faster.

model = model.to(memory_format=torch.channels_last)
x = x.to(memory_format=torch.channels_last)
with torch.autocast(device_type="cuda", dtype=torch.float16):
    logits = model(x)

Extracting multi-scale features for dense tasks

class ConvNeXtFeatures(nn.Module):
    def __init__(self, backbone):
        super().__init__()
        self.stem = backbone.features[0]
        self.stage_modules = backbone.features[1:]

    def forward(self, x):
        outs = []
        x = self.stem(x)
        for m in self.stage_modules:
            x = m(x)
            outs.append(x)
        return outs        # list of maps, from fine to coarse

A detection or segmentation head can take these maps, which differ in size by factors of two, and combine them in a feature pyramid.

A tabular analogue of the block

The block can be adapted to table cells arranged as a tensor of shape batch, rows, columns, features. The depthwise step becomes a mixing step along the row axis only, so column order never matters. The two linear layers mix features inside each cell. A learned scale on the branch keeps deep stacks stable.

class CellMixBlock(nn.Module):
    def __init__(self, dim, k=7, layer_scale=1e-6):
        super().__init__()
        # depthwise over the row axis, no mixing across columns
        self.row_mix = nn.Conv2d(dim, dim, (k, 1), padding=(k // 2, 0), groups=dim)
        self.norm = nn.LayerNorm(dim)
        self.ff = nn.Sequential(nn.Linear(dim, 4 * dim), nn.GELU(), nn.Linear(4 * dim, dim))
        self.gamma = nn.Parameter(layer_scale * torch.ones(dim))

    def forward(self, x):                    # (B, R, C, D)
        h = self.row_mix(x.permute(0, 3, 1, 2)).permute(0, 2, 3, 1)
        h = self.gamma * self.ff(self.norm(h))
        return x + h

This sketch only makes sense when rows have a meaningful order, such as time. For unordered rows, replace the row mixing step with attention across rows.

Strengths and limitations

Strengths

• Competitive accuracy. Matches transformer backbones of similar size on common image tasks.
• Simple design. One repeated block, no attention masks, no windows or shifts.
• Hardware friendly. Convolutions and linear layers are widely optimised.
• Hierarchical features. Works directly as a backbone for detection and segmentation.
• Resolution flexible. Fully convolutional, so it handles varied input sizes.
• Batch-size independent. Layer normalisation removes the batch dependence.
• Good transfer. Available as pre-trained weights in major libraries such as PyTorch.

Limitations

• Heavier than mobile designs. Not the best pick for the smallest devices. See EfficientNet.
• Static spatial mixing. A convolution kernel does not adapt to content as attention does.
• Memory use. Wide hidden layers and large kernels use activation memory.
• Large-kernel speed. Depthwise convolutions with big kernels can be slow if not well optimised.
• Needs a strong recipe. Without long training and augmentation, the gains shrink.
• Grid assumption. Like any CNN, it expects data with a spatial layout.
• Masked pre-training needs care. Standard convolutions do not handle missing patches, so special sparse methods are required.

How it compares with alternatives

A fair summary: when the recipe is equal, the gap between good convolutional and good attention models for images is small. Choose by hardware, data size, deployment needs and available pre-trained weights.

Role in tabular foundation models

ConvNeXt is a vision model. Its relevance to tabular learning is as a source of design principles and as a component in a few places.

Separate mixing across space and across channels

The ConvNeXt block does two jobs in two steps: a depthwise step mixes across positions, and linear layers mix across channels. Tabular foundation models make a similar split. One step mixes across rows (cells in the same column), one mixes across columns (cells in the same row), and a feed-forward layer mixes features within a cell. See axial attention, FT-Transformer and SAINT. The split keeps cost manageable for wide, long tables.

Modernisation as a method

The step-by-step upgrade path is a good template for tabular models, where claims about architecture are often confounded by differences in preprocessing, tuning and training length. Start with a strong baseline recipe, change one thing at a time, and measure on many datasets. See /model/benchmarks for how comparisons across many tables are framed.

Normalisation that does not depend on the batch

Tables vary a lot in size. A batch may contain only a few rows, or rows from a single dataset. Layer normalisation works per sample, so it is the norm of choice in tabular transformers and in ConvNeXt. This is a direct practical overlap.

Layer scale and stable pre-training

Layer scale starts residual branches near zero, so a deep model begins like a shallow one. This helps stability when pre-training on huge, varied collections of synthetic or real tables, where the data change character from one table to the next.

Masked pre-training and its table version

ConvNeXt V2 showed that a convolutional network can be pre-trained with masking, with sparse convolutions that skip hidden patches. That is the vision counterpart of masked cell modeling: hide some cells, predict them from the rest. For tables, the same idea gives imputation for free, because a missing value is just a masked cell. See also imputation networks.

Mixed types, missing values and no names

Image models receive a uniform tensor. Tables need type-aware cell embeddings, a mask token for gaps, and a way to cope with absent or cryptic column names. A column can be described from its values (distribution, range, repeat rate) through a small encoder. Convolutional pieces like those in ConvNeXt can serve as light column encoders over a sample of values, though attention handles the main mixing.

In-context learning and continual learning

A pre-trained table model can read labelled rows as context and predict new rows without weight updates. See TabPFN and TabICL. For long-lived systems that see new data over time, continual learning tools such as experience replay, EWC and LwF limit forgetting. Layer scale and adapter layers make it easy to add capacity without disturbing old behaviour. See adapters.

Anomaly detection and semantics

Features from a pre-trained table encoder can feed an anomaly score, such as reconstruction error or distance from normal rows. See anomaly detection. A light head can label columns or tables by sector and meaning. See semantic sector heads.

Table images

When tables come as scans, photos or charts, a ConvNeXt backbone is a strong choice for finding structure (cells, lines, headers) before reading values. This is the most direct use of the network in a table pipeline.

For the wider architecture story see /model/architecture and /model/research.

Practical guidance and best practices

Use a pre-trained model

Start with weights pre-trained on a large image set. Fine-tune with a small learning rate for the backbone and a larger one for the new head.

Adopt the full recipe

Use AdamW, weight decay, cosine decay with warm-up, strong augmentation, label smoothing, stochastic depth and an exponential moving average of weights. These choices matter as much as the block design.

Watch the layer scale and initialisation

When training from scratch, keep layer scale small at the start. If you remove it, very deep variants may become unstable.

Pick the variant by budget

Tiny and small variants suit most fine-tuning jobs. Larger ones pay off when you have data and compute. Measure speed on your hardware, as large depthwise kernels can behave differently across devices.

Use channels-last memory format

Convolutions and linear layers often run faster with channels-last layout on modern accelerators. Combine with mixed precision.

Control memory

Use gradient checkpointing for high resolution inputs and large variants.

Use it as a backbone for dense tasks

The hierarchical outputs from each stage plug into detection and segmentation heads. Extract the stage outputs and feed them to a feature pyramid.

Evaluate beyond accuracy

Check robustness to shifts, calibration and speed. Report variance across seeds.

Reuse the method

When improving any model, change one factor at a time and keep a table of results.

Fine-tuning checklist

1. Resize and normalise inputs exactly as the pre-trained model expects.
2. Replace the final linear layer to match the number of classes.
3. Use a small learning rate for the backbone, such as a tenth of the head rate.
4. Apply layer-wise learning rate decay so that earlier layers move less than later ones.
5. Keep weight decay moderate and use label smoothing.
6. Track validation accuracy and a calibration metric.
7. Save the best checkpoint and export a copy for serving.

Layer-wise learning rate decay

Earlier layers hold general features and should change little, while later layers should adapt more. Assign each stage a learning rate equal to the head rate times a decay factor raised to the distance from the head. A decay of around 0.7 to 0.9 is common.

def layerwise_groups(model, base_lr=1e-3, decay=0.8):
    stages = list(model.features)
    groups = []
    for i, stage in enumerate(stages):
        depth_from_head = len(stages) - 1 - i
        groups.append({"params": stage.parameters(),
                       "lr": base_lr * (decay ** depth_from_head)})
    groups.append({"params": model.classifier.parameters(), "lr": base_lr})
    return groups

Deployment notes

• Large depthwise kernels may need a good kernel implementation. Test the runtime you will use, such as ONNX Runtime or TensorRT.
• Half precision usually keeps accuracy. Test integer quantisation carefully, since layer normalisation can be sensitive.
• Batch several requests together to raise throughput when latency allows.

Common mistakes

• Comparing with an old recipe. A ResNet trained with a modern recipe is much stronger than one trained the old way.
• Skipping layer normalisation details. Applying it over the wrong axis breaks the model.
• Using a large learning rate for the pre-trained backbone.
• Dropping layer scale in deep variants without testing.
• Ignoring input size. Fine-tuning at a different size than pre-training can help or hurt, so test both.
• Assuming a CNN suits any data. Shuffled columns in a table have no spatial structure.
• Expecting masked pre-training to work with dense convolutions out of the box. Hidden patches leak through padding unless sparse methods are used.
• Measuring parameter counts only. Activation memory and latency decide real cost.
• Mixing up channel-first and channel-last tensors around the norm and linear layers.

Frequently asked questions

What is ConvNeXt?

ConvNeXt is a convolutional neural network family built by modernising a ResNet with ideas from vision transformers. It uses large depthwise convolutions, layer normalisation, an inverted bottleneck and a smooth activation. It has no attention, yet it matches transformer backbones of similar size on image classification and dense tasks.

Why was ConvNeXt created?

It was created to test how much of the success of vision transformers came from attention and how much from better design and training. By upgrading a plain CNN step by step, the authors showed that much of the gap closes without attention, which clarified what each ingredient contributes.

Is ConvNeXt better than a vision transformer?

Not universally. At similar sizes and with similar training recipes the two are close, and the winner depends on data size, task and hardware. ConvNeXt is simpler to deploy and handles varied image sizes naturally. Vision transformers can scale further with very large pre-training sets.

What is the large kernel for?

A 7x7 depthwise kernel gives each layer a wider view of the image than a 3x3 kernel. It brings the receptive field closer to that of attention, which sees the whole image, while keeping the cost low because depthwise convolutions have few weights.

What is layer scale?

Layer scale is a small learned factor, one per channel, that multiplies the output of a residual branch before it is added back. It starts near zero, so each block begins close to an identity. This helps deep networks train stably.

What is new in ConvNeXt V2?

ConvNeXt V2 adds a global response normalisation layer that encourages channels to compete, and it pairs the architecture with a masked autoencoder style pre-training that uses sparse convolutions to skip hidden patches. Together these changes improved results, especially after self-supervised pre-training.

Can ConvNeXt run on small devices?

It can, particularly the tiny variant, but it is heavier than designs made for the smallest devices. For strict limits on memory or power, compact families such as EfficientNet or mobile-focused models are usually better choices. Quantisation and export tools can help close the gap.

Does ConvNeXt work for tabular data?

Not directly, because tables lack the spatial neighbourhoods that convolutions rely on. Its ideas do carry over: separate mixing across positions and across features, normalisation that does not depend on batch size, and careful ablation. It is also useful for reading tables that arrive as images.

How does ConvNeXt relate to masked autoencoding?

The second version was designed together with a masked autoencoder method adapted to convolutions. Hidden patches are skipped using sparse operations so they do not leak into the encoder. This links the model to the same idea used for masked cell modeling in tables.

Should I train ConvNeXt from scratch?

Usually not. Training from scratch needs a long schedule, strong augmentation and a lot of data. Starting from pre-trained weights and fine-tuning is faster and more accurate for most projects. Train from scratch only when your images differ greatly from common photo collections.

Glossary

• Ablation. An experiment that removes or adds one factor to see its effect.
• Backbone. The feature-extracting part of a model.
• Depthwise convolution. A convolution applied to each channel separately.
• GELU. A smooth activation function.
• Global response normalisation. A layer that normalises channel responses against each other.
• Hierarchical. Arranged in stages of falling resolution and rising width.
• Inverted bottleneck. A block with a wider hidden layer.
• Layer normalisation. Normalisation over the features of one sample.
• Layer scale. A learned per-channel factor on a residual branch.
• Patchify. Splitting an image into non-overlapping patches with a strided convolution.
• Receptive field. The input region that influences a unit.
• Sparse convolution. A convolution that processes only the visible positions.
• Stochastic depth. Random skipping of blocks in training.

Summary

ConvNeXt shows that a pure convolutional network, once modernised with a strong training recipe, a patchify stem, large depthwise kernels, an inverted bottleneck, layer normalisation and a smooth activation, can match transformer backbones on images. The design is simple, hardware friendly and works well as a backbone for many tasks.

For tabular foundation models, ConvNeXt offers principles more than parts: separate mixing across positions and across features, batch-independent normalisation, stable deep training through layer scale, masked pre-training that yields imputation, and disciplined step-by-step evaluation. It is also a good reader of table images. Use pre-trained weights, adopt the full recipe, and measure real cost. Related pages include CNN, ResNet and MAE.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: