EfficientNet

Overview

EfficientNet is a family of convolutional networks designed to give high accuracy at a small cost in parameters and computation. Its central idea is compound scaling. Instead of making a network deeper, or wider, or feeding it larger images, EfficientNet grows all three together with one simple rule. The family runs from a small base model, usually called B0, to much larger versions that raise accuracy step by step.

The design matters for two reasons. First, it showed that how you scale a network can matter as much as the building blocks inside it. Second, it became a practical choice for teams that need a good image model on limited hardware, such as phones, edge devices and small servers. This article explains compound scaling, the building block, a worked example with code, strengths and limits, and then how efficiency thinking carries over to models that learn from raw tables.

What it is

EfficientNet is a CNN family built from mobile inverted bottleneck blocks, often written MBConv, with squeeze-and-excitation attention and a smooth activation. A baseline network was found with an automated search for an architecture that balances accuracy against computation. That baseline was then scaled up into a series of models by a fixed formula.

Everyday analogy: think about enlarging a photograph for a poster. If you only stretch the width, the picture is distorted. If you only raise the pixel count, the file grows but the shape is wrong. To keep it sharp and well proportioned you scale everything in step. Compound scaling does the same for a network.

The three dimensions of scale

Each dimension alone saturates. A very deep but narrow network, or a very wide but shallow one, wastes capacity. A large image needs a bigger receptive field, so more depth, and more channels to hold the finer patterns it contains. This is the intuition behind scaling the three together.

The compound coefficient

A single number, the compound coefficient, controls the size of the whole family. Depth grows as a base raised to that number, width as another base, and resolution as a third. The bases are chosen by a small search on the baseline so that doubling the budget of compute raises accuracy as much as possible. The constraint is that compute roughly doubles for each unit step of the coefficient.

Where it came from

• LeNet and AlexNet opened the era of deep CNNs. See CNN.
• VGG and ResNet raised accuracy mainly by adding depth. See ResNet.
• Wide networks and larger inputs were each explored separately as other ways to scale.
• MobileNet (2017 and after) introduced depthwise separable convolutions and, in its second version, the inverted residual block, for efficient models on mobile devices.
• Neural architecture search (mid 2010s) automated the design of blocks. A search method tuned for both accuracy and speed produced mobile-sized networks.
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019) combined a searched baseline with compound scaling and reached top accuracy with far fewer parameters than earlier large models.
• EfficientNetV2 (2021) added fused blocks in early stages, a progressive training schedule that grows image size and augmentation strength over time, and faster training.
• Later models such as ConvNeXt and vision transformers took the lead on accuracy in many settings, while EfficientNet stayed popular where cost matters.

How it works

Step 1: Start with a good baseline

The baseline, B0, is a modest network with a stem convolution, a series of MBConv stages and a final head. Because scaling can only amplify what the baseline already does well, the baseline matters. It was found by searching for a design that gives the best accuracy for a fixed amount of computation.

Step 2: Understand the MBConv block

The MBConv block has three ideas.

1. Expand. A 1x1 convolution raises the number of channels, usually several times.
2. Depthwise filter. A depthwise convolution filters each channel on its own with a 3x3 or 5x5 kernel. It is cheap because it does not mix channels.
3. Project. A 1x1 convolution brings the channels back down.

Plus a residual connection when the input and output shapes match, as in ResNet. The block is "inverted" because the wide part is in the middle, the opposite of a classic bottleneck where the middle is narrow.

Analogy: a funnel pours water into a wide basin, a fine mesh filters it there, and a second funnel brings it back to a narrow pipe. The expensive filtering happens in the basin, but the mesh is cheap because each strand works alone.

Step 3: Add squeeze and excitation

Inside many blocks, a small module looks at the average of each channel, passes these through two tiny layers, and produces a weight for each channel between zero and one. The channels are then rescaled. This lets the block emphasise useful channels and mute others based on the whole image. It costs little and helps accuracy.

Step 4: Use a smooth activation

EfficientNet uses a smooth activation sometimes called swish, defined as the input times its sigmoid. It behaves like ReLU for large positive values but is smooth near zero, which often trains slightly better.

Step 5: Apply compound scaling

To make a bigger model, pick a coefficient. Multiply the number of layers in each stage by the depth factor. Multiply channel counts by the width factor. Resize the input by the resolution factor. Then round to sensible numbers, such as multiples of eight for channels.

Because compute grows roughly in proportion to depth, width squared and resolution squared, the rule keeps total growth predictable. A step of one in the coefficient is designed to roughly double the compute.

Step 6: Train with regularisation

Larger variants overfit more easily, so they use stronger regularisation. Dropout in the head grows with size, stochastic depth drops residual branches at random, and strong data augmentation is used.

Step 7: Deploy

Model files are small, and the block types are supported by common runtimes. For fast serving, export and optimise with tools such as ONNX Runtime or TensorRT.

Why depthwise separable convolutions save compute

A standard 3x3 convolution from C channels to C channels costs about 9 * C * C multiplications per position. A depthwise 3x3 followed by a 1x1 mix costs about 9 * C + C * C. For C = 64 the standard cost is 36,864 and the separable cost is 4,672. That is nearly an eight times saving. The price is a small drop in expressiveness per layer, which the wider expansion in MBConv helps repay.

A note on memory versus FLOPs

Counting operations does not tell the whole story. Depthwise convolutions have low arithmetic intensity, meaning they do little computation per byte moved. On some GPUs they run slower than their operation count suggests. This is one reason later versions swapped some depthwise blocks for regular ones in early stages. Always measure real latency on the hardware you will use.

The shape of the baseline

The baseline network follows a familiar pattern. A stem convolution reduces the image. Seven stages of MBConv blocks follow, each with its own kernel size, expansion ratio and repeat count. A final 1x1 convolution widens the features, then global pooling and a linear layer produce class scores.

The mix of 3x3 and 5x5 depthwise kernels came out of the search. Larger kernels see more context but cost more, so the search placed them where they paid off.

Training recipe in brief

The original models were trained with a tuned optimiser, an exponential moving average of weights, a long schedule, dropout, stochastic depth and strong augmentation. The point for practitioners is that the architecture alone does not deliver the published accuracy. The recipe is half the result. When fine-tuning, you can use a lighter recipe, but keep the pieces that guard against overfitting.

EfficientNetV2 in more detail

The second generation made three changes worth knowing.

1. Fused blocks early on. In the first stages, a single regular 3x3 convolution replaces the pair of expansion and depthwise steps. On modern accelerators this is often faster, because early layers work on large images where depthwise operations are slow.
2. Progressive learning. Training starts on small images with weak augmentation and grows both over time. Early epochs are cheap, and the later ones refine detail.
3. Adjusted scaling. The scaling rule limits image size growth, because very large images slowed training too much.

The result trains several times faster than the first generation at similar accuracy. The lesson is general: a model should be designed with the training cost in mind, not only the inference cost.

Key concepts and terms

A worked example

Numeric walk-through of compound scaling

Suppose a baseline has depth factor 1, width factor 1 and resolution 224. Pick illustrative bases: depth 1.2, width 1.1, resolution 1.15. These numbers are only for illustration.

For coefficient 2:

• Depth multiplier: 1.2^2 = 1.44.
• Width multiplier: 1.1^2 = 1.21.
• Resolution multiplier: 1.15^2 = 1.3225, so the input becomes about 296 pixels.

Compute grows with depth * width^2 * resolution^2. Here that is 1.44 * 1.21^2 * 1.3225^2, which is 1.44 * 1.4641 * 1.7490, about 3.69. With the design aim of roughly doubling compute per step, two steps give about four times the compute. The numbers line up, which is why the three bases are chosen so that base_depth * base_width^2 * base_resolution^2 is close to two.

Check with the illustrative bases: 1.2 * 1.1^2 * 1.15^2 = 1.2 * 1.21 * 1.3225 = 1.92. That is close to two.

Using a pre-trained EfficientNet for transfer learning

import torch
import torch.nn as nn
import torchvision

model = torchvision.models.efficientnet_b0(weights="DEFAULT")
in_features = model.classifier[1].in_features
model.classifier = nn.Sequential(
    nn.Dropout(0.3),
    nn.Linear(in_features, 7),        # 7 target classes
)

# freeze the feature extractor, train only the classifier first
for p in model.features.parameters():
    p.requires_grad = False

opt = torch.optim.AdamW(model.classifier.parameters(), lr=1e-3, weight_decay=1e-2)

Writing an MBConv block

class SqueezeExcite(nn.Module):
    def __init__(self, ch, reduced):
        super().__init__()
        self.fc1 = nn.Conv2d(ch, reduced, 1)
        self.fc2 = nn.Conv2d(reduced, ch, 1)

    def forward(self, x):
        s = x.mean(dim=(2, 3), keepdim=True)
        s = torch.sigmoid(self.fc2(nn.functional.silu(self.fc1(s))))
        return x * s

class MBConv(nn.Module):
    def __init__(self, in_ch, out_ch, expand=6, k=3, stride=1):
        super().__init__()
        mid = in_ch * expand
        self.use_res = (stride == 1 and in_ch == out_ch)
        self.block = nn.Sequential(
            nn.Conv2d(in_ch, mid, 1, bias=False), nn.BatchNorm2d(mid), nn.SiLU(),
            nn.Conv2d(mid, mid, k, stride, k // 2, groups=mid, bias=False),
            nn.BatchNorm2d(mid), nn.SiLU(),
            SqueezeExcite(mid, max(1, in_ch // 4)),
            nn.Conv2d(mid, out_ch, 1, bias=False), nn.BatchNorm2d(out_ch),
        )

    def forward(self, x):
        out = self.block(x)
        return x + out if self.use_res else out

Notice groups=mid in the middle convolution. That makes it depthwise: each channel has its own filter.

Counting the saving

def conv_params(cin, cout, k):
    return cin * cout * k * k

def separable_params(cin, cout, k):
    return cin * k * k + cin * cout

print(conv_params(64, 64, 3), separable_params(64, 64, 3))
# 36864 4672

Scaling a configuration in code

import math

def scale_config(base_depths, base_widths, base_res, phi,
                 a=1.2, b=1.1, g=1.15):
    d = [max(1, math.ceil(x * a ** phi)) for x in base_depths]
    w = [int(round(x * b ** phi / 8) * 8) for x in base_widths]
    r = int(base_res * g ** phi)
    return d, w, r

print(scale_config([1, 2, 2, 3], [16, 24, 40, 80], 224, phi=2))

The function rounds channels to multiples of eight, which suits hardware, and rounds depth up so no stage disappears.

Measuring real latency

Operation counts can mislead. A short benchmark gives the truth for your hardware.

import time
import torch

@torch.no_grad()
def latency_ms(model, size=224, batch=1, warmup=10, iters=50, device="cpu"):
    model = model.eval().to(device)
    x = torch.randn(batch, 3, size, size, device=device)
    for _ in range(warmup):
        model(x)
    if device == "cuda":
        torch.cuda.synchronize()
    t0 = time.perf_counter()
    for _ in range(iters):
        model(x)
    if device == "cuda":
        torch.cuda.synchronize()
    return (time.perf_counter() - t0) / iters * 1000.0

for name in ["efficientnet_b0", "resnet50"]:
    m = getattr(torchvision.models, name)(weights=None)
    print(name, round(latency_ms(m), 1), "ms")

Run this on the device you plan to deploy to. Warm-up runs matter, because the first calls include one-time setup costs. On a GPU, synchronise before reading the clock, or the timer will stop before the work is done.

Exporting for deployment

model = torchvision.models.efficientnet_b0(weights=None).eval()
dummy = torch.randn(1, 3, 224, 224)
torch.onnx.export(
    model, dummy, "efficientnet_b0.onnx",
    input_names=["image"], output_names=["logits"],
    dynamic_axes={"image": {0: "batch"}, "logits": {0: "batch"}},
    opset_version=17,
)

After export, compare outputs of the original and the exported model on a few samples. Small numeric differences are normal, but large ones point to an export problem.

A table-oriented use of the same block

A depthwise-then-pointwise pattern can be used on a grid of table cells arranged as rows by columns with a feature dimension. The depthwise step mixes information within a column across neighbouring rows, which makes sense for ordered rows such as time. The pointwise step mixes features within a cell. It is a cheap way to give a table encoder some local row context.

class RowColumnMixer(nn.Module):
    """Input: (batch, features, rows, columns)."""
    def __init__(self, feat, k=3):
        super().__init__()
        self.dw = nn.Conv2d(feat, feat, (k, 1), padding=(k // 2, 0), groups=feat)
        self.pw = nn.Conv2d(feat, feat, 1)
        self.norm = nn.BatchNorm2d(feat)

    def forward(self, x):
        return x + self.pw(torch.nn.functional.silu(self.norm(self.dw(x))))

The kernel shape (k, 1) slides only along rows. Columns are never mixed by position, so the arbitrary order of columns does not matter. That respects the symmetry of a table. Mixing across columns is left to attention.

Strengths and limitations

Strengths

• High accuracy per parameter. The smaller models beat much larger older designs.
• A smooth size ladder. One family covers tiny to large budgets with a clear rule.
• Good transfer. Pre-trained models fine-tune well on many image tasks.
• Small files. Handy for mobile and edge use.
• Well supported. Available in major libraries such as PyTorch and TensorFlow.
• Clear idea. Balanced scaling is easy to explain and to reuse in other models.

Limitations

• FLOPs do not equal speed. Depthwise layers and large activations can be slow on some hardware.
• High activation memory. Expanded channels in MBConv use a lot of memory during training.
• Sensitive to training recipe. Results depend on augmentation, regularisation and schedule.
• Surpassed in accuracy. Newer convolutional and attention models lead on large benchmarks.
• Complex implementation. Many small parts make custom changes harder than in a plain ResNet.
• Resolution dependence. Larger variants expect larger inputs, which raises the cost of data handling.
• Scaling rule is empirical. The chosen bases come from a search on one baseline, not a law of nature.

How it compares with alternatives

If you need an image model on constrained hardware, EfficientNet is a reasonable first candidate to benchmark against MobileNet-style models. If you have ample compute and data, modern alternatives may give higher accuracy.

Role in tabular foundation models

EfficientNet is a vision model, so its direct use for tables is limited. Its lessons, however, apply to anyone building models that learn from raw tables.

Scale all axes together

Tabular models also have several axes: depth of the network, width of the embeddings, number of rows seen in context, number of columns handled, and amount of pre-training data. Growing only one often wastes effort. The compound view suggests tuning them in step and checking where returns flatten. See /model/architecture for how such trade-offs are framed.

Efficiency in a world of wide tables

A real table can have hundreds of columns and many thousands of rows. Attention over every cell costs the square of the number of cells. Models therefore use tricks that echo EfficientNet: factorised attention across rows and columns instead of all cells together, as in axial attention, or compact cell encoders that act like a depthwise filter per column followed by a light mix. Efficiency in the building block decides whether a model can read a large table in a single pass.

Where convolutional parts appear

• Column encoders. A small depthwise-style convolution over a sample of a column's values can create a column descriptor, which helps when column names are missing.
• Document tables. When tables arrive as images or scanned pages, an EfficientNet backbone can find cells and lines before text and numbers are read.
• Time-ordered columns. One-dimensional versions of separable convolutions can scan long numeric columns cheaply.

Mixed types, missing values and no names

Image models see one kind of value. Tables mix numbers, categories, dates and text, with gaps. Foundation models for tables give every cell a type-aware embedding and a mask token for gaps, then use attention to relate cells. EfficientNet's block design does not address this. What carries over is the principle of cheap per-element processing followed by a mix across elements, which many table encoders follow. See masked cell modeling and imputation networks.

Pre-training, in-context learning and continual learning

Vision models like EfficientNet are pre-trained on large image sets and fine-tuned. Tabular foundation models are pre-trained on many real or synthetic tables, and some solve new tasks by reading labelled rows in context, with no weight update. See TabPFN and TabICL. The efficiency lesson is the same: a small, well-scaled model that was pre-trained widely can beat a larger model trained from scratch on one dataset. When data keeps arriving, continual learning tools such as experience replay and EWC help avoid forgetting.

Imputation, anomalies and semantics

• Imputation. Compact encoders and light decoders fill missing cells at low cost, which matters when millions of rows need repair.
• Anomaly detection. A small model that reconstructs normal rows well can run at scale as a screening step. See anomaly detection.
• Sector and semantic classification. A light classification head on a pre-trained table encoder labels columns or tables by meaning. See semantic sector heads.

For related benchmarks and research notes, see /model/benchmarks and /model/research.

Practical guidance and best practices

Choose the variant by budget

• Start with the smallest variant that meets accuracy needs.
• Measure latency and memory on the target device, not only the parameter count.
• Match the input resolution to the variant. Larger variants expect larger images.

Fine-tune in stages

Train the new head first, then unfreeze the last blocks, then the rest, with decreasing learning rates for earlier layers. This preserves pre-trained features. See transfer learning.

Use the right normalisation

Batch normalisation works with moderate batch sizes. For tiny batches, consider group normalisation or freeze the statistics.

Regularise larger models

Raise dropout and stochastic depth for bigger variants, and use strong augmentation such as random resized crops and mixing methods.

Save memory in training

Use mixed precision and gradient checkpointing. MBConv activations are large, so these two give real savings.

Deploy with an optimised runtime

Fuse batch normalisation into convolutions, consider quantisation, and export to a portable format. Test accuracy after each step.

Benchmark honestly

Compare models under the same data, augmentation and schedule. Report throughput as well as accuracy. Include variance across seeds.

Apply the scaling idea elsewhere

When sizing your own model, avoid changing a single axis. Change two or three axes by small amounts and measure the effect on a validation score and on cost.

A fair comparison protocol

When you evaluate EfficientNet against another backbone, follow a clear protocol.

1. Fix the dataset, the split and the augmentation.
2. Use the same optimiser, schedule and number of epochs, or tune each within a similar budget.
3. Report accuracy, parameter count, peak memory and measured latency.
4. Repeat with at least three seeds.
5. Check calibration, not only accuracy, if the outputs will drive decisions.
6. Test on data shifted from training, such as a different camera or lighting, to see robustness.

Typical use cases

• Product photo classification for catalogues.
• Defect detection on production lines, where models run near the camera.
• Medical image triage as a first-pass filter.
• Mobile apps that classify photos on the device.
• Feature extraction: use the pooled features as input to a simpler classifier.

Sizing advice by situation

Common mistakes

• Judging cost by parameters alone. Activation memory and latency can matter more.
• Using the wrong input size. Feeding a large variant small images wastes capacity.
• Skipping the matching preprocessing. Mean, spread and resize rules must match the pre-trained weights.
• Freezing batch normalisation incorrectly. Mixed train and eval behaviour can hurt accuracy.
• Assuming the baseline can be ignored. Scaling cannot fix a poor baseline.
• Over-trusting FLOPs. Depthwise layers can run below the speed their operation count suggests.
• Training large variants with weak regularisation. They overfit quickly.
• Comparing against older training recipes. Fair comparisons use the same modern recipe.
• Forcing a convolutional model onto tables without neighbourhood structure.

Frequently asked questions

What is EfficientNet?

EfficientNet is a family of convolutional networks that reach high image accuracy with a small number of parameters and modest compute. It combines an efficient baseline built from mobile inverted bottleneck blocks with a rule called compound scaling that grows depth, width and resolution together.

What is compound scaling?

Compound scaling is a method for making a network bigger in a balanced way. One coefficient controls the growth of three factors: the number of layers, the number of channels, and the input image size. Scaling them together gives better accuracy than scaling only one of them for the same cost.

What do B0 to B7 mean?

They are sizes in the original family. B0 is the small baseline, and each larger letter uses a larger compound coefficient, giving a deeper, wider network that expects a larger input image. Accuracy rises with the number, and so do compute and memory needs.

Is EfficientNet faster than ResNet?

Not always. It uses fewer parameters and operations for a given accuracy, but depthwise convolutions and large activations can limit speed on some hardware. On many GPUs a ResNet of similar accuracy can run at comparable or higher throughput, so measure on your own device.

What is an MBConv block?

An MBConv block expands the channels with a 1x1 convolution, filters each channel on its own with a depthwise convolution, then projects back down with another 1x1 convolution. A residual connection is added when shapes match. A squeeze and excitation module often sits in the middle.

What is squeeze and excitation?

It is a small attention step over channels. The block averages each channel into a single number, passes the numbers through two small layers, and produces a weight between zero and one for each channel. Channels are then multiplied by their weights, so useful ones are boosted.

Is EfficientNet good for transfer learning?

Yes. Pre-trained weights fine-tune well on many image tasks, and the smaller variants are quick to train on a single device. Use a staged approach: train a new head first, then unfreeze deeper layers with lower learning rates.

Can EfficientNet be used for tabular data?

Not directly. It expects image-like grids, and tables have no spatial neighbours between columns. It can help with table images such as scanned documents. For actual table values, attention-based and tree-based models are better suited.

What changed in EfficientNetV2?

The second version added fused blocks in early stages, where a regular convolution can run faster than the depthwise form, and used a progressive training schedule that increases image size and augmentation strength over time. These changes aimed to speed up training and improve accuracy.

Should I pick EfficientNet or ConvNeXt?

Pick EfficientNet when parameters, file size and compute are tight, as on edge devices. Pick ConvNeXt when you have more compute and want strong accuracy with a simple, regular layout. Benchmark both on your data, since the winner depends on hardware and task.

Glossary

• Activation memory. Memory used to store layer outputs for backpropagation.
• Compound coefficient. The number that sets overall model size.
• Depthwise convolution. A per-channel spatial filter.
• Expansion ratio. How much an MBConv block widens its channels.
• FLOPs. A count of floating point operations.
• Inverted bottleneck. A block that is wide in the middle.
• MBConv. Mobile inverted bottleneck convolution.
• Neural architecture search. Automated design of network structure.
• Pointwise convolution. A 1x1 convolution that mixes channels.
• Progressive learning. Increasing difficulty and image size during training.
• Receptive field. The input region that affects a unit.
• Squeeze and excitation. Channel reweighting based on global context.
• Stochastic depth. Random skipping of residual blocks during training.
• Swish or SiLU. A smooth activation equal to x times sigmoid of x.

Summary

EfficientNet pairs an efficient, searched baseline built from MBConv blocks with compound scaling, a rule that grows depth, width and resolution together. This gives a ladder of models that deliver strong image accuracy for their size and move well to new tasks through fine-tuning. Its weak points are memory use, uneven real speed on some hardware, and a lead that newer models have taken in accuracy.

For tabular foundation models, EfficientNet is a source of design lessons rather than a direct tool: scale several axes together, keep per-element work cheap, and measure real cost. Convolutional parts can help with table images and ordered columns, while attention across cells, rows and columns handles mixed types, missing values and missing names. Related pages include CNN, ResNet and ConvNeXt.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: