Pruning

Overview

Pruning is the practice of removing parts of a trained neural network that contribute little to its predictions. The parts can be individual weights, whole neurons, convolutional channels, attention heads or even entire layers. After pruning, the network is usually fine-tuned briefly so the remaining parts can compensate. The goal is a model that is smaller, faster or cheaper to run while keeping its accuracy close to the original.

Pruning works because trained networks are typically over-parameterised: they contain many more weights than they strictly need. Large networks are easier to train, but once training is done, a surprising share of their weights can often be removed with little loss. How much depends on the model and task, and the speed benefit depends heavily on whether the hardware and software can take advantage of the removed parts.

An everyday comparison is pruning a fruit tree. A gardener cuts away weak, crossing or unproductive branches. The tree becomes smaller and easier to manage, and with care it keeps bearing as much fruit, because energy goes to the branches that matter. Cut too much, or the wrong branches, and the harvest suffers. Network pruning involves the same judgement about what to cut and how much.

What it is

Pruning methods differ in what they remove, how they decide and when they act.

Pruning is one of three main compression families. The others are knowledge distillation, which trains a smaller model, and quantisation, which stores numbers in fewer bits, as in GPTQ, AWQ or SmoothQuant. Production pipelines often combine all three.

Where it came from

Pruning is almost as old as practical neural networks. In 1989, Yann LeCun, John Denker and Sara Solla described Optimal Brain Damage, which used second-order information, the curvature of the loss, to estimate which weights could be removed with least harm. Babak Hassibi and David Stork extended it in 1993 with Optimal Brain Surgeon, which also adjusted the remaining weights to compensate.

Interest returned with deep learning. In 2015, Song Han and colleagues showed that large image networks could have most of their weights removed by repeatedly pruning small-magnitude weights and retraining, and their Deep Compression work combined pruning with quantisation and coding to shrink models dramatically. Structured methods that remove whole filters and channels followed, aiming for speed-ups on standard hardware.

In 2019, Jonathan Frankle and Michael Carbin published the lottery ticket hypothesis at ICLR. They found that dense networks contain small sub-networks which, if reset to their original initial weights and trained alone, can reach similar accuracy. This sparked much research into why pruning works.

Hardware also moved. NVIDIA's Ampere generation, launched in 2020, added support for 2:4 structured sparsity in its tensor cores. For large language models, one-shot methods appeared in 2023: SparseGPT by Elias Frantar and Dan Alistarh pruned large GPT-style models to around 50 per cent sparsity without retraining, and Wanda, by Mingjie Sun and colleagues, scored weights using both their magnitude and the size of the activations they multiply.

How it works

Step 1: start from a trained model

Most pruning begins with a fully trained, dense model and a validation set to measure accuracy.

Step 2: score importance

Each candidate part receives a score. The simplest is weight magnitude: weights close to zero are assumed to matter least. For channels or neurons, a common score is the norm of their weights. Activation-aware methods multiply weight size by typical input size, since a small weight on a large input can still matter. Second-order methods estimate how much the loss would rise if a weight were removed.

Step 3: remove

The lowest-scoring parts are removed until a target sparsity is reached. In unstructured pruning, removed weights are set to zero and kept at zero with a mask. In structured pruning, whole rows, columns, channels or heads are physically deleted, giving a genuinely smaller dense model.

Step 4: recover

The model is fine-tuned for a while, with the mask applied so pruned weights stay at zero. This lets the remaining weights take over lost functions.

Step 5: repeat or stop

Iterative pruning repeats scoring, removal and fine-tuning in small steps, for example 20 per cent at a time, until the target is reached. This usually keeps more accuracy than cutting everything at once.

Step 6: make it fast

Unstructured sparsity reduces storage if the model is saved in a sparse format, but speed-ups need sparse kernels and high sparsity. Structured pruning speeds up any hardware. Semi-structured 2:4 sparsity runs faster on GPUs that support it.

In PyTorch, a simple magnitude pruning step looks like this:

import torch.nn.utils.prune as prune

prune.l1_unstructured(model.fc1, name="weight", amount=0.5)  # zero 50% smallest weights
# ... fine-tune ...
prune.remove(model.fc1, "weight")  # make the pruning permanent

Key concepts and terms

A worked example

This scenario is illustrative. A team runs an image classifier based on a ResNet on edge devices in a warehouse. It has about 25 million parameters, 76 per cent top-1 accuracy on their validation set and takes 40 milliseconds per image on the device. They need under 25 milliseconds with no more than one point of accuracy loss.

1. Sensitivity analysis. They prune each layer alone by 30 per cent and record the accuracy drop. Early layers and the final classifier are sensitive; middle layers are not.
2. Choose structured pruning. Because the device has no sparse kernels, they remove whole convolutional channels, which makes the network genuinely smaller.
3. Prune gradually. In five rounds, they remove channels with the smallest weight norms, mostly from the insensitive layers, and fine-tune for two epochs after each round.
4. Measure. The final model has about 40 per cent fewer channels in the pruned layers, about 14 million parameters overall, and runs in about 23 milliseconds.
5. Recover. A longer final fine-tune brings accuracy to 75.3 per cent, within the budget.
6. Combine. They add 8-bit quantisation, re-check accuracy and confirm latency on the real device, not only in simulation.

The key decision was structured rather than unstructured pruning, because only structural removal gave real speed on their hardware.

Strengths and limitations

Strengths:

• Can shrink models substantially with modest accuracy loss.
• Structured pruning gives real speed-ups on standard hardware.
• Works on existing trained models without redesigning them.
• Combines well with quantisation and distillation.
• One-shot methods make pruning practical even for very large language models.

Limitations:

• Unstructured sparsity often gives little speed-up without special kernels or hardware.
• Aggressive pruning can harm accuracy, especially on rare cases.
• Recovery fine-tuning adds compute and needs representative data.
• Results vary by architecture and task; there is no universal safe sparsity.
• Pruned models can lose capabilities that standard benchmarks do not measure.

How it compares with alternatives

Practical guidance and best practices

1. Decide your real goal first: smaller files, lower memory, lower latency or lower cost.
2. Check what your hardware and runtime can accelerate before choosing a pruning type.
3. Run a sensitivity analysis and prune insensitive layers more.
4. Prune gradually with fine-tuning between rounds when accuracy matters.
5. Use global rather than strictly per-layer ranking for unstructured pruning.
6. Evaluate on rare classes and edge cases, not only average accuracy.
7. Measure latency on the target device, with the real runtime, after export.
8. Combine with quantisation and distillation for the largest overall gains.

Common mistakes

• Expecting unstructured sparsity to speed up inference on hardware that ignores zeros.
• Pruning the first or last layers aggressively, where networks are most sensitive.
• Skipping fine-tuning after heavy pruning.
• Leaving pruning masks in place without making them permanent before export.
• Reporting parameter counts instead of measured latency and memory.
• Judging pruned language models only by perplexity and missing lost abilities.

Frequently asked questions

How much of a network can be pruned?

It varies widely. Some image networks tolerate very high unstructured sparsity with retraining, while large language models pruned in one shot are often kept around 50 per cent. The only reliable answer comes from measuring your model on your data.

Does pruning make models faster?

Structured pruning usually does, because the model becomes smaller and dense. Unstructured pruning reduces storage but speeds up inference only with sparse kernels, high sparsity or hardware support such as 2:4 sparse tensor cores.

What is the lottery ticket hypothesis?

It is the finding, published by Frankle and Carbin in 2019, that dense networks contain sparse sub-networks which, when reset to their original starting weights and trained alone, can match the full network's accuracy. It suggests that large networks are easier to train partly because they contain many such candidates.

Should I prune or quantise first?

A common approach is to prune and fine-tune first, then quantise, because quantisation is quick and pruning benefits from full-precision fine-tuning. Either way, re-check accuracy after each step.

Can large language models be pruned?

Yes. One-shot methods such as SparseGPT and Wanda prune large language models without full retraining. Structured approaches that remove layers or attention heads are also used, often followed by distillation to recover quality.

Glossary

• Dense model: a model in which almost all weights are non-zero.
• Fine-tuning: further training after a change, here to recover from pruning.
• Filter: a set of convolution weights producing one output channel.
• Over-parameterised: having more parameters than the task strictly needs.
• Sparse kernel: code that skips zero values when computing.
• Tensor core: a specialised GPU unit for fast matrix maths.
• Top-1 accuracy: how often the model's highest-scoring prediction is correct.

Summary

Pruning removes weights, channels, heads or layers that contribute little to a trained network, then lets the rest recover through fine-tuning. Its roots go back to Optimal Brain Damage in 1989, and it gained new life with deep learning, the lottery ticket hypothesis and one-shot methods for large language models. The essential practical point is that real speed depends on hardware: structured or semi-structured pruning gives reliable gains, while unstructured sparsity mainly saves storage unless the runtime can exploit it.

Related

It belongs to Building blocks in my stack, under Deep learning. Others in the same group: