Knowledge distillation
Knowledge distillation trains a small student model to imitate a large teacher model, keeping much of its quality at a fraction of the cost.
Overview
Knowledge distillation is a way to train a small model, called the student, to imitate a larger, more capable model, called the teacher. Instead of learning only from the correct answers in a dataset, the student also learns from the teacher's outputs: its full probability distributions, its intermediate features, or text it generates. The aim is a student that keeps much of the teacher's quality while being cheaper, faster and small enough to run where the teacher cannot.
The key insight is that a teacher's outputs carry more information than a plain label. If an image classifier says a photo is 80 per cent "husky", 15 per cent "wolf" and 5 per cent "malamute", it is telling the student not only the answer but also which mistakes are reasonable. These soft targets give the student a richer signal per example than a single hard label.
An everyday comparison is an apprentice learning from a master carpenter. A textbook gives the right final measurements. Watching the master shows which cuts were nearly chosen, which joints are similar to each other and where the tricky decisions lie. The apprentice may never have the master's full experience, but learns much faster than from the textbook alone.
What it is
Distillation comes in several forms, depending on what the student copies.
| Form | What the student learns from | Example use |
|---|---|---|
| Response-based (logit) distillation | Teacher's output probabilities, softened with a temperature | Image and text classifiers |
| Feature-based distillation | Teacher's intermediate activations or attention maps | Compressing BERT-style encoders |
| Relation-based distillation | Relationships between examples in the teacher's representation space | Metric learning and embeddings |
| Sequence-level distillation | Complete outputs generated by the teacher | Translation and language models |
| Synthetic-data distillation | Large datasets of teacher-written answers, used for SFT | Training small language models |
| Self-distillation | A model teaching a copy of itself, or a later stage teaching an earlier one | Regularisation, self-supervised learning such as DINO |
Distillation is one of three main families of model compression, alongside pruning, which removes weights, and quantisation, which stores them in fewer bits, as in GPTQ or AWQ. The three are often combined.
Where it came from
In 2006, Cristian Buciluă, Rich Caruana and Alexandru Niculescu-Mizil showed in a paper called "Model compression" that a large ensemble's behaviour could be transferred into a single small neural network by training it on the ensemble's predictions. Jimmy Ba and Rich Caruana followed in 2014 by showing that shallow networks could mimic deep ones.
The name "knowledge distillation" comes from a 2015 paper by Geoffrey Hinton, Oriol Vinyals and Jeff Dean, "Distilling the Knowledge in a Neural Network". It introduced the temperature-softened softmax and the idea of the "dark knowledge" held in the small probabilities a teacher gives to wrong classes. In the same period, FitNets extended the idea to matching intermediate features, so students could be thinner and deeper than their teachers.
With the rise of transformers, distillation became a standard way to make language models practical. DistilBERT, released by Hugging Face researchers in 2019, reported a model 40 per cent smaller and 60 per cent faster than BERT that kept about 97 per cent of its language understanding performance on their benchmarks. For large language models, the common approach became training smaller models on teacher-generated text. In early 2025, for example, DeepSeek released smaller open models distilled from its R1 reasoning model's outputs.
How it works
Step 1: train or choose a teacher
The teacher is a strong model for the task, often large or an ensemble. It is frozen during distillation.
Step 2: design the student
The student is smaller: fewer layers, narrower layers, or a different, more efficient architecture. It can be initialised randomly or from a subset of the teacher's layers, as DistilBERT did.
Step 3: soften the teacher's outputs
For classification, the teacher's logits are divided by a temperature T before the softmax. With T = 1 the distribution is usually sharp. Raising T, for example to 2 or 4, spreads probability onto the less likely classes so that their relative sizes become visible to the student.
Step 4: combine two losses
The student's logits are softened with the same T, and a Kullback-Leibler divergence measures how far its distribution is from the teacher's. This soft loss is multiplied by T² to keep its gradients on a comparable scale. The usual cross-entropy with true labels is added as a hard loss. A weight α balances the two.
import torch.nn.functional as F
def distill_loss(student_logits, teacher_logits, labels, T=4.0, alpha=0.7):
soft = F.kl_div(F.log_softmax(student_logits / T, dim=-1),
F.softmax(teacher_logits / T, dim=-1),
reduction="batchmean") * T * T
hard = F.cross_entropy(student_logits, labels)
return alpha * soft + (1 - alpha) * hard
Step 5: optionally match features
For feature distillation, small projection layers map student activations to the teacher's size, and a mean squared error or cosine loss pulls them together at chosen layers.
Step 6: language-model distillation
For generative models, the teacher writes answers, explanations or reasoning traces for a large set of prompts. The student is fine-tuned on these with ordinary next-token training. When both models share a tokenizer, the student can instead match the teacher's full next-token distributions, which carries more information per token.
Key concepts and terms
| Term | Meaning |
|---|---|
| Teacher | The larger, stronger model whose behaviour is copied |
| Student | The smaller model being trained |
| Logits | Raw scores before the softmax |
| Soft targets | The teacher's probability distribution over outputs |
| Hard targets | The true one-hot labels |
| Temperature | A divisor that softens probability distributions |
| KL divergence | A measure of how one distribution differs from another |
| Dark knowledge | Information in the small probabilities given to wrong answers |
| Feature distillation | Matching internal representations |
| Sequence-level distillation | Training on whole outputs generated by the teacher |
| Self-distillation | A model learning from its own or a copy's predictions |
| Capacity gap | When the student is too small to imitate the teacher well |
A worked example
This scenario is illustrative. A company runs a customer-support intent classifier with 40 intents. The current model is a fine-tuned transformer with about 300 million parameters, with 92 per cent accuracy on a held-out set and a latency of 80 milliseconds per query on their CPU servers. They want under 15 milliseconds.
- They choose a student with 6 layers and about 30 million parameters, one tenth of the teacher's size.
- They collect 200,000 unlabelled historical queries in addition to the 20,000 labelled training examples, and run the teacher over all of them to record its logits.
- They train the student with the loss above, using T = 4 and α = 0.7, on all 220,000 examples. The hard loss applies only to the labelled subset.
- As a baseline, they also train the same student on the 20,000 labelled examples alone.
- On the held-out set, the baseline student might reach 86 per cent and the distilled student 90 per cent, close to the teacher's 92 per cent.
- They then apply 8-bit quantisation to the distilled student and confirm accuracy holds before measuring latency.
The large pool of unlabelled queries, labelled softly by the teacher, is often where much of the gain comes from.
Strengths and limitations
Strengths:
- Produces smaller, faster, cheaper models with much of the teacher's quality.
- Can use unlabelled data, since the teacher provides targets.
- Works across tasks: classification, embeddings, generation and reasoning.
- Combines well with pruning and quantisation.
- Lets one large model serve as a teacher for many specialised students.
Limitations:
- Students rarely match the teacher fully, especially on hard or rare cases.
- Requires running the teacher on large amounts of data, which costs compute.
- Very large capacity gaps can make imitation ineffective.
- The student inherits the teacher's errors and biases.
- Model licences or terms of service may restrict using a model's outputs to train another.
How it compares with alternatives
| Technique | How it reduces cost | Retraining needed | Notes |
|---|---|---|---|
| Knowledge distillation | New smaller architecture trained to imitate | Yes, full training of student | Largest size reductions possible |
| Pruning | Removes weights or structures | Usually fine-tuning | Speed gains depend on hardware support |
| Quantisation, such as GPTQ | Stores weights in fewer bits | Often none or light | Quick memory savings |
| LoRA and adapters | Cheap fine-tuning, not compression | Light | Reduces training cost, not inference size |
| Training a small model from scratch | Smaller model on original labels only | Yes | Simpler but usually weaker |
Practical guidance and best practices
- Make the teacher as good as possible first; the student's ceiling depends on it.
- Start with a student around a quarter to a tenth of the teacher's size, then adjust.
- Initialise the student from teacher layers where architectures allow.
- Tune temperature and the soft-hard weighting on a validation set; common temperatures are 2 to 5.
- Add unlabelled, in-domain data scored by the teacher; it is often the biggest win.
- Evaluate on rare and difficult cases, not just averages, where students lose most.
- Check licences and terms before distilling from any third-party model.
- Combine with quantisation at the end and re-check accuracy.
Common mistakes
- Forgetting to multiply the soft loss by T², so it barely affects training at high temperatures.
- Using different tokenizers or label orders for teacher and student.
- Distilling only on the small labelled set and ignoring cheaper unlabelled data.
- Choosing a student so small that it cannot represent the task.
- Reporting average accuracy while ignoring regressions on important subgroups.
- Treating synthetic teacher data as verified ground truth.
Frequently asked questions
Why not just train the small model on the labels?
Soft targets from a teacher carry more information than hard labels, including how similar different classes are. Students trained with them usually reach better accuracy than the same model trained on labels alone, and they can learn from unlabelled data too.
What does temperature do?
It divides the logits before the softmax. Higher temperatures flatten the distribution, making the small probabilities on wrong classes larger and easier for the student to learn from. The same temperature is used for teacher and student during training, and the student uses T = 1 at inference.
Can the student beat the teacher?
Occasionally, on some measures, especially in self-distillation or when the student also sees ground-truth labels. Usually the student is a little worse but much cheaper.
How are large language models distilled?
Most often by generating a large set of high-quality teacher responses and fine-tuning the student on them. When the models share a vocabulary, matching the teacher's full next-token probabilities is a stronger alternative.
Is distillation the same as quantisation?
No. Quantisation keeps the same model and stores its numbers in fewer bits. Distillation trains a different, smaller model. They are complementary and often used together.
Is it allowed to distil from any model?
Not always. Some model licences and service terms forbid using outputs to train competing models. Always check before distilling from a model you do not own.
Glossary
- Compression: reducing a model's size or compute cost.
- Ensemble: several models whose predictions are combined.
- Inference: using a trained model to make predictions.
- Latency: the time taken to produce one prediction.
- Softmax: a function that turns logits into probabilities.
- Synthetic data: training data generated by a model rather than collected.
- Unlabelled data: examples without human-provided answers.
Summary
Knowledge distillation trains a compact student to imitate a larger teacher, learning from soft probabilities, internal features or generated text rather than labels alone. The idea grew from model compression work in 2006, was named and formalised by Hinton, Vinyals and Dean in 2015, and is now a standard route to small, fast models, from DistilBERT to compact language models trained on teacher outputs. It works best with a strong teacher, plenty of in-domain data and careful checks on hard cases and licences.
Related
It belongs to Building blocks in my stack, under Deep learning. Others in the same group: