LwF

Overview

LwF stands for Learning without Forgetting. It is a method for adding a new task to a trained neural network without losing the old tasks and without needing any of the old training data. The trick is to use the old network as a teacher. Before training starts, the old network is run on the new task's inputs and its answers are recorded. While the network learns the new task, it is also asked to keep giving those same answers for the old tasks.

This is an application of knowledge distillation to continual learning. It keeps the model's behaviour in place, not its weights. That makes it simple and often effective when the new inputs look somewhat like the old ones. It is weaker when the new data comes from a very different distribution, because then the old answers recorded on new inputs say little about how the old task behaves on its own data. This article explains the method step by step, shows a worked example, compares LwF with EWC, Synaptic Intelligence and Experience Replay, and explains where it fits among models that learn from raw tables.

What it is

LwF is a training recipe for a network that has a shared body and several task-specific output heads. The body is the part that turns inputs into features. Each head turns features into the answer for one task. LwF adds a new head for the new task and trains the body and all heads, with a loss that has two parts:

• A normal loss for the new task, using the new labels.
• A distillation loss that makes the old heads keep producing the answers that the old network gave on the same new inputs.

No old images, rows or labels are needed. Only the new data and the old model are used.

The everyday picture

Picture a chef who knows a menu of dishes and now wants to learn a new cuisine. Before starting, the chef cooks a small test of the old dishes using the new kitchen's ingredients and writes down how they taste. While practising the new cuisine, the chef keeps checking that the old dishes still taste the same on those ingredients. The chef never goes back to the old kitchen. The recorded tastes act as a reminder.

In this picture, the recorded tastes are the old network's outputs, called soft targets. The check is the distillation loss.

The pieces

• Old model: the network trained on earlier tasks.
• Soft targets: the old model's output probabilities for the new inputs, usually softened with a temperature.
• New head: a fresh output layer for the new task.
• Shared body: the feature extractor, which is allowed to change.
• Combined loss: new-task loss plus distillation loss, with a weight to balance them.

What it does not do

• It does not store old data.
• It does not guard individual weights.
• It does not add capacity for each task apart from a new head.
• It does not guarantee that old tasks are kept when the new data is very different.

Where it came from

LwF was introduced in a paper called "Learning without Forgetting", first presented in 2016 and later published in a journal in 2017 or 2018. The setting was image classification, where a network trained on one dataset is extended to another task, such as a different set of classes or a different dataset altogether.

The method drew on knowledge distillation, a technique described in 2015, in which a small model learns to match the soft output probabilities of a larger one. LwF uses the same idea but for a different purpose: the teacher is the previous version of the same network, and the aim is to avoid forgetting.

Before LwF, the main options for adding a task were these:

• Feature extraction: freeze the body and train only a new head. Old tasks are safe, but new-task accuracy suffers because the features cannot adapt.
• Fine-tuning: train all weights on the new task. New-task accuracy is good, but old tasks degrade.
• Joint training: train on old and new data together. Both are good, but old data must be kept.

LwF aimed to get most of the benefit of joint training without the old data. It sits between fine-tuning and joint training in cost and in quality.

What came after

• EWC and Synaptic Intelligence protect weights directly instead of outputs.
• iCaRL and similar methods combine distillation with a small stored set of examples, which tends to work better in class-incremental settings.
• Many later methods add distillation on intermediate features, not only on outputs.
• Self-distillation and teacher-student schemes are now common in self-supervised learning and in training of large models.

How it works

LwF runs in a few clear steps.

Step 1: Start with a trained network

Suppose the network has a body and one head per old task. It performs well on the old tasks. Call this the old model and keep a frozen copy of it.

Step 2: Record the old model's answers on the new inputs

Pass the new task's inputs through the frozen old model. For each input, store the output probabilities of every old head. These are the soft targets.

Notice what is happening. The old model has never seen these inputs during its own training, yet it still produces answers for them. Those answers are a description of how the old model behaves. LwF treats that behaviour as the thing to protect.

An analogy helps. A translator who knows French is shown English sentences they have not seen. Their French guesses for those sentences are not correct answers, but they show how the translator's mind works. LwF saves those guesses and asks the translator to keep thinking the same way while learning Spanish.

Step 3: Warm up the new head

Before changing the body, train only the new head for a few epochs, with the body and old heads frozen. This avoids a problem: a new head with random weights produces large, noisy gradients that would wreck the body in the first steps. The warm-up gives the head a sensible starting point.

Step 4: Train everything with the combined loss

Now unfreeze the body and the heads and train with:

L = L_new(y_new, y_hat_new) + lambda_old * L_distill(y_old_soft, y_hat_old)

The first term is the usual loss on the new labels. The second term compares the current outputs of the old heads on the new inputs with the stored soft targets.

Step 5: Soften the probabilities

The distillation loss uses a temperature T to soften the probabilities before comparing them. With T above 1, small probabilities are enlarged, which exposes how the old model ranks the classes it considers unlikely. A value of 2 was used in the original work. The softened form is:

p_i = exp(z_i / T) / sum_j exp(z_j / T)

The loss between stored and current softened outputs is typically a cross-entropy or a Kullback-Leibler divergence.

The reason for softening is that a confident old model might give 0.999 to one class and nearly nothing to the others. Matching such a peaked output tells you little. The softened output keeps the information in the small numbers, such as "this is a little like a cat and a lot less like a truck".

Step 6: Use the model

After training, the network has a body and one head per task. For task-incremental use, the right head is chosen by the task identity. For other uses, a rule for combining heads is needed.

Step 7: Repeat for more tasks

For a third task, the model after task two becomes the old model. Soft targets for all old heads are recorded on the third task's inputs. The process repeats. Over many tasks, the stored behaviour is an approximation of an approximation, and drift can build up.

The balance parameter

The weight lambda_old sets how strongly the old behaviour is protected. A large value keeps the old tasks but slows the new task. A small value allows the new task to be learned well but old tasks slip. It is commonly set near one and tuned on validation data from both old and new tasks.

A note on training cost

The old model is run once over the new data to store the soft targets, which can be done before training starts. After that, no second model needs to be kept in memory during training, since the targets are stored. This is a practical advantage over methods that keep a teacher in the loop.

Variants and extensions

Researchers have changed the basic scheme in many ways.

• Feature distillation. Instead of, or as well as, matching head outputs, match the body's intermediate features between the old and new networks. This protects the representation directly.
• Exemplar-based distillation. Keep a few real old examples and distil on them too. This closes much of the gap in class-incremental settings.
• Unlabelled helper data. If the new data is too different, use a separate pool of unlabelled inputs that resemble the old domain as the distillation set.
• Adaptive loss weights. Raise the distillation weight when old-task performance drops and lower it when the new task is stalled.
• Self-distillation chains. Use the previous version of the model as the teacher at every stage and average teachers over time to smooth drift.

None of these is required to get value from the basic method, but each answers a specific weakness.

Why distillation preserves function

A network's function is the mapping from inputs to outputs. Weights are only one way of storing that mapping. Many different weight settings give nearly the same function. LwF cares about the function, so it leaves the weights free to move as long as the function on the sampled inputs stays put.

Think of a bridge. Its owner does not care which bolts are tightened, only that cars cross safely. A weight penalty says "do not touch these bolts". Distillation says "keep the bridge able to carry the same traffic". The second rule leaves more room for repair.

The catch is that function is only checked where you look. LwF looks at the new inputs. If the old task lives in a region of the input space that the new inputs never visit, nothing prevents the function from changing there. This is the root of the input-gap limit.

How much does the temperature matter

The temperature controls how much weight the small probabilities get.

Changing the temperature also rescales gradients. Some implementations multiply the distillation term by the square of the temperature to keep gradient sizes comparable. If you change the temperature, re-check the balance weight.

Key concepts and terms

A worked example

A small numeric example shows the distillation loss at work. The old model has one old head with three classes.

Recording the soft targets

For one new input, the old model outputs logits z = (4.0, 1.0, 0.0).

• With temperature 1, the probabilities are about (0.94, 0.05, 0.02).
• With temperature 2, the logits become (2.0, 0.5, 0.0) and the probabilities are about (0.74, 0.17, 0.10).

At temperature 2 the second and third classes are visible. These softened values are stored as the target for this input.

The student after a step of new training

After a few steps, the old head of the current network outputs logits (3.0, 2.0, 0.0) for the same input. At temperature 2 this is (1.5, 1.0, 0.0) with probabilities of about (0.51, 0.31, 0.19).

The mismatch

The target was (0.74, 0.17, 0.10). The student gives (0.51, 0.31, 0.19). The distillation loss is positive, and its gradient pushes the student toward the target. If the new task's loss is pushing the body in a direction that raises the second class too much, the distillation term resists.

A PyTorch sketch

The code below shows the main parts. It is meant for learning. See PyTorch for the framework.

import torch
import torch.nn as nn
import torch.nn.functional as F

class MultiHead(nn.Module):
    def __init__(self, body, head_sizes, feat_dim):
        super().__init__()
        self.body = body
        self.heads = nn.ModuleList([nn.Linear(feat_dim, k) for k in head_sizes])

    def forward(self, x):
        h = self.body(x)
        return [head(h) for head in self.heads]

def distill_loss(logits, soft_targets, T=2.0):
    log_p = F.log_softmax(logits / T, dim=1)
    return -(soft_targets * log_p).sum(dim=1).mean()

def record_soft_targets(old_model, loader, T=2.0, device="cpu"):
    old_model.eval()
    store = []
    with torch.no_grad():
        for x, _ in loader:
            outs = old_model(x.to(device))      # list of logits, one per old head
            store.append([F.softmax(o / T, dim=1).cpu() for o in outs])
    return store

# new model: copy the old one and add a head for the new task
model = add_head(old_model, n_new_classes)

# 1) warm-up the new head only
for p in model.body.parameters():
    p.requires_grad = False
# ... train model.heads[-1] for a few epochs on the new task ...

# 2) joint training of everything with the combined loss
for p in model.parameters():
    p.requires_grad = True

for (x, y), soft in zip(new_loader, soft_targets):
    outs = model(x)
    loss_new = F.cross_entropy(outs[-1], y)
    loss_old = sum(distill_loss(o, s) for o, s in zip(outs[:-1], soft))
    loss = loss_new + 1.0 * loss_old
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

One practical caution: if the new loader shuffles its data, the stored soft targets must be shuffled in the same way. A simpler option is to compute the old model's outputs on the fly with a frozen copy, at the cost of one extra forward pass per batch.

A tabular flavour

Suppose a network predicts customer churn on one product line (task A) and is later asked to predict a related outcome on another product line (task B) using a shared encoder of the row. Before training on task B, the old churn head is run on task B rows and its outputs are stored. During training the new head learns task B while the old head is nudged to keep its stored outputs on these rows. The test is the old churn performance on held-out task A rows, compared with plain fine-tuning.

This only works well if task B rows are similar enough to task A rows that the stored outputs are informative. If the feature columns differ completely, the old head's answers on them are close to noise.

Strengths and limitations

Strengths

• No old data. Only the new data and the old model are needed.
• Simple. One extra loss term and a new head.
• Output-level protection. It preserves what the model does, which is what users care about.
• No growth in the body. Each task adds only a head.
• No per-weight bookkeeping. There are no importance vectors the size of the model.
• Works with any architecture that has a shared body and task heads.
• Compatible with other methods. It combines easily with replay, adapters and penalties.

Limitations

• Depends on input similarity. If the new inputs are very different from the old ones, the stored outputs do not describe old behaviour well.
• Drift over many tasks. Each round approximates the previous round's behaviour, and errors accumulate.
• Task identity. It is designed for task-incremental use. Choosing among heads without the task identity is hard.
• Slower new learning. The distillation term acts as a brake on the new task.
• Cost grows with tasks. Each new task adds heads that need soft targets and loss terms.
• Class-incremental weakness. When the model must pick among all classes seen so far, it generally trails methods that keep exemplars.
• Calibration. Temperature and the balance parameter need tuning.
• No guarantee of exact retention. It preserves outputs on the new inputs, not on the old data distribution.

Where it shines

The method fits settings in which the new data is related to the old, old data cannot be kept, and the task identity is known. An example is adding another label set on top of images of the same kind. Another is extending a table model to a new target on rows with the same columns.

How it compares with alternatives

LwF versus weight penalties

Weight penalties such as EWC work in parameter space. They do not look at outputs. LwF works in output space. It does not look at weights. The two views are complementary.

• Weight penalties can protect behaviour on inputs never seen again.
• Distillation can protect behaviour even when the weights must move a lot, as long as the outputs stay stable.
• Using both is possible and often helpful.

LwF versus replay

Replay uses real old examples, so it measures old behaviour on the old distribution. LwF measures it on new inputs. When old data can be kept, even in small amounts, replay is usually stronger. When it cannot, LwF is a natural first choice.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types such as numbers, categories, dates and text, missing values, and often no useful column names. Models like TabPFN, TabICL and TabSTAR are pre-trained on many synthetic or real tables so that they can predict on a new table through in-context learning. The training rows of the new table are passed in as context, and the prediction is made in one forward pass with no weight update. See /model/research and /model/architecture for the broader picture.

LwF does not play a role in in-context prediction. It plays a role when the base model is updated and when several tasks share one encoder. Here is how the ideas connect.

A shared table encoder with many heads

A tabular foundation model has a body that turns a table, or a row, into representations. On top of it sit heads for different jobs:

• Classification and regression.
• Imputation networks, trained with masked cell modeling, to fill missing cells.
• Scoring heads for anomaly detection.
• Semantic sector heads, which infer the business domain of a table from its content.

This is exactly the shape that LwF was designed for: a shared body with one head per task. When a new head is added, such as a new sector taxonomy, LwF can protect the existing heads by distilling their outputs on the new tables.

Updating without the old tables

Old tables may be unavailable. They may be large, private or covered by retention rules. With LwF, only the new tables are needed. The old model labels them with its own outputs, and those outputs act as a reminder of what the old heads should keep doing.

Why the input gap matters for tables

Tables vary far more than images. Two tables can differ in number of columns, types, scales and meaning. If the new tables are from a very different domain, the old model's outputs on them may be unreliable. For tabular work, this makes it important to check how close the new tables are to the old ones before relying on LwF. Useful checks include comparing column type mixes, value ranges and missing-value rates.

Handling missing values and no column names

Because LwF compares outputs, it does not care whether the inputs have column names. The old model's behaviour is defined by whatever it does with the raw values. If the model was trained to cope with missing cells and unlabelled columns, the stored outputs include that behaviour. The method therefore carries those skills forward without extra work.

Pre-training on synthetic tables

A model pre-trained on synthetic tables can be extended with real tables. Distilling the synthetic-pretrained model's outputs on the real tables can keep the broad skills, such as handling heavy-tailed columns, while the model adapts. This is a form of staged training where the earlier stage acts as a teacher.

In-context learning and distillation

In-context learning adapts to one table at prediction time, with no forgetting, since the weights do not change. LwF belongs to the slow loop: improving the base model over time. The two work at different time scales.

• In-context learning: fast, per table, no weight change.
• LwF-style updates: slow, across table families, weights change and behaviour is protected by distillation.

Relation to continual learning in general

LwF, EWC, Synaptic Intelligence and Experience Replay are the main building blocks. A realistic update plan for a tabular foundation model could use a small replay set for the hardest families, a weight penalty on the trunk and distillation on the heads. Sequence models with built-in memory, described in MIRAS, offer another direction, in which the memory is updated while the data is read.

Caution

Whether LwF helps a given tabular model must be shown by testing. The test should fix a board of old table families, record performance before and after each update, and report both old and new results, as discussed in /model/benchmarks.

Practical guidance and best practices

Before you start

• Confirm the setting. LwF is best when the task identity is known.
• Check the similarity of old and new inputs. Compare simple statistics, or train a quick classifier to tell the two apart. If it is easy, the gap is large.
• Keep a frozen copy of the old model and a fixed evaluation board for old tasks.

Choosing the loss weight and temperature

• Start with a balance weight near one and a temperature near two.
• Sweep the balance weight on a log scale and examine old and new performance together.
• Higher temperatures reveal more of the small probabilities but can blur the signal. Try a few values.

Warm-up and learning rates

• Always warm up the new head before training the body.
• Use a smaller learning rate for the body than for the new head.
• Reset optimizer state when switching from warm-up to joint training.

Regression and non-probability outputs

LwF was described for classification, but the idea extends. For regression heads, use a mean squared error between the stored and current outputs instead of a softened cross-entropy. For heads that output a distribution, match the distribution parameters.

Using stored targets or a frozen copy

• Stored targets save memory during training and avoid a second forward pass.
• A frozen copy allows data augmentation, since targets can be recomputed on augmented inputs.
• With augmentation, stored targets from unaugmented inputs no longer match, so use the frozen copy.

Combining with other methods

• Add a small replay set if any old data can be kept.
• Add a weight penalty such as EWC on the body.
• Add feature-level distillation on the body's output, not only on head outputs, to keep representations stable.
• Use adapters or LoRA when the new task is very different from the old ones.

Evaluating

• Report average accuracy over all tasks, forgetting for each old task, and the accuracy of the new task.
• Run at least three seeds and report the spread.
• Test with a shuffled order of tasks.
• For tables, use metrics that fit the job, such as AUC or log loss for classification, error for regression and mean error on masked cells for imputation.

Troubleshooting a weak result

If old tasks still degrade, go through these checks.

1. Check the distillation loss. Print it at the start of the new task. It should begin near zero because the student equals the teacher, and rise only a little.
2. Check the targets. Compare a few stored targets with the frozen model's live outputs. They should match.
3. Check the order. Confirm that inputs and stored targets are aligned after shuffling.
4. Check the balance weight. Raise it and see whether old performance recovers.
5. Check the input gap. Train a quick classifier to separate old from new inputs. If it is nearly perfect, the stored outputs are not a good proxy for old behaviour.
6. Try feature distillation. If outputs match but the old task still falls, the body may be drifting in ways the heads hide.
7. Add a handful of real old examples. A large gain suggests the problem needs exemplars.

When to prefer something else

• If old data can be kept, even a small amount, Experience Replay is the stronger baseline.
• If new inputs are very different from old ones, use separate adapters or LoRA modules per task.
• If you cannot tell which task an input belongs to, use a method built for class-incremental settings.
• If you need protection of specific weights, EWC or Synaptic Intelligence is more direct.

A simple decision guide

Common mistakes

• Skipping the warm-up. A random new head can wreck the body in the first steps.
• Recording targets with the model in training mode. Dropout and batch statistics add noise. Use evaluation mode.
• Mismatched shuffling. Stored soft targets must follow the same order as the inputs.
• Using hard labels from the old model. Hard labels throw away the small probabilities that carry the information.
• Ignoring the input gap. If new inputs differ greatly from old ones, the targets are not informative.
• Judging on the new task only. The point is to keep the old tasks. Always report both.
• Forgetting the class-incremental case. Without the task identity, the head choice becomes a problem that LwF alone does not solve.
• Letting temperature differ between recording and training. The same temperature must be used in both.
• Over-weighting the distillation term. The new task then barely learns.
• Chaining many tasks without checks. Drift accumulates, so evaluate on the original board after every task.

Frequently asked questions

What does LwF stand for?

LwF stands for Learning without Forgetting. It is a method for adding a new task to a trained network by distilling the old network's outputs on the new data, so that no old data is needed.

How does LwF avoid using old data?

It records the old model's outputs on the new task's inputs before training and then asks the network to keep producing those outputs on the old heads. The old model's behaviour stands in for the old data.

Why use soft probabilities instead of hard labels?

Soft probabilities carry information about how the old model ranks the classes it finds unlikely, which a hard label discards. Softening with a temperature makes those small values visible and gives a richer target.

Does LwF work when the new data looks very different from the old data?

Not well. The stored outputs describe how the old model behaves on the new inputs, and when those inputs are far from the old distribution the outputs say little about the old task. In that case, replay or adapters are safer.

How is LwF different from EWC?

LwF protects the outputs of the model, using distillation, while EWC protects individual weights, using a penalty weighted by importance. One works in output space and the other in parameter space, and they can be used together.

Does LwF need the task identity at prediction time?

Usually yes. It builds one head per task, and the right head is chosen by the task identity. In settings where the model must choose among all classes without that information, LwF alone tends to perform poorly.

Can LwF be used for regression or for imputation?

Yes, with a change in the loss. For regression, replace the softened cross-entropy with a squared error between stored and current outputs. For imputation heads, the same idea applies to the predicted values for masked cells.

Is LwF useful for tabular foundation models?

It is relevant when a shared table encoder with several heads is updated with new table families and the old tables cannot be kept. It is not used in in-context prediction on a single table, where the weights do not change. Its value should be measured on a fixed board of old table families.

What temperature should I use?

A value near two is a common starting point, since it softens the probabilities without erasing the signal. The best value depends on how peaked the old model's outputs are, so it is worth trying a few values and keeping the same one for recording and training.

Should I combine LwF with a replay buffer?

If any old examples can be kept, yes. Replay measures old behaviour on real old data, distillation protects behaviour on new inputs, and together they usually do better than either alone, especially in harder class-incremental settings.

Glossary

• Body: the shared part of a network that turns inputs into features.
• Catastrophic forgetting: the sharp loss of old-task performance after training on new data.
• Class-incremental learning: a setting where new classes arrive over time and the model must choose among all classes without being told the task.
• Continual learning: learning from a stream of tasks without retraining from scratch.
• Distillation loss: a loss that makes a student's outputs match a teacher's.
• Feature extraction: freezing the body and training only a new head.
• Fine-tuning: training all weights on the new task.
• Head: the output layer for one task.
• Joint training: training on old and new data together.
• Knowledge distillation: transferring behaviour from a teacher model to a student model through its outputs.
• Soft targets: the teacher's probability outputs.
• Task-incremental learning: a setting where the task identity is known at test time.
• Temperature: a number that smooths probabilities before comparison.
• Warm-up: training the new head alone before updating the body.

Summary

LwF adds a new task to a trained network by recording the old model's outputs on the new inputs and asking the network to keep reproducing them while it learns the new task. It needs no old data, adds only a new head, and preserves behaviour rather than weights. Its strengths are simplicity and low storage, and its weaknesses are sensitivity to the gap between old and new inputs, drift over many tasks and weak results when task identity is not known. For tabular foundation models it fits the common pattern of a shared table encoder with several heads, such as prediction, imputation, anomaly scoring and semantic sector classification, where it can protect existing heads as new heads or new table families are added. Good practice is to warm up the new head, use evaluation mode when recording targets, tune the balance weight and temperature, and test on a fixed board of old tasks. Related reading includes EWC, Synaptic Intelligence, Experience Replay and /model/benchmarks.

Related

It belongs to Continual learning in my stack, under Foundation models and applied ML. Others in the same group: