EWC

Overview

Elastic weight consolidation, usually shortened to EWC, is a regularization method that helps a neural network learn new tasks without erasing what it learned on earlier ones. It works by adding a penalty to the training loss. The penalty is small for weights that did not matter for old tasks and large for weights that did. The network is free to change the unimportant weights and is held close to its old values on the important ones.

EWC is one of the best known answers to catastrophic forgetting in continual learning. It needs no stored copy of the old data, only a snapshot of the old weights and a per-weight importance score. That makes it attractive when old data cannot be kept for privacy, size or policy reasons. Its main weaknesses are that the importance score is approximate, that it grows in cost with the number of tasks, and that it often trails replay methods such as Experience Replay when task boundaries are unclear. This article explains the idea, the maths in plain words, a worked example, and where EWC fits next to models that learn from raw tables.

What it is

EWC is a training-time penalty. It is not a new architecture and it does not change the model at inference time. After a network has been trained on task A, EWC records two things: the final weights, and a measure of how sensitive the task A loss is to each weight. When the network then trains on task B, the loss it minimizes is the task B loss plus a quadratic term that pulls each weight back toward its task A value, scaled by that sensitivity.

In symbols, the training loss for task B looks like this:

L(theta) = L_B(theta) + (lambda / 2) * sum_i F_i * (theta_i - theta_A_i)^2

Here theta are the current weights, theta_A are the weights after task A, F_i is the importance of weight i, and lambda is a single number that sets how strongly the past is protected.

The everyday picture

Think of a person learning a second language while wanting to keep the first. Some habits in the first language, such as the sounds of common words, are used all day and must stay fixed. Other habits, such as rare slang, can drift without harm. EWC does the same for a network. It does not freeze everything. It freezes the habits that matter and leaves the rest flexible.

The word "elastic" comes from the picture of springs. Each weight is tied to its old value by a spring. Stiff springs hold important weights near their old values. Soft springs let unimportant weights move freely.

What problem it solves

Neural networks trained on a sequence of tasks tend to forget earlier tasks quickly. This is called catastrophic forgetting. Gradient descent on the new task moves weights without any knowledge of what the old task needed. EWC gives the optimizer that knowledge in the form of the penalty.

What it does not do

• It does not store old examples.
• It does not add parameters or modules for each task.
• It does not detect task boundaries by itself.
• It does not guarantee zero forgetting. It only lowers it.

Where it came from

The method was introduced in the paper "Overcoming catastrophic forgetting in neural networks" in 2017. The paper was inspired by findings in neuroscience, where memories are thought to be protected by making the connections that support them less plastic. The authors turned that idea into a rule for artificial networks.

Before EWC, the field had mostly tried three families of fixes:

• Replay, where old examples are mixed into new training.
• Architectural growth, where new units or columns are added for each new task.
• Simple regularization, such as keeping all weights near old values with the same strength.

EWC sat between these. It used a Bayesian view of learning. In that view, the posterior over weights after task A becomes the prior for task B. The paper approximated that posterior with a Gaussian centred on the old weights, and used the diagonal of the Fisher information matrix as its precision. That is why the penalty is quadratic and why the Fisher values act as spring stiffness.

What followed

EWC started a large line of work on regularization-based continual learning.

• Synaptic Intelligence, covered in Synaptic Intelligence, estimates importance online during training instead of after it.
• Memory Aware Synapses measures how much the output of the network changes when a weight changes, without needing labels.
• Online EWC and related variants keep one running penalty instead of one per task, which fixes the growth in cost.
• LwF takes a different route and preserves outputs rather than weights.
• Replay and hybrid methods, see Experience Replay, often outperform pure regularization on harder benchmarks, and EWC is frequently used as a baseline or as a component in hybrids.

The research on continual learning also feeds into newer memory-based sequence models, which is discussed in MIRAS.

How it works

EWC has two phases for each task. First it learns the task. Then it measures what mattered. The measurements are used in the next task.

Step 1: Train on the first task normally

Train the network on task A until it performs well. Nothing special is needed here. The result is a set of weights theta_A.

Step 2: Measure how important each weight was

For each weight, ask: if I nudge this weight, how much worse does task A get? Weights where a tiny nudge causes a large loss increase are important. Weights where nudges barely matter are not.

The measure EWC uses is the diagonal of the Fisher information matrix. In practice you can compute it like this:

1. Take a sample of inputs from task A.
2. For each input, compute the gradient of the log-likelihood of the model's prediction with respect to every weight.
3. Square each gradient.
4. Average the squared gradients over the sample.

The result is one non-negative number per weight. Large values mean the weight was sensitive.

An analogy helps. Imagine tuning a large mixing desk with thousands of dials. You play a song and wiggle each dial slightly. Some dials change the sound a lot. Others change almost nothing. The Fisher value is your record of how much each dial changed the sound.

Step 3: Store the snapshot

Keep two vectors the same size as the weights: the old weights theta_A and the importance values F. This is the entire memory cost for one task. Nothing about the raw data is kept.

Step 4: Train on the next task with the penalty

During task B training, the loss is the task B loss plus the quadratic penalty. The gradient of the penalty for weight i is lambda * F_i * (theta_i - theta_A_i). So a weight that has moved far from its old value, and that was important, receives a strong push back.

Step 5: Repeat for later tasks

For a third task, there are two common approaches:

• Keep one penalty per past task. This is the original form and costs memory that grows with the number of tasks.
• Merge the penalties into one running penalty. This is the online form. The old importance values are decayed and added to the new ones, and the anchor weights are updated to the latest snapshot.

Why the Fisher and not the plain gradient

At the end of training the gradient of the loss is close to zero, so it says little about importance. The Fisher uses the squared gradient of the log-likelihood, which stays informative. It is also related to the curvature of the loss. High curvature means a small move in that direction costs a lot. That is the quantity EWC needs.

Choosing lambda

The penalty strength lambda sets the trade-off between stability and plasticity. A small value allows learning but forgets. A large value protects old tasks but makes new learning slow or impossible. There is no universal value. It depends on the scale of the Fisher values, the number of tasks and the loss scale. It is usually tuned on validation data from old and new tasks together.

The diagonal approximation

The full Fisher matrix has one entry per pair of weights, which is far too large to store for modern networks. EWC keeps only the diagonal. This ignores interactions between weights. It is the main reason EWC is only approximate. Two weights might be unimportant alone but important together, and the diagonal cannot capture that.

Variants and refinements

The original recipe has been changed in many ways. The main ones are worth knowing because papers and libraries often mean one of them when they say "EWC".

Online EWC

Instead of one penalty per task, keep a single penalty. After each task, the old Fisher is multiplied by a decay factor and the new Fisher is added. The anchor moves to the latest weights.

• Memory stays constant as tasks grow.
• Older tasks fade slowly, which is a controlled form of forgetting.
• The decay factor becomes a second setting to tune.

Empirical versus true Fisher

The true Fisher samples labels from the model's own predictions. The empirical Fisher uses the real labels. The empirical form is easier, but it measures something closer to the gradient of the loss than to curvature. When the model is confident and right, the two agree. When it is uncertain, they can differ.

Per-layer or grouped importance

Some implementations normalize the Fisher per layer, so that a layer with large raw values does not dominate. Others group weights, for example by output unit, and share one importance per group. This cuts memory and can reduce noise.

Importance without labels

Some methods measure how much the output changes when a weight changes, without using labels. This is useful for unlabeled streams, and it is the basis of several later regularization methods. The trade-off is that the output sensitivity is not tied to the task loss.

Better curvature

The diagonal can be replaced by a block-diagonal or low-rank estimate of curvature. These capture some interactions between weights. They cost more memory and compute, and in practice the gain varies by problem.

What happens to the loss landscape

A helpful mental picture is a bowl. After task A the weights sit at the bottom of a bowl for task A. The bowl is steep in some directions and nearly flat in others. Steep directions are the important weights. Flat directions are the unimportant ones.

Task B has its own bowl. Plain fine-tuning rolls the weights downhill in the task B bowl, regardless of the task A bowl. EWC adds the shape of the task A bowl as an extra term. The result is a combined bowl whose lowest point lies in the flat directions of task A. That is where both tasks can be good at once.

This picture also explains a limit. If no point is low in both bowls, no penalty can help. The network simply lacks the capacity, or the tasks conflict.

Key concepts and terms

A worked example

A tiny numeric walk-through shows the mechanics. Suppose a model has two weights and was trained on task A. The final weights are theta_A = (1.0, 2.0). The Fisher values are F = (10.0, 0.1). So the first weight mattered a lot and the second barely mattered.

Now task B prefers theta = (3.0, 5.0) when trained alone. Set lambda = 1.

• Penalty on weight 1 if it moves to 3.0: 0.5 * 1 * 10.0 * (3.0 - 1.0)^2 = 0.5 * 10 * 4 = 20.
• Penalty on weight 2 if it moves to 5.0: 0.5 * 1 * 0.1 * (5.0 - 2.0)^2 = 0.5 * 0.1 * 9 = 0.45.

The first move is very expensive. The second is cheap. The optimizer will keep weight 1 near 1.0 and allow weight 2 to move toward 5.0. That is the whole idea in miniature.

A PyTorch sketch

The following code shows the core of EWC for a classifier. It is a sketch for learning, not a production library. See PyTorch for the framework itself.

import torch
import torch.nn.functional as F

def compute_fisher(model, loader, device, n_batches=50):
    """Diagonal empirical Fisher from task-A data."""
    model.eval()
    fisher = {n: torch.zeros_like(p) for n, p in model.named_parameters()
              if p.requires_grad}
    seen = 0
    for i, (x, y) in enumerate(loader):
        if i >= n_batches:
            break
        x, y = x.to(device), y.to(device)
        # one sample at a time keeps the per-example gradient exact
        for xi, yi in zip(x, y):
            model.zero_grad()
            out = model(xi.unsqueeze(0))
            loss = F.cross_entropy(out, yi.unsqueeze(0))
            loss.backward()
            for n, p in model.named_parameters():
                if p.grad is not None:
                    fisher[n] += p.grad.detach() ** 2
            seen += 1
    for n in fisher:
        fisher[n] /= max(seen, 1)
    return fisher

def snapshot(model):
    return {n: p.detach().clone() for n, p in model.named_parameters()}

def ewc_penalty(model, fisher, anchor, lam):
    total = 0.0
    for n, p in model.named_parameters():
        if n in fisher:
            total = total + (fisher[n] * (p - anchor[n]) ** 2).sum()
    return 0.5 * lam * total

# after task A
fisher_A = compute_fisher(model, loader_A, device)
anchor_A = snapshot(model)

# training on task B
for x, y in loader_B:
    optimizer.zero_grad()
    loss = F.cross_entropy(model(x), y) + ewc_penalty(model, fisher_A, anchor_A, lam=100.0)
    loss.backward()
    optimizer.step()

A tabular flavour

The same code works for a network on tabular rows. Imagine a classifier trained on rows from one region or time period (task A) and then on rows from a later period (task B). Without a penalty the model may drift to fit the later period and degrade on the earlier one. With EWC it keeps the weights that mattered for the earlier distribution stable. The check is simple: after task B, evaluate on held-out rows from both periods and compare against a model trained without the penalty.

How to read the result

Track a small table after each task:

The pattern to look for is that the old-task column falls less with EWC while the new-task column falls only a little. If the new-task column collapses, lambda is too large. If the old-task column still collapses, lambda is too small, or the Fisher estimate is poor.

Strengths and limitations

Strengths

• No raw data storage. Only the weights and importance values are kept.
• Simple to add to existing training code. It is one extra loss term.
• No change to model size or inference cost.
• Grounded in a clear probabilistic story, which helps with reasoning and debugging.
• Works with any differentiable model and loss.
• Useful when privacy or governance blocks keeping old records.

Limitations

• The diagonal Fisher misses interactions between weights.
• The Fisher is computed at one point and becomes stale as weights drift.
• With many tasks, per-task penalties grow in memory and compute unless the online form is used.
• A single global lambda may suit some layers and not others.
• It needs to know when tasks change, or at least when to take a snapshot.
• In class-incremental settings, where the model must pick among all classes seen so far without being told the task, regularization alone usually performs poorly. Output layers drift and old classes lose out.
• It does not cope well with large shifts that need new capacity.
• The Fisher computation adds a pass over old data at the end of each task. That data must be available at that moment, even though it is not kept afterward.

Practical cost

For a model with N parameters, EWC stores about 2N extra numbers per task, or 2N in total for the online form. The Fisher pass costs roughly one extra epoch over a sample of the data. These costs are small next to replay buffers for large datasets, but they can matter for very large models.

Reading the table of strengths in context

Most published comparisons of continual learning methods agree on a few broad patterns.

• Regularization methods such as EWC do reasonably well when each task has its own output head or the task identity is given.
• They do much worse when the model must choose among all classes at once.
• Replay methods are usually more reliable in the hard setting.
• Combining a small buffer with a penalty is a strong, simple default.

These patterns are general. Exact numbers depend on the benchmark, so avoid copying any figure from one setting to another.

How it compares with alternatives

A rule of thumb: if you can keep a small buffer, replay is usually the stronger baseline. If you cannot keep data, EWC and its siblings are the standard first choice. If tasks are clearly separate and you can afford extra parameters, adapters avoid interference by construction.

Role in tabular foundation models

Tabular foundation models are models that learn from raw tables. A table has rows and columns, mixed types (numbers, categories, text, dates), missing values, and often no useful column names. Models such as TabPFN, TabICL and TabSTAR are pre-trained on large collections of synthetic or real tables so that they can make predictions on a new table with little or no gradient training. The ideas behind these models are covered in /model/research and /model/architecture.

EWC enters this picture in the pre-training and update stages, not at inference. Here is how it connects, concretely.

Pre-training across many table families

A tabular foundation model may be pre-trained in stages. One stage may use synthetic tables drawn from a prior over data-generating processes. A later stage may use real tables from many domains. If the later stage simply continues gradient descent, the model can lose skills picked up from the earlier stage, such as handling heavy skew or many categorical levels. A penalty in the style of EWC can protect the weights that carried those skills while the model adapts to real data.

Continual learning as new data arrives

Real systems see new tables over time: new sectors, new schemas, new years. Retraining from scratch is costly. Continual learning lets a pre-trained model absorb new table families while keeping old ones. EWC is a simple baseline for this. The comparison that matters is how performance on old table families changes after each update, measured on a fixed board of tasks. See /model/benchmarks for how such boards are used.

Heads for specific jobs

Tabular models often carry task-specific heads, for example:

• Classification and regression heads for prediction.
• Imputation networks for filling missing cells, trained with masked cell modeling.
• Scoring heads for anomaly detection.
• Semantic sector heads that infer the business domain of a table from its content.

When a new head is added or an old head is refreshed, the shared trunk can drift and hurt the others. EWC on the trunk weights, using Fisher values from the other heads' losses, is one way to limit that drift. The head-specific weights can be left free.

Why the lack of column names matters

When tables have no reliable column names, the model must rely on values and on the statistics of each column. Its internal features then encode patterns such as "this looks like a date" or "this looks like a skewed count". These patterns live in the shared weights and are exactly what a naive update can overwrite. Importance estimates from EWC highlight which weights carry these general skills, which helps preserve them across updates.

In-context learning and continual learning

Many tabular foundation models predict by in-context learning. The training rows of a new table are fed in as context, and the model predicts for test rows in one forward pass, without weight updates. Here, forgetting at prediction time is not an issue, because the weights do not change. The risk appears when the model itself is updated. Two points are worth stating clearly.

• In-context learning gives a form of task adaptation with no forgetting, because nothing is overwritten.
• Weight-level continual learning, where EWC applies, is about improving the base model over time, not about adapting to one table.

Related memory-based approaches

Newer sequence models try to keep a learned memory that is updated while reading. These ideas are described in MIRAS. They offer a different way to carry information forward, inside the forward pass rather than in the weights. EWC and such memory modules are complementary: one protects slow weights, the other manages fast memory.

A cautious note on evidence

The claim that EWC helps a given tabular foundation model should be tested, not assumed. Gains depend on the size of the shift between tasks, the architecture, and the choice of lambda. A fair test uses a fixed set of old tasks, a fixed set of new tasks, and reports both. Anything less can hide forgetting.

Practical guidance and best practices

Before you start

• Define the tasks and their boundaries clearly. Decide whether the setting is task-incremental, domain-incremental or class-incremental. EWC works best in the first two.
• Fix an evaluation board of old tasks before training on new ones. Do not pick the board after seeing results.
• Save a baseline: the model after each task with plain fine-tuning, and a jointly trained model if you can afford one. These give the lower and upper bounds.

Estimating the Fisher well

• Use enough samples. A few hundred to a few thousand examples are often enough for stable estimates, but check by comparing two independent draws.
• Compute the Fisher at the end of the task, with the model in evaluation mode, so that dropout and batch statistics do not add noise.
• Prefer sampling labels from the model's own predictive distribution when you want the true Fisher. Using true labels gives the empirical Fisher, which is cheaper and common, but it is a different quantity and can behave differently when the model is wrong.
• Normalize the Fisher values, for example by dividing by the maximum, so that lambda is easier to compare across tasks.

Setting lambda

• Start with a coarse sweep on a log scale.
• Plot old-task and new-task accuracy against lambda. Choose a point where old-task accuracy is held and new-task accuracy is still acceptable.
• Re-tune when the model size, optimizer or loss scale changes.

Keeping cost under control

• Use the online form when there are more than a few tasks.
• Apply the penalty only to selected layers if memory is tight, such as the shared trunk.
• Use mixed precision carefully. Squared gradients can underflow in half precision, so compute the Fisher in full precision.

Combining with other methods

• Add a small replay buffer if any data can be kept. Replay plus EWC often beats either alone.
• Use adapters or LoRA for tasks that differ a lot, and EWC for tasks that are close.
• Pair with distillation, see LwF, to preserve behaviour at the output as well as in the weights.

Measuring forgetting properly

Continual learning has its own metrics. Use a few of them together.

• Average accuracy: the mean accuracy over all tasks seen so far, measured after the last task.
• Forgetting: for each old task, the best accuracy it ever reached minus its accuracy now. Average over tasks.
• Forward transfer: how much earlier learning helps a new task compared with training that task from scratch.
• Backward transfer: how much later learning changes an old task. Negative values mean forgetting, positive values mean improvement.
• Learning accuracy: how well each new task is learned right after training on it, which shows the price paid in plasticity.

For tables, replace accuracy with the metric that fits the job. Use AUC or log loss for classification, error for regression, and mean error on masked cells for imputation.

Designing the sequence of tasks

The order and shape of tasks change the result a lot.

• Tasks that share structure are easy. Tasks that conflict are hard.
• A sequence of two tasks is a poor test. Try at least five.
• Shuffle the order across several runs and report the spread.
• Include one task that is very different from the rest, to test the limits of the penalty.
• For tabular data, create tasks by splitting on time, region, customer segment or table family. Make sure the split does not leak the label.

Seeds and variance

Continual learning results are noisy. A single run can mislead. Run at least three seeds, report the mean and the range, and avoid claiming gains smaller than the spread.

Monitoring

• Log the penalty term and the task loss separately.
• Track the average distance of important weights from their anchors.
• Re-run the full old-task board after every update, not only at the end.

Diagnosing a run that does not work

When EWC seems to do nothing, check these in order.

1. Is the penalty non-zero? Print its value at the start of task B. It should be near zero at the first step and grow as weights move. If it is always zero, the anchor may be updating or the Fisher may be empty.
2. Is the Fisher well scaled? Print the mean, median and maximum Fisher values. If they are all tiny, lambda needs to be very large. If a few weights dominate, consider clipping or normalizing.
3. Is the old-task board correct? Make sure old-task evaluation uses the same preprocessing as old-task training. A changed scaler or encoder can look like forgetting.
4. Is the optimizer resetting? Adaptive optimizers keep running averages. Reset them at task boundaries, or the first steps may be misleading.
5. Is the head the problem? Look at per-class accuracy. If old classes lose most of their score while the body weights barely moved, the head is the cause.
6. Is capacity the problem? Train a joint model on all tasks. If the joint model also struggles, the network is too small or the tasks conflict.

When to prefer something else

• If tasks arrive in tiny pieces with no clear boundary, a replay-based method is easier to run.
• If old tasks must be preserved exactly, use separate adapters or separate models.
• If you only care about the final output and have new inputs that look like old ones, LwF is simpler.
• If your model is a very large pre-trained network, a parameter-efficient method such as LoRA is usually more practical.

Common mistakes

• Computing the Fisher on the wrong data. Using new-task data to estimate old-task importance gives meaningless scores.
• Forgetting to freeze the anchor. If the anchor weights are updated during training, the penalty stops working.
• Using one lambda forever. A value that works for two tasks may be wrong for ten.
• Judging success on the new task only. The point of EWC is the old tasks. Always report both.
• Ignoring the output layer. In class-incremental settings, output weights for old classes can be pushed down by new classes, and EWC on the body does not fix that.
• Fisher underflow. Tiny squared gradients can vanish in low precision, producing a flat importance map.
• Treating the Fisher as exact. It is an approximation at one point. Large weight movement makes it stale.
• Comparing with weak baselines. A fair comparison includes plain fine-tuning, a small replay buffer and a joint-training reference.
• Not resetting optimizer state. Carrying momentum from the old task into the new one can cause a jump away from the anchor.
• Assuming batch norm is covered. Running statistics are not weights and are not protected by the penalty. They need separate handling.

Frequently asked questions

What does EWC stand for and what does it do?

EWC stands for elastic weight consolidation. It is a training penalty that discourages a network from changing the weights that were important for earlier tasks, so the network can learn new tasks while keeping most of its old skill.

Does EWC need to store the old training data?

No. It stores the old weights and one importance value per weight. The old data is needed briefly at the end of a task to compute those importance values, but it does not have to be kept afterward.

How is EWC different from simply fine-tuning with a small learning rate?

A small learning rate slows all weights equally, which delays forgetting but also slows new learning. EWC slows only the weights that mattered for old tasks and leaves the others free to move, so it targets protection where it is needed.

What is the Fisher information in EWC?

It is a measure of how sensitive the model's predictions are to a small change in each weight. EWC uses its diagonal, one number per weight, as the stiffness of the spring that holds the weight near its old value.

Does EWC completely stop catastrophic forgetting?

No. It reduces forgetting but does not remove it. The diagonal approximation, the staleness of the importance values and the limited capacity of the network all leave some loss, and on hard class-incremental problems the loss can still be large.

How do I choose the penalty strength lambda?

Sweep it on a log scale and look at old-task and new-task performance together. Pick a value that keeps old-task performance close to its earlier level without making the new task impossible to learn. Re-tune when the setup changes.

How does EWC compare with Synaptic Intelligence?

Both add a quadratic penalty weighted by importance. EWC measures importance after a task using the Fisher information, while Synaptic Intelligence accumulates importance during training from the path the weights took. The second is online, the first needs an extra pass.

Is EWC useful for tabular foundation models?

It can be, mainly during updates to a pre-trained base model. In-context prediction on a new table changes no weights and so does not forget. When the base weights are retrained on new table families, an EWC-style penalty can help protect earlier skills, but the benefit should be measured on a fixed board of tasks.

Can EWC be combined with replay?

Yes, and the combination is common. A small buffer of old examples protects behaviour directly, while the penalty protects the weights that mattered. Together they often perform better than either one alone.

Does EWC work with large language models?

It can be applied in principle, but the Fisher storage and computation scale with the number of parameters, which is costly for very large models. Parameter-efficient approaches such as LoRA are often preferred in that setting, sometimes with a penalty applied only to the small trainable part.

Glossary

• Anchor: the saved weight values from the end of a previous task.
• Catastrophic forgetting: the sharp loss of old-task performance after training on new data.
• Class-incremental learning: a setting where new classes arrive over time and the model must choose among all classes without being told the task.
• Continual learning: learning from a stream of tasks or data without retraining from scratch.
• Diagonal Fisher: the per-weight sensitivity scores that ignore interactions between weights.
• Domain-incremental learning: a setting where the task stays the same but the input distribution changes.
• Empirical Fisher: a Fisher estimate computed from the true labels.
• Fisher information: a measure of how much the output distribution changes when a weight changes.
• Lambda: the coefficient that scales the EWC penalty.
• Online EWC: a variant that keeps one running penalty instead of one per task.
• Plasticity: the capacity to learn new things.
• Regularization: adding a term to the loss to guide the solution.
• Stability: the capacity to keep what was learned.
• Task-incremental learning: a setting where the task identity is known at test time.

Summary

EWC protects old knowledge by adding a quadratic penalty that ties each weight to its previous value, with a stiffness given by the diagonal Fisher information. It needs no stored data, adds no parameters and costs little at inference, which makes it a standard baseline for continual learning. Its limits are the diagonal approximation, stale importance values, tuning of lambda, and weak results when the model must separate classes it has not seen together. For tabular foundation models it is most relevant when a pre-trained base is updated with new table families, where it can help keep skills for imputation, anomaly scoring and semantic classification stable. In practice it is best used with clear task boundaries, a fixed old-task board, and often a small replay buffer. Related reading includes Synaptic Intelligence, LwF, Experience Replay, and /model/benchmarks.

Related

It belongs to Continual learning in my stack, under Foundation models and applied ML. Others in the same group: