Synaptic Intelligence

Overview

Synaptic Intelligence, often shortened to SI, is a continual learning method that protects what a neural network has already learned while it trains on new tasks. It does this by keeping a running score for every weight. The score says how much that weight helped to reduce the loss along the path the weights took during training. When a new task arrives, SI adds a penalty that discourages large changes to weights with high scores.

The key idea is that importance is measured online, while training happens, instead of in a separate pass afterward. This makes SI cheap, because the numbers it needs are mostly by-products of ordinary gradient descent. It needs no stored data from old tasks. Like other penalty methods it can reduce forgetting but not remove it, and it works best when task boundaries are known. This article explains the idea in plain words, walks through the formulas with a numeric example, compares SI with EWC and LwF, and shows where it fits next to models that learn from raw tables.

What it is

SI is a regularization method for continual learning. Regularization means adding an extra term to the loss that guides the solution. In SI the extra term is a quadratic penalty on how far each weight moves from the value it had at the end of the previous task. Each weight has its own strength, called its importance.

The name comes from biology. In the brain, the connections between neurons are called synapses, and the idea behind the method is that each synapse could keep a local record of how useful it has been. In the network, each weight plays the role of a synapse and keeps a local, running record. "Local" matters: each weight needs only information about itself, namely its gradient and its change, not the full structure of the network.

The everyday picture

Imagine a team of people building a large model railway over many weekends. On the first weekend they lay the main track. Over the weekend, a log keeps track of which pieces each person moved and how much those moves helped the track work. On the next weekend, when a new line is added, anyone who wants to move a piece that was heavily involved in making the first line work must give a stronger reason. Pieces that were moved a lot but never helped can be shifted freely.

SI keeps that log during the first weekend, as work happens. EWC, in contrast, tests each piece after the weekend by nudging it and seeing what breaks.

What it protects

SI protects the function the network computes for old tasks by limiting changes to the weights that contributed most to learning them. It does not store old inputs, labels or outputs.

What it does not do

• It does not add new parameters or modules.
• It does not change inference. At prediction time the model is an ordinary network.
• It does not find task boundaries by itself.
• It does not guarantee that old tasks will be fully kept.

Where it came from

SI was introduced in the paper "Continual Learning Through Synaptic Intelligence" in 2017. It was proposed shortly after EWC and belongs to the same family of regularization approaches. The authors were motivated by a limitation of EWC: its importance values are computed at the end of a task, in a separate step, and are not available while the task is being learned.

The paper tested the idea on image classification sequences that became standard in the field, such as permuted versions of a digit dataset, where the pixels are shuffled by a fixed permutation for each task, and splits of a small image dataset into several groups of classes. These benchmarks are simple but they helped to compare methods in a clear way.

Related work around it

• EWC uses the Fisher information, computed after each task.
• Memory Aware Synapses measures sensitivity of the output to each weight, without labels.
• Online variants of EWC keep a single running penalty.
• LwF preserves outputs by distillation rather than preserving weights.
• Experience Replay keeps examples and mixes them into training.
• Newer memory-based sequence models, discussed in MIRAS, approach the same goal inside the forward pass.

The broader field is called continual learning, or lifelong learning, and connects to transfer learning and meta-learning.

How it works

SI has three moving parts: a running tally inside each training step, a consolidation step at the end of each task, and a penalty during the next task.

Step 1: Notice that a weight's contribution can be tracked along the path

During training, every optimizer step changes each weight by a small amount and changes the loss by a small amount. If you look at one weight, the change in the loss that is due to that weight's change in one step is approximately the gradient times the step. Summed over all steps, this gives the total reduction in loss that the weight is responsible for.

Think of a hiker walking downhill. Each step changes the height a little. If you track how much of each step was east-west and how much was north-south, you can say how much of the total descent came from each direction. SI does the same for each weight.

Step 2: Keep a running tally per weight

For each weight k, keep a number w_k, starting at zero at the beginning of a task. At every training step, add:

w_k  +=  - g_k * delta_k

Here g_k is the gradient of the loss with respect to weight k at that step, and delta_k is how much the weight changed in that step. The minus sign makes the contribution positive when the loss goes down. Large positive totals mean the weight moved in a direction that really helped the loss.

This costs almost nothing. The gradient is already computed. The step is known from the optimizer. Only one extra number per weight is stored.

Step 3: Normalize by how far the weight moved overall

At the end of a task, compute the total distance each weight moved from its start-of-task value to its end-of-task value. Call it D_k. A weight that contributed a lot of loss reduction while moving only a little is very sensitive. A weight that contributed the same amount while moving far is less sensitive per unit of movement.

The importance for this task is:

Omega_k  =  w_k / ( D_k^2 + xi )

The small constant xi prevents division by zero when a weight did not move. The squared distance in the denominator means that Omega_k is large when a small shift would matter, which is the same idea as high curvature in EWC.

Step 4: Accumulate over tasks

Importance values add up over tasks:

Omega_k  =  sum over past tasks of Omega_k(task)

Weights that mattered for several tasks become progressively stiffer.

Step 5: Penalize movement in the next task

When training on the next task, the loss is:

L(theta) = L_new(theta) + c * sum_k Omega_k * (theta_k - theta_ref_k)^2

where theta_ref is the weight vector at the end of the previous task and c is a strength constant. This has exactly the same shape as the EWC penalty, with SI's Omega in place of the Fisher.

Step 6: Reset the tally and the reference

At the start of each new task, the running tally w_k is set to zero, the reference weights are updated, and training continues.

Why path-based importance is plausible

The tally measures actual contribution during learning, not a snapshot at the end. A weight that moved a lot but did not reduce the loss gets a low score, and the network can later move it again without harm. A weight that helped the loss during learning gets a high score. This is a natural definition: importance equals contribution to what was learned.

A subtle point: the loss can go down by noise

In stochastic training, the gradients are noisy. A weight may appear to contribute because of noise in a minibatch. Over many steps, the noise tends to average out, but it adds variance to the scores. The xi constant and the squared distance help to keep scores stable, but they do not remove all noise.

Comparison of when numbers are computed

Variants and follow-up ideas

Several later methods build on the same scheme or answer its weak points.

• Bayesian variants. Some methods treat the weights as distributions, and use the spread of each distribution as importance. They connect the penalty to a probabilistic story, like the one behind EWC. See also Bayesian deep learning.
• Rotation and curvature fixes. Because the penalty is diagonal, it is sensitive to how the weights are parameterized. Methods that rotate the weight space or use richer curvature try to capture interactions between weights.
• Intelligent synapses with noise handling. Averaging the tally over several runs or using gradient clipping can reduce noise in the contribution scores.
• Unified toolkits. Many continual learning libraries bundle SI, EWC and replay, so that they can be compared under one protocol.

None of these is required to get value from the basic method.

A mental model for the three numbers

Three numbers per weight sit behind the penalty. It helps to read them as three questions.

1. How much did this weight help? This is the tally w_k. It is large when the weight moved in a direction that lowered the loss.
2. How far did it have to go to do that? This is the distance D_k. If a short move gave a large benefit, the weight is sensitive.
3. What is it worth to protect? This is Omega_k. It combines the first two answers into one stiffness.

Reading the numbers this way also makes debugging easier. If Omega is huge for a weight, ask whether the tally is large or the distance is tiny. The first suggests a real contribution. The second suggests a numerical artifact.

What the penalty does to the gradient

The gradient of the penalty for weight k is 2 * c * Omega_k * (theta_k - theta_ref_k). Two things follow.

• The pull is zero when the weight sits exactly at its reference value, so the penalty never blocks a weight from starting to move. It only grows as the weight moves away.
• The pull is linear in the distance, so small moves are cheap and large moves are expensive. This is the "elastic" behaviour shared with EWC.

A consequence is that the penalty does not freeze weights. It changes the cost of moving them. A new task with a strong enough signal can still override it. That is a feature, since sometimes old knowledge should be revised.

Key concepts and terms

A worked example

Take a model with two weights, a and b. Training on task A runs for three steps with a learning rate that gives the following changes. The numbers are made up to show the mechanics.

The tally

For weight a: w_a = -(-2.0 * 0.20) - (-1.0 * 0.10) - (-0.5 * 0.05) = 0.40 + 0.10 + 0.025 = 0.525.

For weight b: w_b = -(-0.10 * 0.01) - (-0.50 * 0.05) - (0.10 * -0.01) = 0.001 + 0.025 + 0.001 = 0.027.

So weight a is responsible for much more of the loss reduction.

The distance

Total movement: D_a = 0.20 + 0.10 + 0.05 = 0.35. D_b = 0.01 + 0.05 - 0.01 = 0.05.

The importance

Take xi = 0.1.

• Omega_a = 0.525 / (0.35^2 + 0.1) = 0.525 / (0.1225 + 0.1) = 0.525 / 0.2225 = about 2.36.
• Omega_b = 0.027 / (0.05^2 + 0.1) = 0.027 / (0.0025 + 0.1) = 0.027 / 0.1025 = about 0.26.

The first weight is about nine times as important as the second.

The penalty on task B

Suppose task B pulls both weights by +1.0 and c = 1. The penalty is 1 * (2.36 * 1 + 0.26 * 1) = 2.62, and the gradient pulling back on weight a is about nine times stronger than on weight b. The optimizer will move b more than a.

A PyTorch sketch

This code shows how the tally is kept during training. It is a teaching sketch. See PyTorch for the framework.

import torch
import torch.nn.functional as F

class SI:
    def __init__(self, model, c=0.1, xi=1e-3):
        self.model, self.c, self.xi = model, c, xi
        self.params = {n: p for n, p in model.named_parameters() if p.requires_grad}
        self.omega = {n: torch.zeros_like(p) for n, p in self.params.items()}
        self.ref = {n: p.detach().clone() for n, p in self.params.items()}
        self.start = {n: p.detach().clone() for n, p in self.params.items()}
        self.w = {n: torch.zeros_like(p) for n, p in self.params.items()}
        self.prev = {n: p.detach().clone() for n, p in self.params.items()}

    def penalty(self):
        total = 0.0
        for n, p in self.params.items():
            total = total + (self.omega[n] * (p - self.ref[n]) ** 2).sum()
        return self.c * total

    def after_step(self):
        # call right after optimizer.step(); gradients still hold the last
        # gradient of the TASK loss only (see note below)
        for n, p in self.params.items():
            delta = p.detach() - self.prev[n]
            self.w[n] += -p.grad.detach() * delta
            self.prev[n] = p.detach().clone()

    def end_task(self):
        for n, p in self.params.items():
            dist = p.detach() - self.start[n]
            self.omega[n] += self.w[n] / (dist ** 2 + self.xi)
            self.ref[n] = p.detach().clone()
            self.start[n] = p.detach().clone()
            self.prev[n] = p.detach().clone()
            self.w[n].zero_()

# training loop for one task
si = SI(model)
for x, y in loader:
    optimizer.zero_grad()
    task_loss = F.cross_entropy(model(x), y)
    task_loss.backward(retain_graph=True)
    task_grad = {n: p.grad.detach().clone() for n, p in si.params.items()}
    (si.penalty()).backward()
    optimizer.step()
    for n, p in si.params.items():
        p.grad = task_grad[n]       # tally uses the task gradient only
    si.after_step()
si.end_task()

The note in the code matters. The original method uses the gradient of the task loss alone for the tally, not the gradient of task loss plus penalty. Mixing them changes what the tally means.

A tabular flavour

For a tabular classifier trained on one time period and then a later one, SI would tally contributions during the first period, store Omega, and then penalize movement of high-Omega weights while fitting the later period. The check is the same as for any continual method: evaluate on held-out rows from both periods and compare with plain fine-tuning.

Strengths and limitations

Strengths

• Online. The tally is a by-product of training. No extra pass over old data is needed.
• No stored data. Only per-weight numbers and reference weights are kept.
• Local. Each weight needs only its own gradient and step, which suits simple implementations.
• Constant memory. The cost is a few extra vectors the size of the model, regardless of the number of tasks.
• No inference cost. The deployed model is unchanged.
• Easy to combine with replay, distillation and adapters.

Limitations

• Depends on the optimization path. Different learning rates or optimizers give different tallies. The importance is a property of the run, not only of the task.
• Noisy. Minibatch noise enters the tally.
• Assumes the loss falls. Contributions are defined through loss reduction. Weights that help by preventing loss increases may be undervalued.
• Needs task boundaries. The reset and consolidation happen at the end of a task.
• Struggles with class-incremental settings. Like other regularization methods, it does not protect output layers well when the model must pick from all classes seen so far.
• Diagonal view. Interactions between weights are ignored.
• Zero-distance edge case. A weight that moves out and back returns a small distance but may carry a large tally, which makes the ratio large and unstable. The constant xi is a patch, not a cure.
• Adaptive optimizers can distort the link between gradient and step, since the step is no longer proportional to the gradient.

Memory and compute cost

SI keeps about five vectors of model size: running tally, previous weights, start-of-task weights, reference weights and Omega. Some of these can be shared. The compute cost is a few multiplications per weight per step, which is small next to a forward and backward pass.

How it compares with alternatives

SI versus EWC in practice

Both are quadratic penalties with per-weight strengths, so they behave alike in many tests. Reports in the literature usually show them close to each other, with one or the other ahead depending on the benchmark and settings. The practical differences are these.

• SI is cheaper at the end of a task. EWC needs a pass over data.
• EWC has a clearer link to a probabilistic story. SI has a clearer link to the training path.
• EWC's importance is independent of the optimizer, to a first approximation. SI's is not.
• Both are weak in the hardest class-incremental setting.

It is reasonable to try both on your own problem. The code for either is short.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types such as numbers, categories, dates and text, missing values, and often no helpful column names. Models like TabPFN, TabICL and TabSTAR are pre-trained on large collections of synthetic or real tables so that they can predict on a new table through in-context learning, with the training rows passed in as context rather than used for gradient updates. The overall ideas are described in /model/research and /model/architecture.

Synaptic Intelligence is not part of in-context prediction. It enters when the base model itself is trained or updated over time. Here is how it connects.

Staged pre-training

A foundation model for tables may be trained in stages: first on synthetic tables sampled from a prior over data-generating processes, then on real tables, then on specialized families such as sensor logs or financial records. Each stage is like a task. SI's online tally can run during each stage at almost no cost, and the resulting Omega values can protect the shared weights when a later stage begins.

Updating without retraining from scratch

When new table families appear, retraining everything is expensive. SI gives a way to continue training from the current weights while limiting damage to what the model already does well. The price is a few extra vectors, not a replay buffer, which can matter when old tables cannot be kept because of size or governance.

Several heads on one trunk

Tabular systems often have several jobs that share a trunk:

• Prediction heads for classification and regression.
• Imputation networks trained with masked cell modeling to fill missing values.
• Scoring heads for anomaly detection.
• Semantic sector heads that infer the domain of a table from its content.

If one head is retrained, SI's importance values, accumulated from the losses of the other heads, can limit how far the trunk moves. This protects imputation quality while a sector classifier improves, for example.

Why path-based importance suits table encoders

A table encoder must handle mixed types and missing values. Different parts of the network specialize: some for numeric scaling, some for category identity, some for missing-value patterns. During training, the weights that carry these skills show large contributions to loss reduction. SI's tally records this without any extra pass. Because the tally is built from what the weights actually did, it follows the specialization the model developed.

No column names

When column names are absent or unreliable, the model infers meaning from value patterns. These learned patterns are spread across many weights and are easy to disturb. A penalty that tracks which weights were useful during learning can help to keep them, but this needs to be checked on a fixed board of old table families after each update, as in /model/benchmarks.

In-context learning and continual learning together

In-context learning adapts to one table with no weight change, so it cannot forget. SI is for the slower loop, in which the base weights improve over months. The two work at different time scales.

• In-context learning: fast, per table, no forgetting.
• Weight-level continual learning with SI: slow, across table families, forgetting possible and managed.

Memory modules

Another route to continual behaviour is to give a sequence model a learned memory that is updated while reading, as described in MIRAS. Such a memory is complementary to SI. One manages fast state inside a forward pass. The other protects slow weights between training runs.

Caution

Claims that SI helps a particular tabular foundation model should be tested. Results depend on the shift between stages, the optimizer and the strength constant. A fair test fixes the old-task board and the new-task board in advance and reports both.

Practical guidance and best practices

Setup

• Decide the setting first: task-incremental, domain-incremental or class-incremental. SI is a better fit for the first two.
• Fix an evaluation board of old tasks before training on new ones.
• Keep baselines: plain fine-tuning, and joint training if affordable.

Optimizer and learning rate

• SI's tally assumes steps relate to gradients. Plain SGD with momentum matches this best.
• With adaptive optimizers, results can still be good but the tally is less principled. Tune carefully and compare with a simpler optimizer.
• Keep the learning rate stable within a task. Large schedule changes alter the path.

Choosing c and xi

• Sweep c on a log scale and look at old-task and new-task results together.
• Choose xi large enough to avoid blow-ups from tiny distances, but small enough not to flatten differences. Values that are small relative to typical squared distances are a common start.
• Re-tune when model size, optimizer or loss scale changes.

Implementation details that matter

• Use the gradient of the task loss only for the tally, not of task loss plus penalty.
• Compute the tally in full precision. If you use mixed precision, keep the tally vectors in float32.
• Handle weight decay with care. Decay changes weights without a gradient from the loss, so it adds movement not accounted for in the tally.
• Exclude batch-norm running statistics. They are not weights and are not covered.

Combining with other methods

• Add a small replay buffer if any data can be kept.
• Add distillation, see LwF, to protect outputs directly.
• Use per-task adapters or LoRA when tasks differ a lot.

Evaluating

• Report average accuracy, forgetting, and backward transfer after each task.
• Run at least three seeds and report the spread.
• Repeat with a shuffled task order.
• For tables, use metrics that fit the job, such as AUC or log loss for classification and error on masked cells for imputation.

Monitoring during a run

• Log the task loss and penalty separately.
• Watch the largest Omega values. A handful of huge values often point to a numerical problem.
• Plot the distance from reference for the top-Omega weights. They should stay small.

Troubleshooting a weak result

If forgetting remains high, work through these checks.

1. Print the Omega statistics. Look at the mean, median and maximum. If they are tiny, raise c. If a few values dominate, check for tiny distances and raise xi.
2. Check the tally source. Confirm that the tally uses only the task gradient.
3. Check the reference. The reference weights must be updated at the end of each task, and not during it.
4. Check the optimizer state. Reset momentum and averages at task boundaries.
5. Look at the head. If most of the loss is in the output layer, the body penalty is not the issue.
6. Compare with joint training. If a jointly trained model is also weak, the capacity or the tasks are the problem.
7. Try a replay buffer of even a few examples per class. If forgetting drops sharply, the setting is likely class-incremental and needs replay or distillation.

When to prefer something else

• When task boundaries are not known, replay is easier to run.
• When old tasks must be kept exactly, keep separate adapters or models.
• When new inputs resemble old inputs, LwF is simple and strong.
• When the network is very large and pre-trained, parameter-efficient tuning such as LoRA often wins on cost.

Common mistakes

• Using the combined gradient for the tally. This mixes the penalty into the importance and gives a distorted signal.
• Forgetting to reset the tally at task boundaries. The next task then inherits a contribution it did not earn.
• Setting xi to zero. Weights that do not move produce infinite or huge importance.
• Ignoring weight decay. It moves weights without a loss gradient and breaks the accounting.
• Judging success on the new task only. The goal is the old tasks. Always report both.
• Comparing against a poorly tuned baseline. Give plain fine-tuning and replay the same tuning effort.
• Expecting SI to fix class-incremental learning. The output layer needs separate care.
• Changing the optimizer between tasks without noting that it changes what the tally means.
• Assuming importance is a property of the task. It is partly a property of the run, so repeat with several seeds.
• Skipping the old-task board after small updates. Forgetting can creep in over many small steps.

Frequently asked questions

What is Synaptic Intelligence?

Synaptic Intelligence is a continual learning method that tracks, during training, how much each weight contributed to reducing the loss. It then uses those contributions as strengths in a penalty that discourages changing important weights while learning new tasks.

How is Synaptic Intelligence different from EWC?

Both add a quadratic penalty with per-weight strengths. EWC computes the strengths after each task using the Fisher information, which needs an extra pass over data. Synaptic Intelligence accumulates them online from the training path, so no extra pass is needed.

Does Synaptic Intelligence need old data?

No. It keeps per-weight numbers and the reference weights from the previous task. It does not store or revisit old inputs or labels.

Why is there a small constant xi in the formula?

The importance divides by the squared distance a weight moved. If a weight barely moved, that distance is close to zero and the ratio would explode. The constant keeps the division stable.

Does it work with any optimizer?

It can be run with any optimizer, but the tally is most meaningful when steps are tied closely to gradients, as in stochastic gradient descent. With adaptive optimizers the link is looser, so results should be checked more carefully.

Can Synaptic Intelligence stop forgetting completely?

No. It lowers forgetting, but it uses a diagonal view of importance, depends on the training path and cannot create capacity the network lacks. In hard class-incremental settings the remaining loss can be large.

Is it suitable for tabular foundation models?

It can be used when the base model is trained in stages or updated with new table families, because it adds little cost and needs no old data. In-context prediction on a new table changes no weights, so it needs no such protection. Any benefit should be verified on a fixed board of old table families.

How do I tune the strength constant?

Sweep it on a log scale and examine old-task and new-task results together. Choose a value that keeps old-task performance near its earlier level without blocking new learning, and repeat the sweep when the model or optimizer changes.

Should I combine it with replay?

If any data can be kept, yes. A small buffer protects behaviour directly and the penalty protects the weights that mattered, and the two together often do better than either one alone.

What is the main practical advantage over EWC?

The main advantage is that the importance comes almost for free during training, so there is no separate estimation step. The main trade-off is that the scores depend on the optimization path and on noise in the gradients.

Glossary

• Catastrophic forgetting: the sharp loss of old-task performance after training on new data.
• Consolidation: the step at the end of a task that turns the tally into importance values.
• Continual learning: learning from a stream of tasks without retraining from scratch.
• Contribution: the part of the loss reduction due to one weight.
• Fisher information: a measure of output sensitivity used by EWC.
• Importance (Omega): the per-weight strength in the penalty.
• Online estimation: computing a quantity during training rather than afterward.
• Path integral: a sum of small contributions along a route.
• Reference weights: weights saved at the end of the previous task.
• Regularization: adding a term to the loss to guide the solution.
• Stability and plasticity: keeping the old and learning the new.
• Task boundary: the moment one task ends and the next begins.
• Xi: a small constant that prevents division by zero.

Summary

Synaptic Intelligence protects earlier learning by keeping a running tally of how much each weight helped reduce the loss during training, turning that tally into an importance value, and penalizing movement of important weights on later tasks. Its main appeal is that the importance is gathered online at almost no cost, with no stored data and no change at inference. Its main weaknesses are dependence on the optimization path, gradient noise, the need for task boundaries and weak results when classes must be separated across tasks. For tabular foundation models it is most relevant when the base model is trained in stages or updated with new table families, where it can help keep skills such as imputation, anomaly scoring and semantic classification stable. Good practice is to use the task-loss gradient for the tally, tune c and xi on a log scale, evaluate old and new tasks together, and consider adding a small replay buffer. Related reading includes EWC, LwF, Experience Replay and /model/benchmarks.

Related

It belongs to Continual learning in my stack, under Foundation models and applied ML. Others in the same group: