MIRAS

Overview

MIRAS is a research framework for describing and designing memory in sequence models. Its central claim is that many modern architectures, including attention layers, linear recurrent models and newer neural memory modules, can be seen as associative memories that learn while they read. Each of them can be described by a small set of design choices: what the memory looks like, what error it tries to reduce when it stores something (the attentional bias), how it decides what to forget (the retention gate), and how it updates itself (the learning rule).

This view is useful for two reasons. It turns a long list of separate models into points in one design space, so that you can compare them and invent new ones by changing a single choice. It also connects sequence modelling to continual learning, because a memory that updates itself at test time has to balance learning the new against keeping the old. This article explains the idea in plain words, gives a numeric and code example, compares MIRAS with related ideas such as EWC and Experience Replay, and then shows how the framework relates to models that learn from raw tables.

What it is

MIRAS is best thought of as a lens, not as a single model. It gives a vocabulary for talking about the memory inside a sequence model. A sequence model reads a stream of items one after another, such as tokens in a sentence, rows in a table or readings from a sensor. At each step it must decide what to keep from what it has seen. The part of the model that does this is its memory.

The framework describes that memory with four choices.

1. Memory architecture. What is the memory made of? It could be a single matrix, a vector, or a small neural network.
2. Attentional bias. What internal objective does the memory use to decide how well it has stored an item? It is the notion of "match" or "error" used when the memory is written to.
3. Retention gate. How does the memory forget? This is a regularization step that keeps the memory from growing without bound or from straying too far from its past state.
4. Memory learning algorithm. How is the memory updated? For example, a plain gradient step, a step with momentum, or a step with an adaptive rule.

The everyday picture

Imagine a notebook that you write in while listening to a long talk. You decide what to write (the attentional bias: what counts as worth noting). You decide how to organize the notebook (the memory architecture: a list, a table, a mind map). You decide what to cross out when pages run low (the retention gate). And you have a habit for how you update it (the learning algorithm: rewrite as you go, or take notes in short bursts).

Two people at the same talk can make different choices on each of those four points and end with very different notebooks. MIRAS says the same about sequence models.

What "associative memory" means

An associative memory stores links from a cue to a response. You see a key and you want to recall the value attached to it. A phone book maps names to numbers. A sequence model's memory maps keys, built from the current input, to values, also built from the input. Reading means giving a query and getting back a blend of stored values.

Learning at test time

A central phrase in this line of work is test-time memorization. The memory is updated while the model processes a new input, even though the main weights of the model are fixed. So the model keeps learning during use, but only inside the memory. The main weights are trained in advance. The memory state is built on the fly from the sequence it reads.

What it does not do

• It is not a single trained model that you download and use.
• It does not replace pre-training of the main weights.
• It does not remove the need for evaluation on long inputs.
• It does not by itself give perfect long-term memory. Memory has finite capacity.

Where it came from

The roots of MIRAS go back several decades.

• Associative memories such as the early associative memory networks of the 1980s stored patterns as links between units.
• Fast weights were proposed as a way to give networks a short-term memory that changes quickly, next to slowly learned long-term weights.
• The delta rule from early neural network research updates a mapping by reducing a squared error on each new pair.
• Linear attention (around 2020) showed that attention can be written as a recurrent update to a matrix memory, which makes long sequences cheaper.
• Gated linear recurrent models and state space models, see Mamba, added decay and input-dependent gates to those memories. The older state space line, covered in S4, laid the groundwork.
• Test-time training layers and Titans (2024 and 2025) made the memory itself a small neural network updated by gradient steps while the sequence is read.

MIRAS, introduced in a 2025 paper titled "It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization", pulled these threads together. It argued that a wide range of architectures are special cases of one recipe with the four choices listed above. It also showed that by swapping parts of the recipe one can obtain new variants. The paper presented several example models built this way, with different choices for the attentional bias and the retention gate.

The older background

The framework sits on top of ideas from transformers, in which attention is a kind of lookup in a store of keys and values, and from LSTM and GRU networks, whose gates decide what to keep and what to erase. MIRAS shows that those gating ideas and the attention idea are close relatives.

Relation to continual learning

The "retention" in the name echoes continual learning. A memory that updates at test time faces the same trade-off as a network trained on a stream of tasks: stability against plasticity. Methods like EWC, Synaptic Intelligence and LwF address it in weight space. MIRAS addresses it inside the memory state. The model families are different, but the question is the same.

How it works

The framework is easiest to follow by walking through one memory update, then looking at how each of the four choices shapes it.

Step 1: Turn the input into a key and a value

At each step the model reads an item, such as a token or a table row. Small learned projections turn it into a key k_t and a value v_t. A third projection gives a query q_t. These projections are part of the main weights and do not change at test time.

Think of a librarian receiving a book. The key is the shelf label, the value is the content summary. The query is the label that a reader later uses to ask for books.

Step 2: Read from memory

The model uses the query to read: y_t = M(q_t). For a matrix memory this is a matrix-vector product. For a neural memory it is a forward pass through the small network.

Step 3: Ask how well the memory already knows this pair

The memory is asked to reproduce the value from the key: M(k_t) should be close to v_t. The measure of closeness is the attentional bias. It is an objective function loss(M(k_t), v_t).

Different choices of this objective lead to different behaviours.

• A dot-product objective rewards simply adding the value along the key direction. It is the logic behind plain linear attention.
• A squared error objective rewards fixing the mistake: it only writes what the memory does not already know. This is the delta rule.
• A robust objective, such as one based on the Huber loss or an absolute error, limits the effect of outliers, so a single odd item does not overwrite the memory.
• Other p-norm objectives give different sensitivity to large and small errors.

The name "attentional bias" reflects that this objective determines what the memory pays attention to when storing.

Step 4: Update the memory

The memory is moved in the direction that lowers the loss, and a forgetting term is applied. In general form:

M_t = Retain( M_{t-1} )  -  step_size * gradient of loss(M_{t-1}(k_t), v_t)

The retention gate is the Retain part. The learning algorithm is how the second term is computed and scaled.

Step 5: Repeat for the whole sequence

The steps run once per item. After the last item, the memory holds a compressed trace of the sequence. Because the memory has fixed size, the cost per item stays the same however long the sequence is. That is the main efficiency appeal, compared with attention, whose cost per item grows with the number of past items.

The four choices in detail

Choice 1: Memory architecture

• Vector memory keeps one state vector. It is small but can hold little.
• Matrix memory keeps a matrix that stores key-value links. This is the form used by linear attention and many gated recurrent models.
• Deep memory uses a small multi-layer network as the memory. It can store more complex relationships and is more expressive. It costs more compute per step.

A helpful analogy is storage furniture: a single hook, a cabinet with drawers, or a whole filing room with its own indexing.

Choice 2: Attentional bias

As covered above, this is the internal objective. It decides what counts as a good match. Changing it changes how the memory treats repeated items, outliers and noise.

Choice 3: Retention gate

The gate decides how much of the old memory to keep. Options include:

• No forgetting. The memory only accumulates. It saturates on long inputs.
• Scalar decay. Multiply the memory by a number below one at each step, possibly depending on the input.
• Regularization toward the past state. Penalize large changes, so the memory is pulled back toward where it was. This is close in spirit to the weight penalties in EWC.
• Sparsity or other penalties. Encourage the memory to stay compact.

In the framework, forgetting is seen as a regularizer on the memory's own learning problem. That is a neat link between gating in recurrent models and regularization in continual learning.

Choice 4: Memory learning algorithm

• Plain gradient step. Move once along the negative gradient.
• Gradient step with momentum. Keep a running average of recent gradients so that surprising items continue to influence the next few updates.
• Adaptive or higher-order steps. Use richer update rules at higher cost.
• Chunked updates. Process several items at once, which suits modern hardware better than one item at a time. See PyTorch for typical implementation tools.

Putting it together

The table is a simplification. Real models differ in details, but the grid shows the point of the framework: each row is a setting of four dials.

A note on surprise

Several neural memory designs use a measure of surprise to decide what to store. An item that the memory predicts badly produces a large gradient, so it changes the memory a lot. An item that is already well predicted produces a small gradient, so it changes the memory little. This is automatic from the squared-error choice. People use the same intuition: you remember the unexpected thing in a routine.

Seeing known models through the framework

The framework earns its keep when you take a familiar model and read off its four dials.

• Plain linear attention. Matrix memory, dot-product bias, no forgetting, simple additive update. It over-counts repeats and saturates on long inputs.
• Delta-rule recurrent models. Matrix memory, squared-error bias, a step that corrects only the error. They avoid over-counting and handle repeats well.
• Gated models. Add an input-dependent decay on top, so the model can choose to forget at some steps and hold on at others. See Mamba for a well-known gated family.
• Deep neural memories. Replace the matrix with a small network, keep the squared-error bias, add momentum and weight decay. The memory can capture richer links at higher cost.
• Robust variants. Swap the bias for a clipped or absolute error, so outliers have a bounded effect.

Reading models this way also shows gaps. If no existing model uses a given combination, such as a robust bias with a deep memory and a strong retention gate, that combination is a candidate for a new design.

Why forgetting is a feature

It can seem odd that a good memory should forget. The reason is capacity. A fixed-size memory must make room. If every link is kept forever, new links collide with old ones and recall gets worse for everything. Forgetting the least useful links keeps the rest sharp.

An everyday example is a desk. If you never throw anything away, you cannot find anything. If you throw away too much, you miss what you need. A good retention gate is like a tidy habit: keep what is used often, let go of what is not.

How momentum helps

Without momentum, each item changes the memory only by its own error. With momentum, a surprising item keeps influencing the next few updates, because the running average of gradients decays slowly. This helps when surprising events come in bursts, such as a regime change in a sensor stream. It can hurt when the stream is noisy, since noise is also carried forward. Tuning the momentum factor is part of tuning the learning algorithm dial.

Chunking for speed

Updating the memory one item at a time is slow on parallel hardware. A common remedy is to split the sequence into chunks. Within a chunk, all updates are computed together using the memory state from the start of the chunk. Between chunks, the state is passed along. This trades a little exactness for a lot of speed. A good test is to compare chunked and item-by-item results on a small input and confirm that they agree closely.

Key concepts and terms

A worked example

A small numeric example shows how the choice of attentional bias changes what is stored. We use a matrix memory with two-dimensional keys and values.

Setup

The memory M starts as a 2 by 2 matrix of zeros. Reading is M @ k. Items arrive as key-value pairs.

• Item 1: k1 = (1, 0), v1 = (2, 0).
• Item 2: k2 = (1, 0), v2 = (2, 0). The same pair repeats.

Dot-product bias, no forgetting (linear attention)

The update is M = M + v k^T.

• After item 1: M = [[2, 0], [0, 0]]. Reading with k1 gives (2, 0), which is right.
• After item 2: M = [[4, 0], [0, 0]]. Reading with k1 gives (4, 0), which is twice the true value.

The memory over-counts repeated items. It has no notion of "I already know this".

Squared-error bias (delta rule), step size 1

The update is M = M + (v - M k) k^T.

• After item 1: M k1 = (0, 0), error (2, 0), so M = [[2, 0], [0, 0]]. Reading gives (2, 0).
• After item 2: M k2 = (2, 0), error (0, 0), so M does not change. Reading still gives (2, 0).

The memory ignores what it already knows. It writes only the surprise. That is the effect of the attentional bias.

Adding a retention gate

Add a decay of 0.9 on M before each update. With the delta rule:

• After item 1: M = 0.9 * 0 + correction, so M = [[2, 0], [0, 0]].
• After item 2: decayed M = [[1.8, 0], [0, 0]], then error v - M k = (0.2, 0), so M goes back to [[2, 0], [0, 0]].

Forgetting is balanced by correction, so the memory stays near the right value. Without new reminders, though, old links would fade by 10 percent per step, which is the cost of forgetting.

Code sketch

The code below runs the three cases above for any sequence. It is a teaching sketch.

import numpy as np

def run_memory(pairs, mode="delta", decay=1.0, lr=1.0, dim=2):
    M = np.zeros((dim, dim))
    for k, v in pairs:
        k = np.asarray(k, float); v = np.asarray(v, float)
        M = decay * M                       # retention gate
        if mode == "linear":                # dot-product bias
            M = M + lr * np.outer(v, k)
        elif mode == "delta":               # squared-error bias
            err = v - M @ k
            M = M + lr * np.outer(err, k)
        elif mode == "huber":               # robust bias (clip the error)
            err = np.clip(v - M @ k, -1.0, 1.0)
            M = M + lr * np.outer(err, k)
    return M

pairs = [((1, 0), (2, 0)), ((1, 0), (2, 0))]
print(run_memory(pairs, "linear"))                # over-counts
print(run_memory(pairs, "delta"))                 # stable
print(run_memory(pairs, "delta", decay=0.9))      # stable with forgetting
print(run_memory(pairs, "huber"))                 # limits big corrections

What the robust version does

In the Huber-style case the correction is clipped. A single wild value, such as an outlier measurement, can move the memory only by a bounded amount. For noisy streams, such as sensor logs or tables with data entry errors, this is a useful property.

A tabular flavour

Consider a stream of table rows where each row has a key made from some columns and a value made from others. A matrix memory with the delta rule would learn the link between them as rows arrive. Repeated rows would not distort it. A new pattern would be written strongly at first, then settle. A row with a wild outlier would be written weakly under a robust bias. The same reading step would then recall values for new keys by blending stored links.

Strengths and limitations

Strengths

• Unifying view. Many architectures become settings of four dials, which helps comparison and design.
• Constant memory per step. Fixed-size memories give linear cost in sequence length.
• Online adaptation. The model adapts inside the sequence, without changing main weights.
• Controlled forgetting. The retention gate gives an explicit and tunable way to forget.
• Robustness options. Choosing a robust attentional bias reduces the effect of outliers.
• Natural link to continual learning. The same stability and plasticity trade-off is handled in a principled way.

Limitations

• Finite capacity. A fixed-size memory cannot hold everything. Details of a long input can be lost.
• Exact recall is hard. Attention can look up any past item exactly, while compressed memories can blur similar items.
• Compute for deep memories. A neural memory updated by gradient steps costs more per item than a matrix memory.
• Hardware efficiency. Per-item updates are sequential. Chunked methods help but add complexity.
• Many choices to tune. Four dials mean a large design space and more experiments.
• Evidence varies. Gains reported for one task or length do not automatically carry over to another. Careful evaluation on long inputs is needed.
• It is a framework, not a recipe. It tells you how to describe designs, not which one is best for your data.

Capacity in plain terms

A matrix memory of size d by d can hold on the order of d independent links before they start to interfere. Past that point, new links blur old ones. Forgetting makes room, but it also erases. Deeper memories hold more, at higher cost. This is why memory choice has to match the length and richness of the input.

How it compares with alternatives

MIRAS versus weight-space continual learning

The two are complementary. One gives slow, durable learning. The other gives fast, temporary learning.

MIRAS versus replay

Experience Replay stores real examples and trains on them again. A memory module stores a compressed summary and reads from it. Replay is exact but needs storage. A module is compact but lossy.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types such as numbers, categories, dates and text, missing values and often no helpful column names. Models like TabPFN, TabICL and TabSTAR are pre-trained on large numbers of synthetic or real tables so that they can predict for a new table through in-context learning: the labelled rows of the new table are given as context and the model predicts in a single forward pass without weight updates. The wider design is described in /model/research and /model/architecture.

MIRAS relates to these models in several concrete ways.

In-context learning is a memory problem

When a tabular model reads the labelled rows of a new table and predicts for new rows, it must store information about the training rows and recall it for each test row. That is an associative memory task. The keys are the features of a row, the values are the labels or the row content, and the query is a test row.

Attention over the rows is one way to do this. Its cost grows quickly with the number of rows, which limits the size of tables that can be used as context. A fixed-size memory with a good attentional bias and retention gate is an alternative that scales linearly in rows. The MIRAS view tells you which dials to turn.

Rows as a sequence, with no natural order

Tables are sets of rows with no inherent order. A sequence memory reads rows one after another, so the result can depend on the order. Designs must either make the memory approximately order-invariant, for example by choosing an update rule that commutes well, or average over several random orders. The attentional bias matters here. A squared-error bias writes only what is new, which reduces the effect of repeated rows and makes the order matter less.

Mixed types and missing values

Rows contain numbers, categories and gaps. The key and value projections must turn these into vectors. Missing cells can be given a learned marker, or a mask can be passed. A robust attentional bias helps, since a row with a wild or corrupted value then moves the memory by a bounded amount.

No column names

Without names, the model relies on values and column statistics. The memory then links patterns of values to outcomes, not names to meanings. This is what associative memory does naturally: it needs a key and a value, and the key can be built from the raw values.

Pre-training on synthetic or real tables

During pre-training, the projections and the learning rule parameters, such as step sizes and gate values, are trained across many tables. At test time they stay fixed, and the memory state adapts to the new table. This split is the same one that the framework describes: slow weights learned in advance, fast memory built while reading.

Continual learning

Tabular foundation models face a continual learning problem at two scales.

• Within one table, where the memory must balance learning new rows with keeping earlier ones. The retention gate handles this.
• Across table families over time, where the base model is updated. Here weight-space methods such as EWC, Synaptic Intelligence and LwF, or Experience Replay, apply.

MIRAS addresses the first. The others address the second. A full system may use both.

Imputation

Filling missing cells is a recall task: given the observed cells of a row as a query, recall the likely value of the missing cell from stored rows. A memory trained with a squared-error bias is a natural fit. See imputation networks and masked cell modeling for the training signals typically used.

Anomaly detection

Surprise is built in. A row that the memory predicts badly produces a large error, which is also a natural anomaly score. This links the framework to anomaly detection. The memory stores what is normal and flags what it cannot reproduce. A robust bias helps keep a few anomalies from corrupting the notion of normal.

Sector and semantic classification

Inferring the domain of a table, as semantic sector heads do, requires summarizing the whole table into a representation. A memory that has read all rows, with a retention gate that keeps the recurring patterns and forgets the noise, gives a compact summary that a classification head can use.

A cautious note

These links are conceptual. Whether a particular memory design improves a particular tabular model has to be shown by experiment on a fixed set of tables with a range of sizes, types and missing rates, as in /model/benchmarks. Claims of improvement should state the sizes of tables used.

Practical guidance and best practices

Start from the question

• If your inputs are short, attention is simple and strong. Use a memory module only when length or cost forces it.
• If your inputs are long and streaming, a fixed-size memory is attractive.
• If outliers are common, consider a robust attentional bias.
• If you need exact recall of specific items, keep some attention or retrieval alongside the memory.

Choose the four dials deliberately

Evaluate on the right tests

• Test on inputs longer than the training length.
• Test recall of specific items at different distances.
• Test with distractors, noise and outliers.
• Test with repeated items to check for over-counting.
• Report cost: time and memory per item.

Implementation details

• Normalize keys so that update sizes are predictable.
• Keep gate values in a safe range, for example between zero and one.
• Use mixed precision with care. Memory updates can accumulate rounding errors, so keep the memory state in higher precision.
• Use chunked updates to match hardware, and verify that they match the item-by-item result.
• Save and restore memory state explicitly when processing long streams in pieces.

For tabular settings

• Make the model insensitive to row order, or average over several random orders.
• Provide a clear way to represent missing cells.
• Scale numeric columns robustly before projecting.
• Test on tables of different sizes, from small to large.
• Combine with a held-out board of table families to check for drift after updates.

Combining with other methods

• Use a weight penalty or replay for the slow base model, and the memory for the fast adaptation.
• Use distillation, see LwF, if you want a new memory design to match the behaviour of an old one.
• Use self-supervised learning signals, such as masked cells, to pre-train the projections.

Troubleshooting a memory that does not behave

1. Recall is poor even on short inputs. Check the key normalization and the step size. Unnormalized keys make updates too large or too small.
2. Recall is good at first and then collapses. The memory is saturating. Add or strengthen a retention gate.
3. Recall is good only for recent items. The gate is too aggressive. Lower the decay or make it input-dependent.
4. Repeated items distort the output. The bias is of the dot-product kind. Try a squared-error bias.
5. A few outliers wreck the memory. Try a robust bias or clip the error.
6. Results change when the order changes. Average over several orders, or use an update rule that is less order-sensitive.
7. Training is unstable. Reduce the step size, clip gradients, keep the memory in higher precision.
8. Speed is poor. Switch to chunked updates and verify that results still match.

When to prefer something else

• If inputs are short and exact recall matters, plain attention is simpler and strong.
• If you must remember specific facts from a very large store, add retrieval, not only a compressed memory.
• If the problem is across-task drift of the base model, use a weight-space method such as EWC or a replay buffer.
• If the task is a standard small table prediction, a pre-trained in-context model such as TabPFN may be enough without any custom memory.

Common mistakes

• Treating MIRAS as a model to download. It is a framework for describing and designing memory.
• No forgetting on long inputs. Without a retention gate, the memory saturates and blurs.
• Too much forgetting. Aggressive decay erases information that was still needed.
• Over-counting repeats. A dot-product bias adds repeated items again and again.
• Ignoring outliers. A squared-error bias can be dragged by a single extreme value. Consider a robust bias.
• Evaluating only on short inputs. Memory problems appear at length.
• Comparing cost unfairly. Deep memory costs more per item than a matrix, and this should be reported.
• Order sensitivity in tables. Rows have no natural order, and the result should not depend on it.
• Confusing fast memory with slow learning. A memory that adapts inside one input does not replace updating the base weights across tasks.
• Claiming a gain from one setting. Check across lengths, noise levels and table types.

Frequently asked questions

What is MIRAS?

MIRAS is a research framework that describes the memory inside sequence models through four choices: the memory architecture, the attentional bias, the retention gate and the memory learning algorithm. It shows that many known models are special cases and suggests new ones.

What does attentional bias mean in this context?

It is the internal objective a memory uses to decide how well it has stored a key-value pair. Different objectives, such as a dot product, a squared error or a robust loss, lead to different behaviour for repeated items, new items and outliers.

What is a retention gate?

The retention gate is the part of the update that decides how much of the old memory to keep. It can be a simple decay, an input-dependent gate or a penalty that pulls the memory toward its past state, and it prevents the memory from saturating.

Is MIRAS the same as a transformer?

No. A transformer keeps all past keys and values and attends to them, which is exact but grows in cost with length. MIRAS describes fixed-size or compressed memories, and it shows how attention and recurrent memories are related.

How does MIRAS relate to continual learning?

A memory that updates itself while reading faces the same stability and plasticity trade-off as a network trained on a stream of tasks. The retention gate plays the role that a weight penalty plays in methods such as EWC, but inside the memory rather than in the main weights.

Does MIRAS update the model weights during use?

No. The main weights stay fixed. Only the memory state changes as the input is read, which is why the idea is called test-time memorization.

Can MIRAS ideas be used for tabular data?

Yes, in principle. Reading labelled rows and recalling values for new rows is an associative memory task, and missing values, outliers and anomalies map well onto the choices of attentional bias and retention. The benefit for a given model must be tested on tables of varied size and type.

How is it different from in-context learning with attention?

In-context learning with attention keeps every context item and looks them up exactly, at a cost that grows with the number of items. A MIRAS-style memory compresses the context into a fixed-size state, which costs less for long contexts but can lose detail.

Does a deeper memory always work better?

No. A deeper memory can store more complex relationships, but it costs more per item and is harder to train and tune. The right depth depends on how much must be remembered and how much compute is available.

Where should I start if I want to try these ideas?

Start with a simple matrix memory and a squared-error bias, add a scalar retention gate, and test on long inputs with repeated items and outliers. Then change one dial at a time and measure the effect, keeping cost and recall quality in view.

Glossary

• Associative memory: a store of links from keys to values.
• Attentional bias: the objective that defines what counts as a good match when writing to memory.
• Context length: the number of past items a model can use.
• Delta rule: an update that corrects the memory by the error between recalled and true values.
• Fast weights: parameters that change quickly and act as short-term memory.
• Gate: a learned value that controls how much information passes or is kept.
• Huber loss: a loss that is quadratic for small errors and linear for large ones, which limits the effect of outliers.
• Key, value, query: three views of an input used for storing and reading.
• Linear attention: attention written as a fixed-size memory update.
• Momentum: a running average of recent updates.
• Retention gate: the rule that decides what the memory forgets.
• Surprise: how badly the memory predicted an item.
• Test-time memorization: updating a memory while reading an input, with fixed main weights.

Summary

MIRAS describes the memory in a sequence model by four choices: the memory architecture, the attentional bias, the retention gate and the memory learning algorithm. This turns attention, linear recurrent models and neural memory modules into points in one design space and suggests new variants by changing one dial at a time. Its strengths are a unifying view, constant cost per item for fixed-size memories, and explicit control of forgetting. Its limits are finite capacity, weaker exact recall than full attention and a large design space. For tabular foundation models, the framework fits in-context learning on a new table, which is an associative memory task, and connects naturally to imputation through recall, to anomaly detection through surprise, and to semantic classification through compact summaries. It complements weight-space continual learning such as EWC, Synaptic Intelligence, LwF and Experience Replay, which work on a slower time scale. For related reading see /model/architecture and /model/benchmarks.

Related

It belongs to Continual learning in my stack, under Foundation models and applied ML. Others in the same group: