Experience replay

Overview

Experience replay is a training technique in which a learner stores past examples in a memory, called a replay buffer, and keeps training on random samples from that memory while it also learns from new data. It was made famous in reinforcement learning, where an agent reuses old transitions instead of throwing them away after one update. It is now also one of the most reliable ways to fight catastrophic forgetting in continual learning, where a model must learn new tasks without losing old skills.

The idea is simple and the effect is large. Learning from a stream of data, one example after another, gives the model a very narrow and correlated view of the world. Replay breaks that correlation and keeps old knowledge alive. In practice, a small buffer often recovers most of the performance that a model would otherwise lose. For models that learn from raw tables, replay is the plain and strong baseline for adding new datasets over time, and it sits next to regularization methods such as EWC and LwF. See also /model/research for how continual learning fits into tabular model research.

What it is

Experience replay has three parts.

• A buffer. A fixed-size store that holds past examples. In reinforcement learning an example is a transition: state, action, reward, next state, and a done flag. In supervised continual learning an example is an input and its label, and sometimes the model's old output for that input.
• A write rule. A policy for what goes into the buffer and what gets thrown out when it is full. The simplest rule is first-in, first-out. A better rule for streams is reservoir sampling.
• A read rule. A policy for how to sample from the buffer when forming a training batch. The simplest rule is uniform random sampling. Smarter rules weight examples by how surprising or how useful they are.

Every training step mixes fresh data with replayed data. The model sees both in the same batch, so the gradient pulls toward solving the new task and the old ones together.

Two families of use

The two families use the same mechanism but for different reasons. In reinforcement learning, replay mainly improves stability and sample efficiency. In continual learning, replay mainly prevents forgetting.

An everyday analogy

Think of a student preparing for exams. If the student only studies the newest chapter, the older chapters fade. A good student keeps a stack of flash cards from every chapter and shuffles a few old cards into each new study session. The flash card stack is the replay buffer. The shuffle is the sampling rule. The student does not need every card from every chapter. A well chosen small stack is enough to keep the old chapters fresh.

Where it came from

The idea of replaying stored experience goes back to early work on reinforcement learning in the early 1990s. The first clear description of an experience replay mechanism for neural network based agents came in 1992. The motivation was practical: interaction with the world is slow and costly, so each experience should be used many times.

The technique became widely known after the deep Q network work of 2013 and 2015, which combined a convolutional network with a replay buffer and a separate target network to learn to play many video games from raw pixels. That work showed that replay was a key part of making deep reinforcement learning stable. You can read about the model in DQN, and about later refinements in Double DQN, Dueling DQN and Rainbow. Many other off-policy algorithms rely on a replay buffer too, including DDPG, TD3 and SAC.

Several important extensions followed.

• Prioritized experience replay (2015). Instead of sampling uniformly, sample transitions with probability that grows with their temporal difference error, and correct the bias with importance weights.
• Hindsight experience replay (2017). Relabel failed attempts with the goal that was actually reached, so that sparse reward tasks yield useful learning signals.
• Rehearsal for continual learning. Work in the late 2010s showed that a small episodic memory of past samples is a strong defence against forgetting in classification. Methods such as iCaRL (2017) and gradient episodic memory (2017) used stored exemplars, either for rehearsal or to constrain gradients.
• Generative replay (2017). Instead of storing real samples, train a generative model to produce pseudo samples of old tasks, then replay those.
• Dark experience replay (2020). Store the logits the model produced in the past, and use them as distillation targets during replay.

Later surveys and benchmarks of continual learning repeatedly found that simple replay is hard to beat, especially when compute and memory budgets are comparable. This is the main reason the technique is described as the strongest simple defence against forgetting.

How it works

Step 1: collect or receive data

In reinforcement learning, the agent acts in an environment and produces transitions. In continual learning, a data stream delivers a new task or a new dataset. Either way, a chunk of new data arrives.

Step 2: write to the buffer

Each new example is considered for the buffer. If the buffer has room, the example is added. If the buffer is full, the write rule decides which stored example to evict.

Common write rules:

• First in, first out. The oldest example leaves. This is fine in reinforcement learning, where old experience becomes less relevant as the policy changes. It is poor for continual learning, because the earliest tasks vanish first.
• Reservoir sampling. Every example seen so far has the same chance of being in the buffer. It needs no knowledge of task boundaries and keeps a fair sample of the whole stream.
• Class or task balanced selection. Keep an equal number of examples per class or per task, so rare ones are not squeezed out.
• Herding or diversity selection. Keep examples that are close to the class mean or that cover the feature space well.
• Gradient or loss based selection. Keep examples that are hard, or that give diverse gradients.

Step 3: sample a replay batch

At each training step, draw a mini-batch from the buffer. Uniform sampling is the default. Prioritized sampling gives more weight to examples with high error. Class balanced sampling gives rare classes a fair share.

Step 4: combine and update

Form a combined batch that contains new examples and replayed examples. Compute the loss on the combined batch, or compute two losses and add them with a weight. Then take a gradient step.

for new_x, new_y in stream:                # data that just arrived
    batch_x, batch_y = new_x, new_y
    if len(buffer) > 0:
        old_x, old_y = buffer.sample(k)    # replayed examples
        batch_x = concat([new_x, old_x])
        batch_y = concat([new_y, old_y])

    loss = loss_fn(model(batch_x), batch_y)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

    buffer.add(new_x, new_y)               # write rule, e.g. reservoir

Step 5: repeat

Over time the buffer holds a compact summary of everything the learner has seen. Each step keeps that summary in the training mix.

Why correlation matters

Neural networks are trained with the assumption that training examples are drawn independently from a fixed distribution. A stream breaks both parts of that assumption.

• Consecutive examples are strongly correlated. Frames from a video game level look almost the same from one step to the next.
• The distribution shifts as time passes. New tasks look different from old tasks.

Training on such a stream leads the weights to chase the latest pattern. Replay restores something close to independent and identically distributed sampling by shuffling old and new data together. This is the stability argument. It is also why replay is used even when memory is plentiful.

Why replay fights forgetting

Catastrophic forgetting happens because the weights that solved an old task are overwritten while solving a new one. The gradient for the new task has no term that cares about the old task. Replay adds that term. The loss on replayed examples acts as a reminder, so the weight update must remain good for old data as well.

An analogy: a pianist learning a new piece keeps playing a few bars of old pieces at each practice. The old pieces do not need a full rehearsal, only a regular touch to stay in the fingers.

Off-policy learning

In reinforcement learning, replay requires an off-policy learner. The data in the buffer was produced by older versions of the policy, so the learner must be able to learn from data that does not match its current behaviour. Q-learning is off-policy by design, which is why it pairs naturally with replay. Pure on-policy methods such as the basic policy gradient do not use a buffer in the same way, though some, such as PPO, reuse a batch for a few epochs before discarding it.

Common variants at a glance

Replay comes in many flavours. Each one changes a single part of the basic recipe.

Variants of the write rule

• Ring buffer. A circular array that overwrites the oldest slot. Cheap and fast.
• Reservoir buffer. A uniform sample of the whole stream.
• Per-class buffer. A fixed quota for each class, so no class disappears.
• Coreset buffer. A small set chosen to approximate the whole data in some measure.

Variants of the read rule

• Uniform. Every stored example is equally likely.
• Prioritized. Examples with larger error are more likely.
• Recency weighted. Newer examples are more likely.
• Class balanced. Each class gets an equal share of the replay batch.
• Gradient diverse. Examples are picked so that their gradients point in different directions.

Variants of the stored content

• Raw inputs and labels.
• Inputs with the old model's soft outputs.
• Latent features rather than raw inputs, which saves space and can protect privacy.
• Generator seeds and settings, when the data comes from a known generator.
• Class prototypes, a single average example per class.

Variants of the loss

• Plain cross-entropy on the mixed batch.
• Separate losses for new and replayed data, added with a weight.
• A distillation term on replayed data.
• A gradient constraint, as in methods that project the new gradient so it does not raise the loss on stored examples.

What to watch while it runs

Replay is easy to start and easy to get subtly wrong. A few signals tell you whether it is working.

• Loss on replayed batches should stay low and flat. If it climbs, forgetting has begun.
• Loss on the new data should fall at a reasonable pace. If it stalls, the replay ratio may be too high.
• The gap between training loss on the buffer and validation loss on held out old data should be small. A large gap means the buffer is being memorized.
• The class histogram of the buffer should match the plan. Print it from time to time.

Key concepts and terms

A worked example

Take a small and fully numeric example. A learner receives data from three tasks in order. Each task is a binary classification problem on tabular rows. The buffer holds 6 examples and uses reservoir sampling.

Reservoir sampling by hand

Reservoir sampling works like this. For the first 6 examples, add each to the buffer. For example number n after that (n greater than 6), draw a random integer j between 1 and n. If j is at most 6, replace the buffer item at position j with the new example. Otherwise, discard the new example.

Check that the chance each example is in the buffer is uniform. After 6 examples, each is in the buffer with probability 1. When example 7 arrives, it enters with probability 6 out of 7. An old example survives that step with probability 1 minus (1/7), since it is replaced only if j equals its position, which happens with probability 1 out of 7. So an old example is in the buffer with probability 6/7, and the new one is also in the buffer with probability 6/7. In general, after n examples, every example is in the buffer with probability 6/n. This is the property that keeps the buffer a fair sample of the whole stream without knowing task boundaries.

Training with and without the buffer

Suppose that after task 1 the accuracy on task 1 is 0.92. Now the learner trains on task 2 and task 3.

The exact numbers depend on the data, so no figures are claimed here. The pattern is what matters: no replay buys the best accuracy on the newest task and the worst on old ones. Replay trades a little of the first for a large gain in the second.

A runnable sketch in Python

The following code builds a buffer with reservoir sampling and a training loop for a stream of tabular tasks. It uses PyTorch (see PyTorch).

import random
import torch
import torch.nn as nn

class ReservoirBuffer:
    def __init__(self, capacity):
        self.capacity = capacity
        self.x, self.y = [], []
        self.seen = 0

    def add(self, xb, yb):
        for xi, yi in zip(xb, yb):
            self.seen += 1
            if len(self.x) < self.capacity:
                self.x.append(xi)
                self.y.append(yi)
            else:
                j = random.randint(1, self.seen)
                if j <= self.capacity:
                    self.x[j - 1] = xi
                    self.y[j - 1] = yi

    def sample(self, k):
        k = min(k, len(self.x))
        idx = random.sample(range(len(self.x)), k)
        return (torch.stack([self.x[i] for i in idx]),
                torch.stack([self.y[i] for i in idx]))

model = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 2))
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()
buffer = ReservoirBuffer(capacity=500)

def train_task(loader, replay_k=32):
    for xb, yb in loader:
        if len(buffer.x) > 0:
            rx, ry = buffer.sample(replay_k)
            xb_all = torch.cat([xb, rx])
            yb_all = torch.cat([yb, ry])
        else:
            xb_all, yb_all = xb, yb
        loss = loss_fn(model(xb_all), yb_all)
        opt.zero_grad()
        loss.backward()
        opt.step()
        buffer.add(xb, yb)

A reinforcement learning variant

In a Q-learning agent, the buffer stores transitions and the update uses a target network.

# one training step of a DQN-style agent
states, actions, rewards, next_states, dones = buffer.sample(64)
with torch.no_grad():
    target = rewards + gamma * (1 - dones) * target_net(next_states).max(1).values
pred = q_net(states).gather(1, actions.unsqueeze(1)).squeeze(1)
loss = nn.functional.smooth_l1_loss(pred, target)

The structure is the same: store, sample, update. Only the loss changes.

Strengths and limitations

Strengths

• Simple. A buffer, a write rule and a read rule are all you need.
• Strong. In many benchmarks, replay matches or beats more complex methods at similar cost.
• Model agnostic. It works with any network and any loss, with no change to the architecture.
• Task boundary free. With reservoir sampling it works on a stream with no marked task boundaries.
• Sample efficient in reinforcement learning. Each costly interaction is reused many times.
• Stabilizing. Mixed batches reduce the variance and the correlation of gradients.
• Composable. It combines with regularization (for example EWC), distillation (for example LwF) and parameter isolation methods.

Limitations

• It needs memory. The buffer costs storage, and the cost grows with the size of each example.
• It may raise privacy and policy issues. Storing raw past data may conflict with retention rules, consent limits or contracts. In such cases generative replay or stored summaries can help, but they are less accurate.
• Buffer bias. A small buffer can overfit. The model may memorize the few stored examples, and the result looks good on them without generalizing.
• Stale data. In reinforcement learning, old transitions come from an outdated policy, which can mislead the learner if the mismatch is large.
• Compute. Each step processes more examples per update.
• Imbalance. Without balancing, a recent large task can crowd out small old tasks.
• Hyperparameters. Buffer size, replay ratio and sampling temperature all affect results.
• Not a cure for every shift. Replay protects what is stored. It does not help for parts of the distribution that were never stored.

How it compares with alternatives

Replay is often the best first choice. It is easy to test, and if it already holds forgetting to a small level, there may be no reason to add anything else. Regularization methods are attractive when raw data cannot be kept. Hybrid recipes, for example replay plus distillation, are common in published results.

Role in tabular foundation models

A tabular foundation model learns from raw tables: rows of mixed numeric and categorical values, often with missing cells and sometimes with no useful column names. Such a model is pre-trained on a large collection of tables, either synthetic ones drawn from a generator of random data-producing processes, real ones from public repositories, or both. Models such as TabPFN, TabICL and TabSTAR show how far this idea goes. For a broader view see /model/architecture and /model/benchmarks.

Experience replay matters here for continual learning: how to add new capabilities to an existing tabular model without damaging the old ones.

Why continual learning is needed for tables

A tabular model is rarely finished. New data arrives and new needs appear.

• New domains, such as a new sector or a new type of sensor table.
• New skills, such as imputation of missing cells, anomaly detection, or classification of columns into semantic sectors.
• Better synthetic priors, which produce new kinds of training tables.
• New real datasets that become available later.

Retraining from scratch on everything is costly. Fine-tuning on only the new material makes the model forget the old material. Replay is the natural middle path.

What goes in the buffer

For tables, a stored example can be one of several things.

• A whole small table, kept as a unit, because in-context learning models treat a table (with its labelled context rows and query rows) as one training example.
• A row with its target, for models that train on rows.
• A masked-cell example, where a cell was hidden and the model must predict it. This is the format used in masked cell modeling and in imputation training.
• A synthetic table recreated from a stored random seed and generator settings. Because synthetic tables come from a generator, the buffer can store the recipe (the seed and the configuration) rather than the data. This makes the buffer almost free in memory.

That last point is a real advantage of tabular pre-training. Since many tables come from a prior that can be sampled at will, replaying old synthetic tasks is just resampling from the old prior. The old prior itself plays the role of a perfect generative replay model.

Replay across task types

A tabular foundation model often serves several task types: classification, regression, imputation, anomaly detection, and semantic or sector classification. Each type has its own loss. When a new task type is added, a replay mix keeps the earlier losses in the training objective.

Handling the lack of column names

Many real tables have no useful names for the columns, or names that carry no meaning. A tabular model that must work without names learns to identify columns by their values and by how they relate to other columns. Replay should include tables whose names have been removed or scrambled, so that the model does not start relying on names when a new dataset with rich names arrives. Otherwise, a model fine-tuned on well named tables may forget how to work from values alone.

Handling mixed types and missing values

Tables mix numeric, categorical, boolean, date and text columns. Each type is encoded in a different way. When continual training concentrates on one type, the encoders for others can drift. A replay mix with balanced column types protects them. The same goes for missing values. A fine-tuning set with no missing cells may teach the model to ignore the missing-value signal. Replay with a realistic share of missing cells keeps that skill.

Interaction with in-context learning

In-context learning models do not update their weights at inference time. They read the context rows and predict. The weights are trained once in pre-training, and continual updates change those weights later. Because the behaviour at inference depends on how well the model uses context, replay batches should keep the in-context format: a set of labelled rows and a set of query rows, with the same sampling of context size. If replay uses a different format from the original pre-training, the model can lose its ability to do in-context learning even if its row-level accuracy looks fine.

Replay with other continual tools

In practice the strongest recipes combine several tools.

• Replay plus EWC, to protect the most important weights for the old tasks.
• Replay plus distillation, in the spirit of LwF, so that the new model keeps the old model's outputs on replayed tables.
• Replay plus adapters or LoRA, which limit how much of the network can change at all.
• Replay plus a small curriculum, so that new and old tasks are mixed in a sensible order.

See also transfer learning and meta-learning, which are the neighbouring topics in tabular model research.

Practical guidance and best practices

Choose the buffer size on purpose

Start with the budget you can afford, then test a few sizes. The gain from adding more examples usually flattens. A common observation is that a small memory per class or per task already recovers a large share of the lost accuracy. Plot accuracy on old tasks against buffer size and pick the knee of the curve.

Pick a write rule that fits the stream

• If task boundaries are unknown, use reservoir sampling.
• If task boundaries are known, use a per-task quota so that every task keeps a share.
• If examples vary in value, add a diversity or hardness criterion.

Pick a replay ratio

A typical choice is to make the replay part of each batch about as large as the new part, then tune. A very high ratio slows learning of the new task. A very low ratio gives weak protection. Tune on a validation set that includes both old and new tasks.

Keep the format the same

The replayed examples should pass through exactly the same preprocessing, encoding and masking as during original training. Mismatched preprocessing is a quiet source of failure.

Use balanced sampling

Sample so that rare classes, small tasks and unusual column types still appear. Balanced replay is often more important than a larger buffer.

Store more than the label when you can

Storing the old model's outputs (soft labels) with each example lets you add a distillation term. This often improves retention at almost no extra memory cost.

Measure forgetting directly

Track three numbers after each stage:

1. Accuracy on the newest task.
2. Average accuracy over all earlier tasks.
3. The drop on each earlier task from its best value (the forgetting measure).

Report all three. A method that raises the first and collapses the second is not a success.

In reinforcement learning, mind the staleness

Keep the buffer large enough to decorrelate data, but not so large that very old transitions dominate. Consider prioritized replay with importance weights when rewards are sparse. Update the target network on a schedule that matches the pace of the buffer turnover.

Respect data rules

If storing raw rows is not allowed, consider storing summaries, synthetic stand-ins, generator seeds, or model outputs only. For synthetic pre-training tables, store the seed and configuration.

Keep the buffer reproducible

Log the random seed, the write rule, and the buffer content hash. This makes results repeatable and helps debug regressions.

A short checklist before you train

1. Write down the tasks or datasets that must not be forgotten.
2. Decide the memory budget in examples and in bytes.
3. Choose the write rule and the read rule.
4. Check that stored examples pass through the same pipeline as the originals.
5. Set up an evaluation that covers every old task, not just the new one.
6. Run a no-replay baseline, so you know how much forgetting there is to fix.
7. Run a joint-training reference if you can afford it, so you know the ceiling.
8. Log the buffer contents and the seed.

Common mistakes

• Using first in, first out for continual learning. The earliest tasks vanish first, which is the opposite of what replay is for.
• Forgetting to update the buffer. A buffer that is filled once and never refreshed may stop reflecting the stream.
• Training on the buffer only. This leads to overfitting on a few stored examples.
• Sampling the buffer with a different preprocessing pipeline. The model learns on one view and is tested on another.
• Ignoring class imbalance in the buffer. A big recent task dominates.
• Making the buffer too small for a large number of tasks. Per task memory falls below a useful level.
• Measuring only average accuracy. Averages hide the collapse of one old task.
• Leaking test data into the buffer. This inflates the results and is a common source of false claims.
• Using replay with an on-policy algorithm without correction. The data no longer matches the policy.
• Applying prioritized replay without importance weights. The sampling bias changes what the model converges to.
• Not tuning the learning rate after adding replay. A learning rate chosen for plain fine-tuning may be too high or too low for the mixed batch.
• Keeping duplicates. Near-identical stored examples waste space. See deduplication.

Frequently asked questions

What is experience replay in simple words?

Experience replay means keeping a memory of past examples and mixing some of them into each new training step. The model keeps seeing old material while it learns new material, much like a student who reviews old flash cards during each new study session. This keeps old skills from fading and makes training steadier.

What is a replay buffer?

A replay buffer is a fixed-size store of past examples. It has a rule for adding new examples and removing old ones when full, and a rule for choosing which examples to draw for training. In reinforcement learning it stores transitions, and in continual learning it stores labelled inputs from earlier tasks or datasets.

How does experience replay prevent catastrophic forgetting?

Forgetting happens because the weights that solved an old task are changed while solving a new one, and nothing in the new loss protects them. Replay adds old examples to the training batch, so the loss includes a term for the old tasks. The update must then work for both, and the old solution is not overwritten.

Why is experience replay used in deep Q-learning?

It breaks the correlation between consecutive transitions and lets each costly interaction be used many times. Training on shuffled past transitions is much closer to the independent sampling that neural network training assumes, which makes learning more stable and more sample efficient. It works because Q-learning is off-policy and can learn from older data.

How big should the replay buffer be?

There is no single answer. It depends on the task, the cost of storing an example and how many tasks there are. A good method is to test several sizes and plot accuracy on old tasks against size, then choose the point where gains flatten. Small buffers with balanced sampling often work surprisingly well.

What is reservoir sampling and why is it used for replay?

Reservoir sampling is a way to keep a uniform random sample of a stream of unknown length in a fixed-size memory. Every example seen so far has the same chance of being stored. It is used for replay because it needs no knowledge of task boundaries and keeps a fair summary of the whole stream.

What is the difference between rehearsal and generative replay?

Rehearsal stores real past examples and trains on them again. Generative replay trains a generative model on past data and uses samples from it instead, so no raw data is stored. Rehearsal is simpler and usually more accurate. Generative replay is useful when raw data cannot be kept, but its quality depends on the generator.

What is prioritized experience replay?

Prioritized experience replay samples stored transitions with a probability that grows with their learning value, usually measured by the size of the temporal difference error. Because this changes the sampling distribution, importance weights correct the bias. It helps most when informative events are rare, such as in tasks with sparse rewards.

Can experience replay be used with tabular foundation models?

Yes. When a model that learns from raw tables is extended with new datasets or new task types, replaying earlier tables, masked-cell examples or recipes for synthetic tables keeps earlier skills such as classification, imputation and anomaly detection alive. For synthetic data the buffer can store only generator seeds, which makes it very cheap.

Does experience replay work without storing raw data?

Partly. Options include storing model outputs, feature summaries, class prototypes or generator seeds, and using a generative model to produce pseudo samples. These options reduce privacy and storage concerns but usually give less protection than replaying real examples. Combining them with weight regularization can close some of the gap.

Glossary

• Replay buffer: a fixed-size memory of past examples used during training.
• Experience replay: training on random samples from a replay buffer along with new data.
• Transition: one step of interaction, made of state, action, reward, next state and done flag.
• Rehearsal: retraining on stored examples of earlier tasks.
• Reservoir sampling: a method that keeps a uniform sample of a stream in fixed memory.
• Catastrophic forgetting: the abrupt loss of old knowledge when a network learns something new.
• Continual learning: learning from a sequence of tasks or datasets without losing earlier skills.
• Off-policy: learning from data generated by a behaviour different from the current policy.
• Prioritized replay: sampling in proportion to how useful an example is for learning.
• Importance weight: a factor that corrects bias from non-uniform sampling.
• Generative replay: replaying samples from a generative model instead of stored data.
• Distillation: training a model to match the outputs of another model.
• Replay ratio: the fraction of each batch that comes from the buffer.
• Stability-plasticity trade-off: the balance between keeping old knowledge and acquiring new knowledge.
• Target network: a slowly updated copy of a network used to compute stable learning targets.
• In-context learning: making predictions by reading labelled examples in the input, without changing weights.

Summary

Experience replay stores past examples in a buffer and mixes them into training with new data. In reinforcement learning it decorrelates data and reuses costly experience. In continual learning it prevents catastrophic forgetting by keeping old tasks in the loss. The main design choices are the buffer size, the write rule (reservoir sampling is a strong default), the read rule and the replay ratio.

Replay is simple, strong and compatible with other methods such as EWC, LwF, adapters and LoRA. For tabular foundation models, it keeps earlier skills such as classification, imputation, anomaly detection and semantic labelling alive while new data and new tasks are added, and with synthetic pre-training it can be almost free because the buffer only needs the generator recipe. Keep the format identical, balance the sampling, measure forgetting directly and respect data rules. For more on how these ideas fit together, read /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Continual learning in my stack, under Foundation models and applied ML. Others in the same group: