# Continual learning benchmark: keeping old tasks while learning new ones How a tabular foundation model learns task after task without forgetting: the failures, the replay recipe that stayed, and what stayed open. > Informational documentation, based on the paper Data Language Models: A New Foundation Model Class for Tabular Data (arXiv:2605.06290, https://arxiv.org/abs/2605.06290). Reference material, not a hosted service; hosts, keys, and ids are placeholders. The continual learning benchmark asks whether a model can learn many tasks in a row and still remember the first ones. This page explains how retention is measured, how I built the capability, what made the difference, and what is still open. The short version is this. The model can take on new tasks without a full retrain. The recipe that stayed is experience replay with 48 raw rows per task. Retention comes from replay, not from the weights alone, and the replay buffer grows linearly. There is no published competitor, so there is no leaderboard to climb. There are no scores on this page on purpose. The model card carries them with their protocols. For the wider picture, see Continual learning for tables, the Pretraining page and the full Benchmarks timeline. ## What this benchmark measures Continual learning is about a sequence. You fine-tune a model on task one. Then on task two. Then on task three, and so on, for a long chain of tasks. At the end you ask a simple question. How much of the first tasks is still there? That is retention. ### An everyday picture Think of learning languages. You learn French. Then Spanish. Then Italian. Then German. If you stop practising French, the Spanish and Italian words start to crowd it out. After a year you may find that you can barely order coffee in French. A neural network behaves in a similar way. When it is trained on new data, the weights that stored the old skill get overwritten. This is called catastrophic forgetting. The word catastrophic is fair. The loss is often sudden and large, not gentle. ### Why tables need this In real work, tabular tasks arrive one after another. Another plant sends its data next month. A platform owner wants one model that has seen all of them. See Solutions for the use case where per-plant tasks accumulate in one checkpoint without full retraining. ### What is scored Retention is measured with a fixed formula. The formula is fixed before the runs, so it cannot be bent after the fact. Backward transfer was measured as well. Backward transfer asks a related question. When you learn a new task, does your performance on old tasks go down, stay the same, or go up? A negative value means the new task hurt the old ones. That is forgetting. A value near zero means the old tasks were kept. ## Who the model is compared with There is no published competitor. I want to be exact about that, because it shapes what the benchmark can claim. Two kinds of existing methods are worth naming. - **Tree ensembles.** Methods such as gradient-boosted trees, for example XGBoost, have no update path. To learn a new task they retrain fully. Retraining from scratch does not forget, but it is not continual learning. - **Frozen foundation models.** A foundation model that predicts in context has no update path at all. It does not train on your table, so there is nothing to forget and nothing to learn. It reads context and answers. So the comparison is a different kind of statement. The model is the only one in the comparison that can take on new tasks without a full retrain. That is a capability claim, not a ranking claim. An earlier generation of the model reports a higher number on a different task sequence. The two cannot be compared directly, because the sequence of tasks is part of the measurement. A hard sequence and an easy sequence give different retention for the same method. I say this here so nobody lines the two numbers up. ## How I built it The story has a bug at its centre, a set of controlled experiments, and one recipe that survived. ### The first runs forgot badly The first runs forgot badly. After a few tasks, the early ones were almost gone. My first instinct was to blame the usual suspects: the learning rate, the number of epochs, the size of the model. I spent some time on these. They were not the cause. ### The cause was an architectural bug The adapter was re-created from scratch for every task. An adapter is a small set of extra weights added on top of a large frozen model. You train the adapter and leave the main model alone. This is cheap, and it is meant to keep the main model safe. See Adapters for the general idea, or the Architecture page for how the heads sit on the shared core. Here is what went wrong. At the start of each task, my code built a brand new adapter. It then trained that adapter on the new task. The adapter that held the knowledge of the previous task was thrown away. So each task erased the last. Nothing carried over, by construction. This was not forgetting in the usual sense. It was forgetting built into the plumbing. Think of a notebook. Every time you start a new subject, you tear out all the pages from the last subject. Of course you remember nothing. ### The fix: one adapter that carries over After the fix, one adapter lives across tasks, and its weights carry over from one task to the next. For that to work, the adapter must not depend on the number of columns. Different tasks have different tables. One has ten columns. Another has forty. If the adapter's shape depended on the column count, it could not be reused. So I checked that the adapter did not depend on the number of columns. Its weights are the same size for any table. This is in line with how the model reads tables in general, see Cell encoding and Column name independence. Checking is the right word. I did not assume the property. I tested it, because an adapter that quietly depends on width would fail only on a table of a different width, and I might not notice. ### Controlled experiments With the bug gone, I ran controlled experiments. I changed one thing at a time and kept everything else fixed. - **Warm starts.** Begin each task from the weights left by the last one, instead of from a fresh start. - **An epoch brake.** Limit how long each task trains, so the weights have less time to drift. - **An L2 anchor.** Add a penalty that pulls the weights back toward their old values. This idea is close to elastic weight consolidation, see EWC. In my setting it failed. - **Seeded runs.** Fix the random seed, so that two runs of the same recipe can be compared. ### The seed discovery The seeded runs came from a surprise. I found that a missing random seed alone moved results by 15 to 20 percent. That is a huge amount of noise. It means that if you run the same recipe twice without fixing the seed, you can see a difference of 15 to 20 percent that has nothing to do with your recipe. Before I found this, some of my "findings" were just luck. A recipe looked better because the dice rolled well. Once I fixed the seed, I could tell real effects from noise. This is a lesson that applies far beyond this benchmark. If your experiment has randomness, control it before you draw conclusions. ### Why the L2 anchor failed An L2 anchor sounds sensible. If you do not want the weights to move, penalize movement. The trouble is that it treats every weight as equally important. Some weights matter a great deal for old tasks. Others matter little. A single uniform pull either holds back learning of the new task, or fails to protect the weights that matter. Elastic weight consolidation tries to fix this by weighting the pull per weight by how important each weight was. That is a better idea in principle. In my runs, the simple anchor did not help, and I moved on. ### The recipe that stayed: experience replay The recipe that stayed is experience replay with 48 raw rows per task. Experience replay keeps a small memory of examples from earlier tasks. When the model learns a new task, it also practises on some of those stored examples. See Experience replay for the general idea. The picture is a student who keeps a small deck of flash cards from every past subject. While learning a new subject, the student also flips through a few old cards each day. Here, the deck holds 48 raw rows per task. Raw rows are actual rows from the earlier task, not summaries or invented ones. ### The rejected variant I also tried pseudo-rehearsal. In that approach you do not store real rows. You make up fake examples that mimic the old task and practise on those. The pseudo-rehearsal variant had a higher mean. On average it kept more. But it had a terrible worst case. On some task sequences it lost badly. A method that is great on average and awful in the worst case is risky, because you do not know which case you will be in. I rejected it. I prefer a method whose worst case is acceptable over a method with a prettier mean. ## What made the difference ### Fixing the plumbing The biggest jump came from the adapter fix. Everything else was fine tuning on top of a repaired base. ### Replay with raw rows Among the controlled experiments, replay with raw rows was the one that kept its place. It was simple, it was stable, and its worst case was acceptable. ### A fixed way to measure Retention is measured with a fixed formula, and backward transfer was measured as well. Fixing the formula early meant that every recipe was judged by the same yardstick. ### Seeded runs Controlling the seed turned a noisy field into a readable one. Without it, I could not have told a real improvement from luck. ## Why this matters for tables ### New tasks without a full retrain The model is the only one in the comparison that can take on new tasks without a full retrain. Instead of retraining on all history each time, the model adds a task and keeps the rest. See Capabilities for how this sits with the other outputs. ### Memory in the architecture The Architecture page describes memory-augmented modules and regularization methods as part of the design. The benchmark is where those ideas meet a number. ### One checkpoint Continual learning shares one core with every other capability. The unified checkpoint page, Unified checkpoint, explains how the heads and adapters are merged into one object. ## The protocol ### Fixed formula Retention is computed with a fixed formula. Backward transfer is reported alongside. ### Seeds Runs are seeded. Because a missing seed alone moved results by 15 to 20 percent, I would not trust any result that did not control the seed. ### One sealed configuration My figures come from one sealed configuration per benchmark, with no per-dataset tuning. For continual learning that means the replay size, the epoch limit and the other settings were chosen before the final runs and not changed after. ### No blending Results measured under different datasets, metrics or masking schemes are never blended into one number. A different task sequence is a different measurement, which is why the earlier generation's number is not placed next to mine. ### No competitor to re-run There is no published competitor here. So the rule that competitor figures are third-party values has nothing to apply to. I did not invent a rival to make the table look fuller. ### Contamination Each run is checked for contamination against the training corpus. If a task in the sequence had leaked into pre-training, retention would be inflated, because the model would be recalling rather than retaining. ### Summary of the rules | Rule | Why it matters here | |---|---| | Fixed retention formula | Every recipe is judged the same way | | Backward transfer measured | Shows whether new tasks hurt old ones | | Seeded runs | A missing seed alone moved results by 15 to 20 percent | | One sealed configuration | No settings chosen after seeing results | | No blending | Different task sequences are different measurements | | Contamination check | Recall of leaked tasks is not retention | ## Lessons from the build ### Suspect the plumbing before the maths The cause of the worst forgetting was a line of code that rebuilt the adapter. It was not a subtle point of theory. When a result looks catastrophic, check the plumbing first. ### Test the property you rely on The adapter must not depend on the number of columns. I tested it. A property you assume but never test is a bug waiting for a table of a new shape. ### Beware the pretty mean The pseudo-rehearsal variant had a higher mean and a terrible worst case. A mean can hide a disaster. Always look at the worst case. ### Control the noise A missing seed moved results by 15 to 20 percent. Controlling randomness is cheaper than any clever recipe. ### Report what failed The L2 anchor failed. I say so. Someone else might try it and save a week. ## What stayed open Honesty about open items is part of the protocol. ### Retention comes from replay Retention here comes from replay, not from the weights alone. If you removed the stored rows, the retention would drop. The weights by themselves do not hold the old tasks safely. This matters for privacy and storage. Replay keeps raw rows from earlier tasks. If those rows are sensitive, you must handle that. A method that retained skills in the weights alone would not have this concern. ### The buffer grows linearly The replay buffer grows linearly. Each new task adds 48 raw rows. After many tasks, the buffer is large. A linear growth is gentle at first. Over a very long sequence it adds up. A future improvement could compress or select what to keep. ### A different sequence cannot be compared An earlier generation reports a higher number on a different task sequence. The two cannot be compared directly. I do not claim to beat it, and I do not claim to lose to it. ### No external check Because there is no published competitor, there is no outside leaderboard to confirm the result. The claim rests on my protocol, the fixed formula and the seeded runs. I have tried to make those as strict as I can, but a third-party benchmark would be a stronger test. ## What this means for a user If you have tasks arriving over time and you want one model that keeps them all, the model can take on new tasks without a full retrain. If your tasks are few and your data is small, a full retrain may be simpler. See Limitations for what the model does not claim, and Evaluation for how scores should be read. ## Frequently asked questions ### What is continual learning in a tabular foundation model? It is the ability to learn a series of tasks one after another without losing the earlier ones. The model is updated for each new task and still performs the old ones. This avoids retraining from scratch every time new data arrives. ### What is catastrophic forgetting? It is the sudden, large loss of an old skill when a network is trained on something new. The weights that stored the old skill get overwritten. It is called catastrophic because the loss can be nearly total, not gradual. ### Why did the first runs forget so badly? The adapter was re-created from scratch for every task, so each task erased the last. It was an architectural bug, not a tuning problem. After I made one adapter live across tasks, the forgetting dropped and the controlled experiments became meaningful. ### What recipe stayed? Experience replay with 48 raw rows per task. The model keeps a small set of real rows from each earlier task and practises on them while learning a new one. It was simple, stable and had an acceptable worst case. ### Why did you reject pseudo-rehearsal? It had a higher mean but a terrible worst case. A method that sometimes loses badly is risky, because you cannot know in advance which case you will meet. I preferred a method whose worst case I could live with. ### Why did a missing seed matter so much? I found that leaving out the random seed alone moved results by 15 to 20 percent. That is enough noise to hide or fake an improvement. After that I ran every comparison with seeds fixed. ### Is there a published competitor? No. Tree ensembles retrain fully for each task, and frozen foundation models have no update path at all. So the model is the only one in the comparison that can take on new tasks without a full retrain. That is a capability claim, not a ranking. ### Does the model remember through its weights alone? No. Retention here comes from replay, not from the weights alone. The stored rows are part of the mechanism. This is a stated limit, and it has storage and privacy consequences. ### How big does the replay buffer get? It grows linearly. Each task adds 48 raw rows, so the buffer grows in step with the number of tasks. That is gentle for a short sequence and noticeable for a long one. ### Can I compare your retention with an earlier generation's number? Not directly. The earlier generation reports a higher number on a different task sequence. The sequence is part of the measurement, so numbers from different sequences do not belong in one table. ## Summary The continual learning benchmark measures how much of the first tasks remains after a long sequence of fine-tunes. There is no published competitor: tree ensembles retrain fully and frozen foundation models cannot update. The first runs forgot badly because the adapter was rebuilt for every task. After I fixed that and checked the adapter did not depend on the number of columns, controlled experiments showed what worked. Warm starts and an epoch brake were tested, an L2 anchor failed, and a missing seed alone moved results by 15 to 20 percent. The recipe that stayed is experience replay with 48 raw rows per task. Pseudo-rehearsal had a higher mean but a terrible worst case, so I rejected it. What stayed open: retention comes from replay and not from the weights alone, the buffer grows linearly, and an earlier generation's higher number is on a different sequence. Read next: Continual learning for tables, Row-level prediction, Anomaly detection, and Benchmarks.