# Continual learning for tables: learning new tasks without forgetting old ones How a tabular foundation model learns task after task without forgetting: adapters, replay of 48 raw rows per task, what failed, and what stays open. > Informational documentation, based on the paper Data Language Models: A New Foundation Model Class for Tabular Data (arXiv:2605.06290, https://arxiv.org/abs/2605.06290). Reference material, not a hosted service; hosts, keys, and ids are placeholders. Continual learning for tables means teaching one model a long series of tabular tasks, one after another, so that it keeps the early tasks while it learns the late ones. It can take on new tasks without a full retrain, and that is a capability that tree ensembles and frozen foundation models do not offer. This page is the concept guide. The benchmark story, with its protocol and its failures in date order, sits on a separate page: Continual learning benchmark. There are no scores here on purpose. The model card carries them with their protocols. ## The short version - Forgetting is the default. A network that is trained on a new task will overwrite weights that an old task needed. - The first runs forgot badly because of an architectural bug: the adapter was re-created from scratch for every task, so each task erased the last. - After the fix, one adapter lives across tasks. Its weights are the same size for any table, because it does not depend on the number of columns. - The recipe that stayed is experience replay with 48 raw rows per task. Elastic weight consolidation sits beside it as a regulariser. An L2 anchor failed, and pseudo-rehearsal was rejected for its worst case. - A missing random seed alone moved results by 15 to 20 percent, so every continual run is seeded. - Retention comes from replay, not from the weights alone. The buffer grows linearly with the number of tasks and holds raw rows, so it is a storage and privacy matter. - There is no published competitor. The claim is a capability claim: the model can take on new tasks without a full retrain. ## What continual learning means Most machine learning assumes that all the training data is available at once. You gather it, shuffle it, train, and ship. Continual learning drops that assumption. Tasks arrive in a sequence, and you may not be able to go back to the old data, or may not want to pay to process all of it again. ### An everyday picture Think of a cook who learns one cuisine a month. In January it is Thai, in February Italian, in March Mexican. A good cook ends the year with all three. A forgetful cook, who rewrites the same small notebook each month, ends with Mexican only. The notebook is the weights of the network, and the rewriting is gradient descent on the newest task. ### Stability and plasticity Two forces pull against each other. - **Plasticity** is the ability to learn something new. A model with no plasticity cannot take on the next task. - **Stability** is the ability to keep what it already knows. A model with no stability forgets everything the moment it learns something else. Every continual learning method is a position on that scale. Freeze everything and you are stable and useless. Fine-tune everything with no protection and you are plastic and forgetful. The craft is in finding a place between the two that you can defend, and in measuring it honestly. ### Catastrophic forgetting The sudden, large loss of an old skill after training on something new is called catastrophic forgetting. The word catastrophic is fair, because the loss is often near total and happens within a few updates, not gently over time. It happens because a neural network stores many skills in the same shared weights. A step that lowers the error on the new task is blind to what the same weights were doing for the old one. ## Why tables make it harder Continual learning was developed mostly on images and text. Tables bring their own difficulties, and they explain why this page exists as a separate topic. ### There is no shared input space Every image classifier sees a grid of pixels. Every language model sees tokens from one vocabulary. Tables do not share an input space across tasks. The next sends forty columns of vibration and temperature. A third sends a customer file with dates and free text. If a part of the model is tied to the number or the identity of columns, it cannot be reused on the next table. This is the reason column independence matters here. The model never uses a column's name or identity, only its values and how they relate to other columns, so tasks with different schemas still land in the same space. See Column name independence and Cell encoding. ### Shapes change from task to task Even without names, the width and length of tables differ. A weight matrix whose size depends on the column count cannot be carried from one task to the next. The adapter in this model was checked for exactly this property, and I come back to that below. ### Tasks may differ in kind A sequence of tabular tasks can mix classification and regression, tables from different sectors, and different target columns. The same rows can even carry different meaning in different tasks. That makes the question "how much of task one is left" depend heavily on which tasks are in the sequence, which is why numbers from different sequences cannot be lined up. ### The prior matters The body of the model was pre-trained on a broad mix of real and synthetic tables, so it already carries general knowledge about tables. That is a reason to keep it mostly fixed and to put the task-specific change in a small module. See Pretraining. ## What the alternatives do Before describing what the model does, it helps to say what the other options are. This is the field that the "no competitor" statement refers to. Notice that the first two rows are not failures. A tree ensemble that retrains from scratch does not forget, and a frozen in-context model has nothing to forget. They simply do not offer a way to accumulate tasks in one object without paying the full price or giving up learning. An earlier generation of the model reports a higher number on a different task sequence. The sequence is part of the measurement, so the two numbers cannot be compared directly. ## Four families of methods The research literature groups continual learning methods into a few families. It is useful to know them, because the model uses pieces of three. ### Regularisation methods These add a penalty that discourages the weights from moving where they mattered for old tasks. Elastic weight consolidation, covered in EWC, estimates how important each weight was and pulls harder on the important ones. A plain L2 anchor does the same with equal weight on every parameter, which is a much blunter tool. ### Replay methods These keep a small memory of old examples, or generate stand-ins, and mix them into training on the new task. Experience replay keeps real examples. Pseudo-rehearsal makes up fake ones. See Experience replay. ### Parameter isolation methods These give each task its own parameters, or protect part of the network, so that a new task cannot touch the old one. Adapters are in this family. A small module is trained while the main body stays fixed or nearly fixed. ### Architectural growth These add capacity as tasks arrive. The model here does not grow its body for each task. ### Where the model sits The model combines isolation (an adapter on a mostly fixed body), replay (48 raw rows per task) and regularisation (elastic weight consolidation). The Architecture page lists memory-augmented modules and regularisation as part of the design. The mix is deliberate: each family covers a weakness of the others. ## How the model does it ### The body stays broad The body is trained once on a very large and varied collection of tables, with a self-supervised goal. That training is not repeated for each task. A new task changes a small part, not the foundation. This is the reason a new task is cheap compared with a retrain. See Pretraining. ### The adapter carries the task An adapter is a small set of extra weights added on top of a large body. You train the adapter and leave the body mostly alone. For continual learning it plays two roles: it is where the new task is learned, and it is the object whose weights are passed forward from task to task. ### The property that makes it reusable The adapter does not depend on the number of columns. Its weights have the same size for a table of six columns and a table of sixty. This is not an accident of the code. I tested it, because an adapter that quietly depends on width fails only when it meets a table of a new width, and I might not notice the first time. ### One adapter, created once After the fix, the adapter is created once at the start of the sequence and carried across tasks. Each task starts from the weights the last one left, which is a warm start, and not from a fresh initialisation. ### Replay mixes the past into the present While the adapter learns the new task, each training step also draws from the buffer of stored raw rows from earlier tasks. The recipe that stayed keeps 48 raw rows per task. Raw rows are real rows of the earlier task, not summaries and not invented examples. ### Regularisation as a second guard Elastic weight consolidation penalises changes to weights that mattered for earlier tasks. It complements replay: replay shows the model what to keep, and the penalty discourages it from drifting where it should not. ## The bug that taught the most The first continual runs forgot badly. My first instinct was to blame the usual suspects: the learning rate, the number of epochs, the size of the model. None of them was the cause. The cause was an architectural bug, and it had three steps. - At the start of each task, my code built a brand new adapter. - It trained that adapter on the new task. - It discarded the adapter that held the earlier knowledge. Each task erased the last by construction. It was forgetting built into the plumbing, not forgetting in the theoretical sense. Think of a notebook where every new subject begins by tearing out all the old pages. You will remember nothing, and no study technique will help. The lesson I took from it is to suspect the plumbing before the mathematics. When a result looks catastrophic, look for the line of code that resets state. ## What I tried The experiments changed one thing at a time with everything else fixed. The table lists what happened. | Idea | What it is | Result | |---|---|---| | One adapter across tasks | Create once, carry the weights forward | The fix for the bug. Everything else sat on top of it | | Warm starts | Begin each task from the last task's weights | Tested as a control, kept as part of the design | | Epoch brake | Limit how long each task trains | Tested. Less time means less drift | | L2 anchor | Pull all weights equally toward old values | Failed in my runs | | Seeded runs | Fix the random seed | Needed, because a missing seed alone moved results by 15 to 20 percent | | Experience replay | Mix in 48 raw rows per earlier task | The recipe that stayed | | Pseudo-rehearsal | Practise on invented examples instead of stored ones | Higher mean, terrible worst case, rejected | ### Why the L2 anchor failed An anchor sounds sensible: if you do not want the weights to move, penalise movement. The trouble is that it treats every weight as equally important. Some weights matter a great deal for old tasks and others matter little. A single uniform pull either holds back the new task or fails to protect the weights that count. Elastic weight consolidation is a better idea in principle because it weights the pull by importance. In my runs the simple anchor did not help, and I moved on. ### Why pseudo-rehearsal was rejected Pseudo-rehearsal avoids storing real rows, which is attractive for privacy and size. The variant I tried had a higher mean. But on some task sequences it lost badly. A method that is excellent on average and awful in the worst case is risky, because you do not know in advance which case you will be in. I prefer a method whose worst case I can live with over one with a prettier mean. ### Why the seed matters I found that a missing random seed alone moved results by 15 to 20 percent. A seed is the starting number of a random process. It decides several things in a continual run: - the order in which rows are shown, - which rows are drawn from the buffer at each step, - how the weights start. Without it, two runs of the same recipe can differ by an amount that has nothing to do with the recipe. Before I found this, some of my findings were luck. Once the seed was fixed, I could tell real effects from noise. ## Measuring retention honestly A continual learning result is only as good as the yardstick. Three habits kept mine honest. ### The performance matrix Imagine a grid. Row i is the model after learning task i. Column j is the task you test on. The cell says how well the model does on task j at that moment. Everything in continual learning can be read from this grid. ```text tested on task 1 task 2 task 3 task 4 after task 1 a11 after task 2 a21 a22 after task 3 a31 a32 a33 after task 4 a41 a42 a43 a44 ``` - The diagonal (a11, a22, ...) shows how well each task was learned when it was new. - The last row (a41, a42, ...) shows what is left of each task at the end. - Retention compares the last row with the diagonal. A large gap means forgetting. ### Retention and backward transfer Retention asks how much of the early tasks is still there at the end. Backward transfer asks a related question: when the model learns a new task, does performance on the old ones go down, stay the same, or go up? A negative value means the new task hurt the old ones. A value near zero means the old ones were kept. Both were measured, with a fixed formula. ### Fix the formula first The formula is fixed before the runs, so it cannot be bent after the fact. If you choose the yardstick after you see the results, you will choose the one that flatters you. This is the same rule as the one sealed configuration per benchmark: no settings chosen after seeing results. ### Look at the worst case A mean can hide a disaster. The pseudo-rehearsal variant showed that. Always report the worst task or the worst sequence next to the average, and prefer the method whose worst case is acceptable. ### Do not blend sequences A hard sequence and an easy sequence give different retention for the same method. Results measured under different task sequences are never blended into one number. That is the whole reason the earlier generation's higher figure is not placed next to mine. ## Protocol in brief | Rule | Reason | |---|---| | Fixed retention formula, backward transfer reported | Every recipe is judged the same way | | Seeded runs | A missing seed alone moved results by 15 to 20 percent | | One sealed configuration | No settings chosen after seeing results | | No per-dataset tuning | The result is closer to what you would see on a task I never saw | | No blending across sequences | A different sequence is a different measurement | | Contamination check | Recall of a leaked task is not retention | | No invented rival | There is no published competitor, so none is shown | The contamination check deserves a sentence. If a task in the sequence had leaked into pre-training, the model would be recalling it and not retaining it, and the result would be inflated. Each run is checked against the training corpus with an identifier blocklist and a distributional near-duplicate check. See Evaluation. ## A worked scenario The details below are invented for illustration. They are not results. A company runs ten plants. A platform owner wants one model that has seen all of them, and wants to add the eleventh plant next month without retraining on the first ten. - **Without continual learning.** Either train ten separate models and keep ten sets of everything, or pool all the data and retrain each time a plant joins. - **With continual learning.** Keep one checkpoint with its adapter. When plant eleven arrives, train the adapter on plant eleven while replaying a few stored rows from each earlier plant, then store 48 rows of plant eleven for the future. - **What you pay.** A buffer that holds 48 raw rows per plant, which must be stored and protected, and a measurement step to confirm that earlier plants still work. This is the use case behind the platform-owner line on the Solutions page. ## Planning for the replay buffer ### How large will it get The buffer grows linearly. Each task adds 48 raw rows, so the arithmetic is simple. | Number of tasks | Raw rows in the buffer | |---|---| | 10 | 480 | | 100 | 4,800 | | 1,000 | 48,000 | The rows themselves are small, so for a short sequence the buffer is trivial. A future improvement could compress the buffer or select which rows to keep. I do not claim to have done that. ### The buffer is data Raw rows from earlier tasks are real data. If those rows are sensitive, the buffer needs the same care as any other copy of the data: access control, retention rules, and deletion on request. A method that kept its skills in the weights alone would not have this concern, and that is exactly the limit of the recipe. See Limitations. ### Deleting a task If a customer asks for their data to be removed, you must remove their rows from the buffer. You must also ask whether the adapter weights still carry what they learned. Replay makes the first part easy. The second part is an open problem for every continual learning method, and I do not promise more than the first. ## When to retrain instead Continual learning is not always the right tool. Use these rules of thumb. - **Few tasks, small data.** A full retrain is simpler and has no forgetting to manage. - **Tasks that conflict.** If a new task contradicts an old one, such as the same column meaning the opposite thing, one set of weights cannot serve both. Use separate endpoints or adapters. - **Strict audit needs.** If you must show that a given prediction came from a known training set, a fresh training run is easier to document than a long chain of updates. - **Pure in-context use.** If a table already carries enough labelled rows, in-context prediction needs no update at all. See In-context prediction. ## Adapters and in-context prediction side by side The model offers two ways to use a new table, and they answer different needs. | Question | In-context prediction | Adapter with replay | |---|---|---| | Is anything trained? | No. The labelled rows are read as a support set | Yes. A small module is updated | | What carries the task? | The rows in the table | The adapter weights and the replay buffer | | Can it forget? | There is nothing to forget | Yes, which is why replay is needed | | Best for | A table that holds its own labels | Many related tasks accumulated over time | Neither replaces the other. An in-context pass is the fastest path to a prediction for a single table. The adapter path is for owners who want knowledge to accumulate. See Capabilities for how the outputs fit together. ## How this fits the rest of the model - **Architecture.** The adapter and memory modules sit on the shared representation. See Architecture. - **Column independence.** The width-free adapter relies on the same idea that makes the model name-blind. See Column name independence. - **One checkpoint.** Adapters are merged with the core, heads and code into one object of about 2.18 billion parameters, tested in isolation. See Unified checkpoint. - **Serving.** A merged continual-learning adapter adds a few weights, not a second model. See Compression and serving. - **Other boards.** The benchmark sits beside Row-level prediction and Anomaly detection, and the whole list is on the Benchmarks page. ## The training loop in outline The sketch below shows the shape of the loop, not a specific tool. It is here to make the order of events concrete. ```text adapter = create_adapter() # once, at the start of the sequence buffer = [] # raw rows from earlier tasks set_seed(seed) # every run is seeded for task in sequence: # warm start: the adapter already holds the previous task's weights for step in range(limited_epochs): # the epoch brake batch_new = sample(task.rows) batch_old = sample(buffer) # replay of raw rows loss = task_loss(adapter, batch_new) loss += task_loss(adapter, batch_old) loss += consolidation_penalty(adapter) # elastic weight consolidation update(adapter, loss) # the body stays mostly fixed buffer += pick_rows(task.rows, k=48) # 48 raw rows per task record_row_of_grid(adapter, sequence) # fills one row of the matrix ``` Three details are worth pointing out. - The adapter is created before the loop. The bug I found was that this line sat inside the loop. - The replay batch is drawn at every step, so the past is rehearsed as the present is learned. - The grid row is recorded after each task, so retention and backward transfer can be computed from one run. ## What stayed open I list the open items without softening them. ### Retention comes from replay If you removed the stored rows, retention would drop. The weights by themselves do not hold the old tasks safely. This is a stated limit, with storage and privacy consequences. ### The buffer grows linearly Gentle at first, noticeable for a long sequence. Compressing or selecting what to keep is a future improvement and not a claim. ### No external check Because there is no published competitor, there is no outside leaderboard to confirm the result. The claim rests on my protocol, the fixed formula and the seeded runs. A third-party benchmark would be a stronger test. ### A different sequence cannot be compared An earlier generation reports a higher number on a different task sequence. I do not claim to beat it and I do not claim to lose to it. ### Sensitivity to seeds A fluctuation of 15 to 20 percent from a missing seed shows how fragile continual learning results can be. Anyone who reruns this should seed their runs and report the seed. ### Long sequences are not proven forever The tests cover a long sequence of fine-tunes, not an unlimited one. Nothing here promises that retention holds across thousands of tasks. ## Common misreadings ### "The model never forgets" It can forget. Replay and regularisation reduce forgetting. They do not abolish it. ### "Continual learning means no storage" The recipe stores raw rows. Storage is part of the mechanism. ### "A frozen model that predicts in context is the same thing" A frozen model has no update path. It reads context and answers. It is useful, and it does not accumulate anything between tables. ### "Replay of 48 rows is a tuned magic number" It is the setting I sealed for the benchmark, chosen before the final runs and not changed after. Your own tasks may justify another size, and you should test it with seeded runs. ### "Higher mean means better method" Not if the worst case is terrible. That is why pseudo-rehearsal was rejected. ## A checklist for your own continual runs - Is the adapter created once, and carried across tasks? - Does anything in it depend on the number or order of columns? - Is every run seeded, and is the seed reported? - Is the retention formula written down before the first run? - Is the worst task reported next to the mean? - Is the buffer size planned, and is the buffer protected like any copy of the data? - Is each task sequence reported as its own measurement? - Is contamination against the training corpus checked? ## Fact sheet | Item | Value | |---|---| | Date of the continual learning work | 13 July 2026 | | Task | Retention across a long sequence of fine-tunes | | Adapter | One, created once, independent of column count | | Replay | 48 raw rows per task | | Regularisation | Elastic weight consolidation as a companion | | Failed | L2 anchor | | Rejected | Pseudo-rehearsal, for its worst case | | Noise found | A missing seed moved results by 15 to 20 percent | | Competitor | None published | | Buffer growth | Linear in the number of tasks | | Retention source | Replay, not the weights alone | ## Terms used on this page - **Adapter.** A small trained module added to a large body. See Adapters. - **Backward transfer.** The effect of learning a new task on earlier tasks. Negative means forgetting. - **Buffer.** The stored raw rows from earlier tasks. - **Catastrophic forgetting.** A sudden, large loss of an old skill after training on something new. - **Elastic weight consolidation.** A penalty that protects weights that mattered for earlier tasks. See EWC. - **Experience replay.** Mixing stored real examples from earlier tasks into later training. - **Plasticity.** The ability to learn something new. - **Pseudo-rehearsal.** Practising on invented examples that mimic earlier tasks. - **Retention.** How much of the early tasks remains at the end of the sequence. - **Seed.** The starting number of a random process, fixed so a run can be repeated. - **Stability.** The ability to keep what is already known. ## Quick answers - Can the model take on new tasks without a full retrain? Yes. - Is there a published rival? No. - Does the adapter depend on the number of columns? No. - What is replayed? 48 raw rows per earlier task. - Does retention live in the weights? Not alone. - Does the buffer shrink? No, it grows linearly. - Are the scores here? No, they are in the model card with their protocols. ## Frequently asked questions ### What is continual learning for tables? It is learning a sequence of tabular tasks one after another while keeping the earlier ones. The model is updated for each new task through a small adapter, and it still performs the old tasks. This avoids retraining from scratch every time new data arrives, and it works across tables with different shapes because the adapter does not depend on the number of columns. ### Why is forgetting worse for tables than for images? Images share one pixel grid, and text shares one vocabulary. Tables share neither: each task can bring a different number of columns, different types and a different meaning. Any part of a model tied to a fixed width or to column names cannot be carried to the next table. A model that ignores column identity and uses a width-free adapter avoids that trap. ### What was the biggest cause of forgetting in the first runs? An architectural bug. The adapter was re-created from scratch for every task, so each task erased the last. The learning rate, the number of epochs and the model size were not the cause. After the adapter was created once and carried across tasks, the controlled experiments became meaningful. ### What recipe stayed? Experience replay with 48 raw rows per task, on top of one adapter that carries across tasks, with regularisation such as elastic weight consolidation as a companion. It was simple and stable, and its worst case was acceptable. ### Why not store nothing and use pseudo-rehearsal? I tried a pseudo-rehearsal variant, which invents stand-ins for old examples. It had a higher mean but a terrible worst case, because on some sequences it lost badly. ### Does retention live in the weights? No. Retention here comes from replay, not from the weights alone. If the stored rows were removed, retention would drop. ### How is retention measured? With a fixed formula chosen before the runs, comparing performance on each earlier task at the end with performance when that task was new. Backward transfer is reported alongside it, to show whether learning a new task hurt the old ones. Runs are seeded, and results from different task sequences are never blended. ### Is there a competitor to compare with? No published one. Tree ensembles retrain fully for each task, and frozen foundation models have no update path at all. So the claim is that the model can take on new tasks without a full retrain, which is a capability claim and not a ranking. ### When should I retrain instead of using continual learning? When you have few tasks and small data, when tasks contradict each other, or when you need a clean audit trail of exactly what a prediction was trained on. ## Summary Continual learning for tables asks a model to learn tasks in sequence without losing the early ones, and tables make this harder than images or text because tasks do not share a width, a vocabulary of column names or a type mix. The first runs forgot because the adapter was rebuilt for every task. After that fix, an L2 anchor failed, pseudo-rehearsal had a higher mean but a terrible worst case, and a missing seed alone moved results by 15 to 20 percent. What stayed open is clear: retention comes from replay, the buffer grows linearly and holds raw rows, there is no external leaderboard, and a different sequence cannot be compared. For the benchmark story read Continual learning benchmark. For the design read Architecture, and for the limits read Limitations.