Pre-training

The model is pre-trained once, on a very large and varied collection of tables, with a self-supervised goal: hide cells and learn to recover them. That single habit teaches it what a plausible value looks like in a column, how columns move together, and what a table of a given industry tends to contain.

After that, new tables need no training at all.

This page explains the objective, the data, the training setup, what went wrong on the way, and what happens after training. It is the companion to Architecture, which describes the network being trained.

Pre-training in one paragraph

A language model reads huge amounts of text and learns to predict missing words. This model reads huge amounts of tables and learns to predict missing cells.

Because the tables come from many sectors, with many shapes and many patterns of missing data, the model learns general rules of tabular data rather than the quirks of one dataset. The first model of this class was a 140M parameter model trained on more than 2.3 million synthetic and real-world datasets.

The model on this site is its successor, with more than 1.5 billion parameters. See Research for the paper behind the idea.

The self-supervised objective

The main objective is masked cell modeling. Cells are hidden at random in a table, and the model learns to reconstruct them from the remaining cells.

See masked cell modeling and self-supervised learning for the general methods.

input     :  [ 34 | "Ana" | ?  | 2024-11-03 ]
target    :  recover the hidden cell from the rest of the row and the column

No labels are needed. The table is its own teacher.

That matters because labelled tables are rare and expensive, while unlabelled tables are everywhere.

An everyday picture

Imagine a crossword where you can see most of the letters. You do not need an answer key to learn English spelling from it.

You just keep filling blanks and checking against what the grid already says. Masked cell modeling is a crossword with billions of grids.

What the model has to learn to win

To recover a hidden cell, the model must use two kinds of evidence.

• Row evidence. The other cells in the same record. A hidden city is easier to guess from the postal code beside it.

• Column evidence. The other values in the same attribute. A hidden age is easier to guess when you have seen thirty other ages in that column.

These are exactly the two axes that the model attends along. The objective and the architecture fit each other.

See axial attention.

Why it also teaches imputation

Reconstructing a hidden cell is imputation. The model that wins at masked cell modeling is, by construction, a model that can fill gaps.

At use time, a really missing cell is treated like a masked one. That is why imputation is not bolted on later.

It is the training task itself. See imputation networks and the benchmark page on imputation.

Missing cells in the input

Real missing cells are encoded explicitly, as their own kind of input, not as an error. During pre-training the model sees tables that already have holes, as well as holes that were added on purpose.

So it learns two things at once: how to read around a gap, and that a gap can itself be a signal.

The three ways data goes missing

Missing data comes in three flavors, and a good pre-training mix covers all of them. These are the standard mechanisms.

The hardest case is the last one, because the reason for the gap is the very thing you cannot see. In the imputation benchmark, the widest margin over the other methods came under not-at-random missingness.

See Missing-value imputation and Missing-data robustness.

Why column identity is zeroed during training

Column embeddings are always zeroed. The model never learns to rely on a column's name or identity, in pre-training or afterwards.

If the name were available, the shortcut would be tempting: learn that a column called age holds ages, and stop looking at the values. That shortcut breaks the moment a table arrives with a column called c3.

By closing the shortcut during training, the model is forced to learn what a phone number, a date, or a price looks like from the values and from how columns relate. This is the root of its name-independent behavior.

See column name independence and Metadata independence.

The data

A foundation model for tables needs more variety than the real public tables can offer. So the training mix is a blend of real tables and large-scale synthetic data.

See synthetic data generation.

Why synthetic data

Real public tables are limited in three ways.

• Count. There are far fewer of them than there are web pages for text.

• Coverage. Whole industries are barely represented in open data.

• Control. You cannot ask a real dataset to contain a specific pattern, a specific missing-value mechanism, or a specific shape.

Synthetic tables fix all three. They can be made in the millions.

They can be generated for many sectors, with realistic per-column ranges, relationships between columns, and missing-value patterns. And they can be built to cover the cases the model must handle.

The prior is part of the model

A synthetic generator is not neutral. It encodes beliefs about what tables look like.

That set of beliefs is called the prior, and the model absorbs it. The idea is the same as in other prior-fitted approaches such as TabPFN: the model learns to be a good guesser for tables drawn from the prior.

This has a sharp consequence. If the prior has a flaw, the model inherits it.

A real example: the XOR wall

Early in the work on row-level prediction, I ran nine different similarity-based decoders, and every one hit the same wall. There were two causes.

1. The support sets were physically too small: 24 supports, where 128 or more were needed.

2. Deeper still, the training prior turned quantile-band targets into XOR-like patterns. An XOR pattern is one where neither input alone predicts the output, only the combination does. A decoder that looks for similar neighbours cannot learn that.

Changing the decoder did nothing, because the problem was upstream, in what the model had been shown. A cluster-style prior removed the wall.

Error-correcting output codes added only bit noise, so I kept direct retrieval over the supports. The lesson: when many different heads fail in the same way, look at the data the body was trained on.

Real tables

Real tables ground the model in real distributions, real naming noise, real units, and real mess. They are also used to build the sector knowledge: the model needs to see what actual retail, hospital, and logistics tables look like to name a sector from raw values later.

For the sector task, I trained a small delta on the last blocks of the body using 1,401 tables that are excluded from the test set. That separation is not a detail.

It is what keeps the sector score honest. See Blind sector identification.

Coverage across sectors

The mix spans the industries a table can come from. The point is breadth.

A model that has only seen retail tables will find a hospital table strange. A model that has seen both learns what is general (a date is a date) and what is specific (diagnosis codes appear in hospital tables).

What the model learns, layer by layer

It helps to picture the knowledge as layers, from simple to rich. This is an intuition, not a measured claim about specific layers.

The last level is why cross-table matching works without shared keys.

Training at scale

A model of more than 1.5 billion parameters, trained on millions of tables, does not fit on one machine. The standard tools apply.

• Distributed multi-GPU training with data parallelism and tensor parallelism. See data parallelism and tensor parallelism.

• Mixed-precision arithmetic to cut memory and time. See mixed precision.

• Hyperparameter tuning and systematic evaluation on standard academic datasets.

Data parallelism, in plain words

Each GPU gets a different batch of tables. Every GPU computes how the weights should change.

The changes are averaged and applied everywhere, so all GPUs stay in step. It is like several cooks tasting different pots of the same soup and then agreeing on one change to the recipe.

Tensor parallelism, in plain words

When one layer is too big for a single GPU, its matrices are split across GPUs. Each computes its slice, and the slices are combined.

It is like several people carrying one very long table: each takes a corner.

Mixed precision, in plain words

Most arithmetic is done in a shorter number format, which is faster and uses less memory. A few sensitive steps stay in a longer format.

Batches of tables are uneven

Tables differ in size, so a batch is not a neat block. The model is also asked to work in the shape it will meet at use time: some rows to read as context, and other rows to answer.

Beyond the main objective: heads and adapters

The body is trained broadly. Capabilities that need a specific output are trained on top of it, usually as a head or an adapter.

The sections below describe the main ones and what was learned. See Architecture for what each head reads.

Probabilistic regression head

A 999-quantile head is trained with a CRPS loss. That follows the benchmark paper's own finding: you win on the scoring rule you train for.

It gave first place on CRPS. CRLS lagged, and the diagnosis was precise.

The target had been standardized by its standard deviation, which for heavy-tailed targets such as house prices, runtimes, and car prices made the predicted distribution too narrow. CRLS punishes a narrow distribution hard.

The fixes, in order:

1. A log transform for heavy tails.

2. A CRLS fine-tune, which I later dropped when both metrics got worse.

3. Split-conformal calibration with an isotonic correction.

See Probabilistic regression.

Anomaly detection

The anomaly scorer is trained on normal rows only, which is the semi-supervised setting in which published methods are measured. My first chain used the wrong engine, the weaker imputer.

I switched to the imputation engine, then fixed its integration bugs one at a time: the output scale (a per-column linear calibration), an empty head when the data has no missing values (I inject 15 percent random missingness before fitting), and import-time side effects. See Anomaly detection.

Sector head

A hierarchical head with coarse families came first. Then a delta was trained on the last blocks of the body.

I tried eighteen other ideas, and two helped. Test-time augmentation, a favourite from vision, destroyed the signal in tables.

Matching adapters

For entity matching, over-training was the mistake. The Music runs trained for 8,000 steps, which hurt. 1,200 steps with a two-model ensemble and a calibrated threshold fixed it and cut the run-to-run variance.

A related lesson: a checkpoint that was not preserved at the time of sealing taught me to copy every sealed adapter at the moment of sealing.

Continual learning during and after training

As new data and tasks arrive, the model is updated without retraining from scratch. See continual learning for tables.

The mechanics:

• Adapters that do not depend on the number of columns, so their weights carry from one task to the next.

• Experience replay with 48 raw rows per task. See experience replay.

• Regularization such as elastic weight consolidation, which limits changes to weights that mattered before. See EWC.

The first runs forgot badly because the adapter was re-created from scratch for every task. After the fix, I tested warm starts, an epoch brake, and an L2 anchor, which failed.

I also found that a missing random seed alone moved results by 15 to 20 percent, so every continual run is seeded. A pseudo-rehearsal variant had a higher mean but a terrible worst case, so I rejected it.

Be clear about what this gives. Retention comes from replay, not from the weights alone, and the buffer grows linearly.

See Continual learning.

Keeping the ruler honest

A large part of training a model like this is checking that the measurements mean what they appear to mean. Several of the largest improvements came from fixing the harness, not the network.

None of these are model failures. They are the kind of silent error that makes a model look better or worse than it is.

Pre-training is only as good as the signal used to steer it.

Seeds and sealed configurations

Every benchmark result comes from one sealed configuration, with no per-dataset tuning. Seeds are fixed and reported.

A win that sat inside its own seed noise was dropped. Rivals often get hours of tuning per dataset.

The model gets one default configuration. See Benchmarks and Evaluation.

Contamination checks

A model trained on millions of tables might have seen a benchmark table. If it had, the score would flatter it.

So each run is checked for contamination against the training corpus.

The audit used two checks:

1. An identifier blocklist: known benchmark dataset identifiers are excluded from training.

2. A distributional near-duplicate check: tables that are close to a benchmark table in their statistics are flagged even if they carry another name.

The tables used to train the sector delta are excluded from the sector test set, and the test set was frozen. The audit was completed before the model card was published on 7 August 2026.

After training: compression and packaging

Once trained, the model is packaged.

One checkpoint

The core and every head and adapter were merged into one unified checkpoint of about 2.18 billion parameters, with the code embedded, so one line of bootstrap gives access to every capability. The merged model was tested in isolation.

See One checkpoint.

Post-training quantization

Post-training quantization and model compression reduce the size of the weights for efficient serving with minimal loss of accuracy. The model was dumped at half precision, 8-bit, and 4-bit, and fidelity was measured with TurboQuant.

See TurboQuant and compression and serving.

The idea: store each weight with fewer bits. A 4-bit weight takes a quarter of the space of a 16-bit one.

The risk is that rounding adds error, so fidelity is measured against the full-precision model on the real tasks.

What pre-training does not do

Be precise about the limits.

• It does not memorize your table. Your data is read at use time, in context.

• It does not guarantee accuracy on a distribution that differs from the one in front of the model. A held-out score describes unseen rows of the same table. See Evaluation.

• It does not remove the need to check. Synthetic data is a stand-in. It is useful for testing and filling gaps, and not a substitute for real records in a final decision.

• It does not make every task equally strong. Regression and multiclass boards are behind the leaders. See Limitations.

A timeline of the training work

The order is not a script. Many findings in later steps changed how I read earlier ones.

See Benchmarks for each entry.

One training step, walked through

A concrete step makes the objective easier to hold in your head. The numbers below are for illustration only.

1. A table is drawn from the mix. It might be real or synthetic, from retail or from a hospital.

2. Some cells are hidden. The model sees a blank, encoded as a missing cell, in their place.

3.

4. For each hidden cell it proposes a value. For a number, a number. For a category, a distribution over the labels.

5. The proposal is compared with the true value, which the training code kept aside.

6. The error flows back through the network and the weights move a small step.

Repeat this across a very large number of tables and the model improves at the one thing the loop rewards: guessing hidden cells from context.

Choosing what to hide

The choice of mask shapes what is learned. A mask that hides single cells at random teaches local filling.

A mask that hides a whole column teaches the model to infer an attribute from the rest of the row. A mask that follows the not-at-random mechanism teaches it that blanks are not innocent.

A varied mix of masks is the safer choice, for the same reason a varied mix of tables is. A model that only ever saw one kind of hole will be surprised by the next kind.

Watching a checkpoint while it trains

During training, a checkpoint is judged on masked cells it has not trained on. That is a cheap, label-free signal.

It is not the final test. The final tests are the benchmarks, each run on one sealed configuration, and each change is tested on the full board rather than on one dataset, because a gain on one dataset can hide a loss on ten others.

Why not just fine-tune a language model on text versions of tables?

This is the most common alternative, so it deserves a direct answer. Flattening a row into text discards the joint structure between columns.

The model then has to rediscover, from a string, that two numbers belong to the same record and that a column is a column. It also pays for every digit as a separate piece of text.

Reading cells as cells keeps that structure from the start. The paper behind the model calls the flattening route a bottleneck of its own, and the pre-training here is built to avoid it.

See Research.

What changes if the pre-training mix changes

Because the prior is part of the model, any change to the data is a change to the model. Three examples show how a mix choice turns into behavior.

This is why the training data is audited and why a change to the prior is tested on the whole board before it is kept.

Key terms used on this page

• Mask. A cell that has been hidden so the model can try to recover it.

• Prior. The set of beliefs about what tables look like, built into the data generator.

• Support set. The labelled rows the model reads as context.

• Adapter. A small trainable module attached to the body for one task.

• Replay. Mixing a few stored raw rows from earlier tasks into later training.

• Sealed configuration. The one fixed setting used for a benchmark, with no per-dataset tuning.

• Contamination. Evaluation data that also appears in the training corpus.

• Quantization. Storing weights with fewer bits to shrink the model.

A pre-training checklist from the lessons above

These are the habits that the earlier sections argue for.

They are written as a list so they can be checked off.

• Check the prior before changing a head that fails in the same way many times.

• Keep column identity out of the inputs.

• Cover all three missingness mechanisms in the mix.

• Seed every run, and report the seed.

• Keep training tables for a head out of its test set.

• Test each change on the full board, not on one dataset.

• Copy every sealed adapter at the moment of sealing.

• Audit for contamination before any claim is published.

How the parts connect

• Architecture is the network that this training shapes.

• Capabilities lists what comes out.

• Tabular foundation models puts this training style next to the alternatives.

• In-context prediction shows how the pre-trained weights are used without updating them.

Frequently asked questions

What is the main pre-training objective?

Masked cell modeling. Cells are hidden at random in a table and the model learns to reconstruct them from the remaining cells in the row and column.

It needs no labels, and it teaches both what a plausible value looks like and how columns relate.

Why is synthetic data part of pre-training?

Real public tables are too few, too narrow in sectors, and impossible to control. Synthetic tables can be made in the millions, for many sectors, with realistic ranges, relationships, and missing-value patterns.

The trade-off is that the generator's prior is absorbed by the model, so a flaw in the prior becomes a flaw in the model.

Does the model train on my data?

No. At use time it reads your table in context and returns outputs.

No weights are updated for a prediction. Pre-training happened before, on other tables.

How is the model kept from using column names?

Column embeddings are always zeroed, in training and in use. The model sees only values and how they relate to other columns, so it learns what a phone number or a date looks like instead of relying on a header.

How big is the training set?

The first model of this class was trained on more than 2.3 million synthetic and real-world datasets. The model on this site is its successor.

The documentation does not state a dataset count for the successor, so none is claimed here.

What hardware and methods are used for training?

Distributed multi-GPU training with data and tensor parallelism, and mixed-precision arithmetic. Hyperparameters are tuned and the model is evaluated systematically on standard academic datasets.

How is contamination avoided?

The evaluation sets are checked against the training corpus with an identifier blocklist and a distributional near-duplicate check. Tables used to train the sector head are kept out of the test set, and the test set was frozen before the final claims.

Does the model keep learning after pre-training?

It can take on new tasks through adapters, a small replay buffer of raw rows, and regularization such as elastic weight consolidation, without retraining from scratch. Retention depends on replay, and the buffer grows linearly, so this is a managed process, not free memory.

What happens after training finishes?

The core, heads, and adapters are merged into one checkpoint of about 2.18 billion parameters, and the weights can be quantized to half precision, 8-bit, or 4-bit for serving, with fidelity measured against the original.

What was the biggest lesson from pre-training?

Check the ruler and the prior before changing the network. Several of the largest gains came from fixing a harness error, a data prior, a seed, or a scale choice, not from adding model capacity.

Summary

Pre-training teaches the model about tables before it sees yours. Masked cell modeling gives it a label-free goal that doubles as imputation.

A mix of real and synthetic tables, with realistic ranges, relationships, and gaps, gives it breadth, and the generator's prior is treated as part of the model. Column identity is zeroed so the model learns from values.

Distributed training with data and tensor parallelism and mixed precision makes the scale practical, heads and adapters add specific outputs, replay and regularization allow later learning, and contamination checks keep results honest. Afterwards the parts are merged into one checkpoint and compressed for serving.

For the network itself, read Architecture, and for what it returns, read Capabilities.