XTab

Overview

XTab is a framework for cross-table pre-training of tabular transformers. It trains one shared transformer body on many different tables at once, while giving each table its own small input layer and output layer. After pre-training, the shared body is copied to a new task, the new task gets its own input and output layers, and the whole thing is fine-tuned on the target table.

The reason this is interesting is that tables from different sources have different columns, types and meanings, so it has long been hard to reuse knowledge between them. XTab shows one practical way: keep the part that is hard to learn, the general table-processing machinery, and swap the parts that depend on the schema. It is an early and clear example of the pre-train and fine-tune pattern for tables, and it sits on the path toward the larger tabular foundation models discussed at /model/research and /model/architecture.

What it is

XTab is a pre-training method and a model design, not a single frozen model for every table. It has three kinds of components.

• Table-specific featurizers. Small modules that turn the columns of one table into tokens. Each pre-training table has its own.
• A shared transformer backbone. The main body of the network, shared by all tables during pre-training. It is the part that carries over to new tasks.
• Table-specific projection heads. Small output modules that map the backbone's output to the training objective of each table.

The backbone is a transformer of the kind used in FT-Transformer. The featurizer is a feature tokenizer, so numeric and categorical columns become tokens of the same size. The heads handle things such as classification or reconstruction for each table.

The core idea in one picture

Think of a group of translators who all work with one shared reasoning expert. Each translator knows one language, which is one table's schema. They turn their language into a common format and hand it to the expert. The expert reasons in the common format and passes the result back. The expert gets better by working with many translators. When a new language shows up, you hire one new translator and keep the same expert.

What it learns

The shared backbone learns general habits for tables: how to relate features inside a row, how to combine numeric and categorical information, how to build row-level summaries that are useful for prediction. These habits are not tied to any specific column name or meaning, so they transfer.

What it is not

XTab does not make a zero-shot predictor. After pre-training you still fine-tune on the target dataset with gradient descent. That sets it apart from in-context learners such as TabPFN and TabICL, which predict without any training on the new table. It also does not use column names as text, which sets it apart from TabSTAR.

Where it came from

XTab was introduced in 2023 in the paper "XTab: Cross-table Pretraining for Tabular Transformers", presented at ICML 2023. The authors came from an applied machine learning group working on automated machine learning for tables.

The paper started from a clear observation. In vision and language, pre-trained models changed everything, because a single model could be reused on many tasks. Tabular data had no equivalent. Each dataset has its own set of columns, its own types and its own scale. Previous attempts at tabular pre-training mostly pre-trained on a single table, as in TabNet and SAINT, which helps a bit with unlabeled rows but does not move knowledge between tables.

Ideas it combined

• Feature-token transformers, shown to be strong in FT-Transformer.
• Shared-body, separate-head training, common in multi-task learning.
• Federated-style training, where a shared model is updated from many separate sources while each source keeps its own private parts. The authors drew on this idea to organize training across tables with different schemas.
• Self-supervised objectives, such as reconstructing masked or corrupted inputs, and supervised objectives, when labels exist.

What the study reported

The paper pre-trained on a collection of public tabular datasets and compared the result with the same transformer trained from random initialization on downstream tasks. In general terms, the pre-trained version did better and trained faster to a good result, which supports the claim that useful structure exists across tables. As always, the size of the gains depends on the dataset, so it is best to read the exact numbers in the paper and in the benchmark pages at /model/benchmarks.

Where it fits in history

How it works

The method has a pre-training phase and a fine-tuning phase. Below is each step in order.

Step 1: Collect many tables

Gather a set of tables with different schemas. They can have different numbers of columns, different types and different targets. Some may have labels, others none. The aim is breadth, so the backbone sees many kinds of structure.

Step 2: Give each table its own featurizer

For every pre-training table, create a feature tokenizer. As in FT-Transformer:

• A numeric column gets a learned vector scaled by the value, plus a bias.
• A categorical column gets an embedding lookup, plus a bias.
• A CLS token is added to gather a row summary.

Every table ends up as a sequence of tokens with the same width, no matter how many columns it had. That common width is what lets one backbone serve all tables.

Step 3: Share one transformer backbone

All tables send their token sequences through the same transformer layers. The backbone holds attention and feed-forward blocks. Because tokens from every table live in the same space, the backbone can learn general rules, such as "tokens that vary together may be related" without knowing what the columns mean.

Analogy: a gym trainer who has worked with athletes from many sports learns general rules of strength, balance and recovery, even though each athlete has a different sport. The trainer is the backbone, and each athlete's sport-specific gear is the featurizer.

Step 4: Give each table its own projection head

The output of the backbone, usually the CLS vector, goes to a head that belongs to that table. The head turns it into whatever the table's objective needs, for example class scores or a reconstruction of masked inputs.

Step 5: Choose the pre-training objectives

Two families of objectives are used.

• Reconstruction. Hide or corrupt some inputs and predict them back. This needs no labels, so unlabeled tables can contribute. It is related to Masked cell modeling.
• Supervised prediction. If a table has labels, predict its target. This ties the backbone to useful prediction patterns.

Using both lets the backbone learn structure from raw cells and from labeled outcomes.

Step 6: Train across tables

Training cycles over the tables. For each batch from a table, the model uses that table's featurizer and head and the shared backbone. The backbone is updated by every table. The featurizers and heads are updated only by their own table. Over time the backbone averages out the quirks of individual tables and keeps what is common.

One practical question is how to update a shared body from sources that disagree. The authors used ideas from federated learning, where updates from different sources are combined into a shared model. This stabilizes learning when tables differ a lot.

Step 7: Move to a new task

For a new target table:

1. Build a new featurizer for its columns, with random initialization.
2. Copy the pre-trained backbone.
3. Build a new head for its target.
4. Fine-tune all parts, or freeze some of the backbone at first and unfreeze it later.

The pre-trained backbone gives a head start. The new model usually needs fewer steps, and often generalizes better, especially when the target table is small.

Step 8: Evaluate and compare

Compare the fine-tuned model with the same architecture trained from scratch, and with tuned trees. If the gain from pre-training disappears when the model is well tuned, you have learned that the target table does not benefit from transfer.

Why it can work even though tables differ

Different tables share abstract structure even when their words differ. Many have a mix of numeric measurements and categories. Many have correlated features. Many have a target that depends on a few key features and some interactions. A backbone that has seen many tables learns to find such patterns quickly. This is like learning to read maps of many cities: each city is different, but the skill of reading a legend and following a route carries over.

Key concepts and terms

A worked example

The following sketch shows the structure of cross-table pre-training in PyTorch. It is simplified to make the roles clear. See PyTorch.

import torch
import torch.nn as nn

class Featurizer(nn.Module):
    """Table-specific: turns numeric columns into tokens of width d."""
    def __init__(self, n_features, d):
        super().__init__()
        self.w = nn.Parameter(torch.randn(n_features, d) * 0.02)
        self.b = nn.Parameter(torch.zeros(n_features, d))
        self.cls = nn.Parameter(torch.zeros(1, 1, d))

    def forward(self, x):                       # (batch, n_features)
        tok = x.unsqueeze(-1) * self.w + self.b
        cls = self.cls.expand(x.size(0), -1, -1)
        return torch.cat([cls, tok], dim=1)

class Backbone(nn.Module):
    """Shared across all tables."""
    def __init__(self, d=64, layers=3, heads=4):
        super().__init__()
        block = nn.TransformerEncoderLayer(d, heads, 2 * d, 0.1,
                                           batch_first=True, norm_first=True)
        self.enc = nn.TransformerEncoder(block, layers)

    def forward(self, tokens):
        return self.enc(tokens)

class TableHead(nn.Module):
    """Table-specific: maps the CLS vector to this table's outputs."""
    def __init__(self, d, n_out):
        super().__init__()
        self.out = nn.Sequential(nn.LayerNorm(d), nn.ReLU(), nn.Linear(d, n_out))

    def forward(self, h):
        return self.out(h[:, 0])

Pre-training loop across tables

backbone = Backbone()
tables = {
    "A": dict(feat=Featurizer(10, 64), head=TableHead(64, 2)),
    "B": dict(feat=Featurizer(25, 64), head=TableHead(64, 1)),
    "C": dict(feat=Featurizer(7, 64),  head=TableHead(64, 3)),
}
params = list(backbone.parameters())
for t in tables.values():
    params += list(t["feat"].parameters()) + list(t["head"].parameters())
opt = torch.optim.AdamW(params, lr=1e-4)

for step in range(10000):
    name = pick_table(step)                      # cycle or sample over tables
    x, y = next_batch(name)
    t = tables[name]
    h = backbone(t["feat"](x))
    loss = task_loss(name, t["head"](h), y)      # supervised or reconstruction
    opt.zero_grad(); loss.backward(); opt.step()

Transfer to a new table

new_feat = Featurizer(n_features=40, d=64)       # fresh, random
new_head = TableHead(64, n_out=1)                # fresh, random
# backbone is the pre-trained one, reused as is

for p in backbone.parameters():                  # optional warm-up: freeze first
    p.requires_grad = False
# train new_feat and new_head for a few epochs, then unfreeze everything
for p in backbone.parameters():
    p.requires_grad = True
# fine-tune all parts with a small learning rate on the target table

Numeric picture

Suppose pre-training used three tables with 10, 25 and 7 columns. Each featurizer outputs sequences of length 11, 26 and 8 including CLS, all with width 64. The backbone sees all three kinds of sequence, so its weights must work for any length and any mix of tokens. A new table with 40 columns gets a featurizer of its own that outputs 41 tokens of width 64. The backbone accepts them without any change, because attention is not tied to a fixed length. After fine-tuning for a few epochs, the model reaches a good score faster than a copy trained from scratch.

A simple experiment to run

1. Train a transformer from random initialization on a small target table. Record validation score per epoch.
2. Train the same model with a pre-trained backbone. Record validation score per epoch.
3. Repeat for table sizes of 500, 5,000 and 50,000 rows.
4. Expect a larger gap at small sizes and a smaller one at large sizes.

Pre-training objectives in more detail

A sketch of the masked-cell loss for a table with numeric columns:

def masked_reconstruction_loss(backbone, feat, head_recon, x, p=0.3):
    mask = torch.rand_like(x) < p
    x_in = torch.where(mask, torch.zeros_like(x), x)   # hide masked cells
    h = backbone(feat(x_in))
    x_hat = head_recon(h[:, 1:])                       # one prediction per column token
    return ((x_hat.squeeze(-1) - x) ** 2)[mask].mean()

The loss is computed only on hidden cells. Without that restriction, the model would be rewarded for copying visible inputs, which teaches nothing.

Why a shared body needs care

Sharing a body across tables is not free. Three issues come up.

1. Scale mismatch. One table may have large losses and another small ones. Without balancing, the large-loss table dominates the updates. Normalize losses or weight them.
2. Sampling imbalance. A table with millions of rows can swamp one with thousands. Sample tables evenly or by a tempered share.
3. Conflicting gradients. Two tables may push the same weight in opposite directions. The federated-style combination of updates and smaller learning rates for the shared body reduce this.

Layer freezing strategies

Reading the transfer curve

Plot validation score against the number of fine-tuning steps, once for the pre-trained start and once for random initialization. Three shapes are common.

• Head start only. The pre-trained curve starts higher but both end at the same point. Pre-training saves time but not accuracy.
• Higher plateau. The pre-trained curve ends higher. Pre-training adds real value on this table.
• No difference or worse. The tables are too unlike the pre-training data, or the target table is large and easy.

Knowing which shape you have tells you whether to keep investing in the pre-training stage.

How many tables are enough

There is no fixed number. Transfer tends to improve with the number and diversity of tables, with diminishing returns. A modest set of diverse tables beats a large set of near-copies. Coverage of data types, row counts, column counts and target types is more important than raw volume.

Building blocks you can swap

• Featurizer. Linear numeric tokens, periodic tokens or binned tokens can all be used, as discussed in FT-Transformer.
• Backbone. A standard transformer, a deeper one, or a variant with row attention as in SAINT.
• Heads. Classification, regression, reconstruction or contrastive projection heads.
• Objective mix. More reconstruction for unlabeled-heavy corpora, more supervised signal when labels are common.

From XTab to larger designs

Larger designs keep the same skeleton and change one part at a time. Some replace the per-table featurizer with a shared one that works on any column by using column statistics. Some add text encoders for names. Some add attention across rows and feed labels into the context. Others scale up on synthetic tables. Seen this way, XTab is a stage in a progression, and its main lesson is that the backbone can be shared once the inputs are mapped into a common token space.

Strengths and limitations

Strengths

• Transfers knowledge between tables. The backbone carries general table skills to new tasks.
• Handles different schemas. Table-specific featurizers and heads absorb the differences.
• Better starts and faster convergence. Fine-tuning often needs fewer steps than training from scratch.
• Uses unlabeled and labeled data. Reconstruction and supervised objectives can be combined.
• Builds on a proven body. The feature-token transformer is well understood.
• Gives a path to larger models. The same pattern can scale with more tables and bigger backbones.

Limitations

• Still needs fine-tuning. It is not a zero-shot predictor, unlike in-context learners.
• Featurizers start from scratch. Each new table needs a new input module, trained on that table alone.
• No column-name understanding. Names are ignored, so semantic hints in headers are lost.
• Gains vary. Large, clean tables with strong signals may see little benefit over a tuned tree model.
• Dependent on pre-training data. If the pre-training tables are very unlike the target, transfer is small or negative.
• Compute at pre-training time. Cross-table training takes more resources than training one small model.
• Tree models remain strong. Tuned XGBoost, LightGBM and CatBoost are still tough baselines.

How it compares with alternatives

The table shows XTab's niche. It moves beyond single-table pre-training, and it keeps the familiar fine-tune workflow. It does not remove the training step, and it does not read names.

Role in tabular foundation models

XTab is one of the clearest examples of the idea behind foundation models for tables: learn general structure from many tables, then reuse it. It is a bridge between per-table neural networks and the in-context learners that predict with no training.

Row and column structure

Inside the backbone, tokens come from columns and the CLS token summarizes a row. Attention runs across columns of one row. XTab does not model dependencies between rows, so it is closest to FT-Transformer in structure. Foundation models that attend across rows as well, as SAINT and in-context learners do, add a second axis. See /model/architecture for how those parts fit together.

Mixed types

The featurizer handles types: numeric columns by scaled vectors and categorical columns by lookups. After that, the backbone sees only tokens, which is how a single body can serve all types. This type-agnostic backbone is a requirement for any foundation model for tables.

Missing values

A reconstruction objective trains the backbone to infer hidden inputs from visible ones. This is directly useful for missing data. If a cell is missing at inference, it can be treated like a masked cell, so the model has seen a similar situation in pre-training. For a deeper treatment, see Masked cell modeling and Imputation networks.

No column names

XTab ignores column names, and each featurizer is tied to one table's columns. That keeps the design simple and avoids any need for text, but it throws away information. A column called "age" and one called "tenure_months" carry meaning that a name-aware model can use. TabSTAR addresses this by embedding names and values with text encoders. When names are missing, as in many raw exports, an XTab-style model is on equal footing, since it never used them anyway.

Pre-training on synthetic or real tables

XTab pre-trains on real tables. That brings realistic distributions, but real public tables are limited in number and variety. TabPFN chose synthetic tables generated from random causal structures, giving effectively unlimited, labeled and clean training data. Each choice has pros and cons: real data matches real-world quirks, synthetic data scales and covers rare structures. Mixed strategies are an active topic, covered in /model/research. Comparative scores are shown at /model/benchmarks.

In-context learning

XTab adapts by fine-tuning weights, not by reading labeled rows as a prompt. In-context learners such as TabPFN and TabICL skip the fine-tuning. A useful way to see the difference: XTab teaches the model "how tables work" and then coaches it on your table, while an in-context model has already been taught "how to learn from a table" and just reads yours. The two can be combined, for instance by fine-tuning an in-context model on a domain.

Continual learning

Fine-tuning a pre-trained backbone on new tables raises the risk of forgetting what it learned before. Standard remedies apply. Elastic weight consolidation protects important weights. Experience replay mixes old samples into new training. Synaptic intelligence and Learning without forgetting offer other options. Because XTab separates the backbone from the table-specific parts, it has a natural form of protection: new tables get new featurizers and heads, so old ones are untouched.

Imputation and anomaly detection

A backbone pre-trained with reconstruction can be used to fill in missing cells and to flag rows that reconstruct badly. Cross-table training can make these skills more general than a model trained on one dataset. For classical options see Isolation forest and Anomaly detection.

Sector and semantic classification

To label a table or its rows by domain, you need both structure and meaning. XTab supplies structure, a representation of rows learned across many tables. A classifier head on top can assign domain labels, similar to Semantic sector heads. Meaning from column names would need an extra text component, as in TabSTAR. When both are present, the combination is stronger than either alone.

Practical guidance and best practices

When to consider XTab-style pre-training

• You have many related tables from the same domain and want to share what they have in common.
• Your target tables are small, so a head start matters.
• You can afford a pre-training stage, and then repeated cheap fine-tuning for each new task.
• You want a neural model and a familiar fine-tune workflow.

When another approach is better

• You have one large table. Direct training of a tuned tree model or a neural network is simpler.
• You need predictions with no training at all. Try an in-context model such as TabPFN.
• Column names matter a lot. Try a text-aware approach such as TabSTAR.

Building a pre-training corpus

1. Collect tables from your domain, with clear licensing and privacy checks.
2. Remove duplicates and near-duplicates across tables, which would inflate scores.
3. Standardize obvious type issues, such as dates stored as text.
4. Keep a set of held-out tables never used for pre-training, to measure transfer honestly.
5. Include a variety of sizes, column counts and type mixes.

Fine-tuning recipe

1. Initialize the backbone from pre-training and the featurizer and head randomly.
2. Warm up the new parts with the backbone frozen for a few epochs.
3. Unfreeze and train all parts with a small learning rate.
4. Use early stopping on a validation split.
5. Compare against training from scratch and against tuned trees.

Hyperparameters

Measuring transfer honestly

• Use held-out tables, not only held-out rows of tables seen in pre-training.
• Report results with and without pre-training under the same tuning budget.
• Include tree baselines tuned equally.
• Repeat with several seeds and report the spread.
• Break results down by table size to see where pre-training helps.

Governance and privacy

When pre-training on many tables, track the origin and permissions of each table. Remove direct identifiers. Check that the backbone does not memorize rare rows, which can happen with small tables and long training. Document which tables went into the model so that results can be reproduced and audited.

Common mistakes

• Leaking test tables into pre-training. Transfer scores are only meaningful on tables the backbone never saw.
• Near-duplicate tables across splits. Different exports of the same data inflate results.
• No scratch baseline. Without it you cannot tell whether pre-training helped.
• Weak tree baselines. Compare with tuned trees, not defaults.
• Using a large learning rate for fine-tuning. It can erase what pre-training provided.
• Unfreezing too early. Randomly initialized featurizers can send noisy gradients into the backbone.
• Mismatched embedding width. The new featurizer must output the width the backbone expects.
• Assuming transfer from unrelated domains. Pre-training on very different tables may give little or negative transfer.
• Ignoring preprocessing consistency. Scaling and encoding must follow the same rules in pre-training and fine-tuning.
• Single-seed claims. Results can vary widely between runs on small tables.

Frequently asked questions

What is XTab?

XTab is a framework that pre-trains a shared transformer backbone across many different tables, using a separate input featurizer and output head for each table. The pre-trained backbone is then reused on new tables by adding fresh table-specific parts and fine-tuning.

How does XTab handle tables with different columns?

Each table has its own featurizer that converts its columns into tokens of a common width, so the shared backbone always sees sequences of vectors of the same size. The number of columns can differ because attention does not need a fixed sequence length.

Is XTab a foundation model for tables?

It is a step toward one. It shows that a backbone can learn general knowledge from many tables and transfer it. However, it still needs fine-tuning and new featurizers for each target table, so it is not a zero-shot model like in-context learners.

How is XTab different from TabPFN?

XTab pre-trains on real tables and adapts to a new table by fine-tuning weights. TabPFN pre-trains on synthetic tables and predicts on a new table in a single forward pass, using the labeled rows as context, with no gradient training on the target data.

Does XTab use column names?

No. It treats columns by position through table-specific featurizers and ignores header text. That keeps the method simple, but it means semantic information in the names is not used, which text-aware models such as TabSTAR try to recover.

What objectives does XTab use for pre-training?

It combines reconstruction objectives, where hidden or corrupted inputs are predicted, with supervised objectives on tables that have labels. This lets both unlabeled and labeled tables contribute to the shared backbone.

Does pre-training always help?

No. It tends to help most on small tables and on tables similar to the pre-training data. On large tables with strong signals, a well-tuned model trained from scratch, or a tuned tree ensemble, may do as well. Always run a scratch baseline to check.

Can I use XTab for imputation or anomaly detection?

Yes in principle, since the reconstruction objective trains the backbone to predict hidden cells. You can use that to fill missing values and to score rows by reconstruction error, but you should validate on your own data because quality varies across tables.

Why use a shared backbone with separate heads?

Separate featurizers and heads absorb everything that is specific to a table, such as column count and target type. That leaves the backbone free to learn what is common, so it can be reused. It is the same logic as multi-task learning in other fields.

Does XTab replace gradient boosted trees?

Not in general. Tree ensembles like XGBoost and LightGBM remain strong and cheap baselines on tables. XTab is most useful when you have many related tables, small target tables or a need for a neural backbone that can be reused and extended.

Glossary

• Backbone: the shared network body reused across tasks.
• Cross-table pre-training: pre-training on many tables with different schemas.
• Featurizer: a module that turns a table's columns into tokens.
• Fine-tuning: continuing training on a target task after pre-training.
• Frozen: not updated during training.
• Head: the final layers that produce an output for a task.
• Initialization: the starting values of weights.
• Multi-task learning: training one model on several tasks that share parts.
• Negative transfer: when pre-training hurts a new task.
• Projection head: a small output module specific to a table or objective.
• Reconstruction: predicting hidden or corrupted inputs.
• Schema: the structure of a table, meaning its columns and types.
• Self-supervised learning: learning from signals made from the data itself. See Self-supervised learning.
• Transfer learning: reusing knowledge learned on other tasks. See Transfer learning.
• Zero-shot: predicting on a new task with no task-specific training.

Summary

XTab pre-trains one shared transformer backbone across many tables, with table-specific featurizers at the input and projection heads at the output. After pre-training, the backbone moves to a new table, which gets its own new featurizer and head, and the whole model is fine-tuned. Reconstruction and supervised objectives together give the backbone general habits for reading tables.

It matters because it was an early, concrete demonstration that tables of different shapes can share knowledge, and it keeps the familiar fine-tune workflow. It still needs training for each new task and ignores column names, so it differs from in-context learners like TabPFN and TabICL and from text-aware models such as TabSTAR. Compare it with tuned trees like XGBoost, with the backbone it builds on in FT-Transformer, and with single-table methods like SAINT and TabNet. For the wider picture, see /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Related models and baselines in my stack, under Foundation models and applied ML. Others in the same group: