Large tabular foundation models

Overview

A tabular foundation model is a neural network pretrained on a very large number of tables, real or synthetic, so that it can handle a new table it has never seen. Instead of training a fresh model for each dataset, you give the pretrained model your labelled rows and ask it about new rows. Many such models do this purely by in-context learning: the training rows are part of the input, and no weights are updated. Others are fine-tuned briefly or adapted with a small head.

This page is a technical guide rather than an introduction. It covers why tables resist deep learning, how models turn rows and cells into vectors, the two pretraining strategies (synthetic priors versus real tables), the main architectures, the outputs beyond classification, the context-size limits, and how these models are benchmarked against gradient-boosted trees. For a gentler start, read what is a tabular foundation model.

An everyday picture: an experienced auditor handed an unfamiliar company's ledger does not need years of training on that company. They have seen thousands of ledgers, so they read the columns, look at a few dozen examples and quickly spot what predicts what. A tabular foundation model tries to be that auditor: general experience across many tables, applied to a new one from a handful of examples.

What it is

Two design axes separate most models. The first is whether the model depends on column names. Column-name-independent designs treat a column purely by its values, so they work on anonymised or coded data; column-aware designs read names and text, which helps when meaning lives in the words. The second is whether the model predicts by in-context learning, by fine-tuning, or both.

Where it came from

Deep learning on tables struggled for years. Two 2022 studies made the gap explicit. Shwartz-Ziv and Armon's "Tabular Data: Deep Learning Is Not All You Need" found tuned boosted trees usually ahead of the deep models of the time. Grinsztajn, Oyallon and Varoquaux's NeurIPS 2022 paper "Why do tree-based models still outperform deep learning on typical tabular data?" traced the gap to three properties of tables (see below).

Architectures adapted to tables appeared in parallel. TabNet (Arik and Pfister, 2019) used sequential attention for feature selection. TabTransformer (Huang and colleagues, 2020) gave categorical features contextual embeddings. VIME (Yoon and colleagues, 2020) used mask-and-reconstruct self-supervision. In 2021, SAINT (Somepalli and colleagues) added attention across rows as well as columns, and FT-Transformer (Gorishniy, Rubachev, Khrulkov and Babenko) showed a plain Transformer over feature tokens is a strong baseline.

The foundation-model turn came from prior-data fitted networks. Müller and colleagues' ICLR 2022 paper "Transformers Can Do Bayesian Inference" trained a Transformer on samples from a prior so that it approximates Bayesian prediction in one forward pass. TabPFN (Hollmann, Müller, Eggensperger and Hutter, ICLR 2023) applied this to small classification tables, up to about 1,000 training rows, 100 features and 10 classes. A much larger successor was published in Nature in January 2025, handling up to 10,000 rows and 500 features, regression as well as classification, and reporting that in 2.8 seconds it outperformed an ensemble of strong baselines tuned for 4 hours.

Cross-table and text-aware work grew alongside. XTab (Zhu and colleagues, ICML 2023) pretrained a shared Transformer across tables with dataset-specific featurisers, evaluating on 84 tasks from the OpenML AutoML Benchmark. CARTE (Kim, Grinsztajn and Varoquaux, ICML 2024) represented rows as small graphs of column names and values. TabICL (Qu, Holzmüller, Varoquaux and Le Morvan, ICML 2025) scaled in-context learning to much larger tables. TabDPT (2024) pretrained on real tables rather than synthetic ones. TabSTAR (Arazi, Shapira and Reichart, 2025) targeted tables with free-text fields. TabArena (Erickson and colleagues, NeurIPS 2025 Datasets and Benchmarks track) became a maintained, living benchmark.

How it works

Why tables are hard for deep learning

Grinsztajn and colleagues identified three reasons. First, the target functions in real tables are often irregular, with sharp thresholds, and neural networks are biased towards smooth functions, while trees naturally cut at thresholds. Second, tables often contain many uninformative columns, which hurt multilayer perceptrons more than trees. Third, columns have individual meaning, so the problem is not rotation-invariant; a model that treats any rotation of the feature space as equivalent, as a plain MLP effectively does, throws away useful structure. On top of this, tables mix numbers, categories, dates and text, have missing values, and are often small, so there is little data to learn representations from scratch. Pretraining across many tables attacks exactly the small-data problem.

Cell and row representations

Most modern models give every cell its own vector. A numeric value is usually normalised and passed through a learned map; some models also use quantile binning or periodic embeddings so the network can represent sharp jumps. Categories get learned embeddings or are converted to numbers. In column-aware models, the column name and string values pass through a pretrained text encoder, so "annual_income" and "salary" land near each other. A row is then a set of cell vectors. Attention across a row's cells (column attention) builds a row representation; in TabICL this happens first, compressing each row into a fixed-size embedding before rows interact.

Column-name-independent versus column-aware designs

A column-name-independent model sees only values and their statistical relations. It works on coded or anonymised data and cannot be misled by a misleading header, but it cannot exploit meaning in names. A column-aware model can transfer knowledge such as "age is usually positively related to blood pressure" across tables, and handles text cells naturally, but may rely on names that are absent, cryptic or wrong. TabPFN-style models are largely value-based; TabSTAR and CARTE read text. Some systems combine both.

Pretraining on synthetic priors

A prior-data fitted network is trained on millions of artificial datasets drawn from a designed prior: random structural causal models, random small networks, random trees, noise and missingness processes. For each synthetic dataset, some rows are labelled context and others are queries, and the model learns to predict the query targets. Done well, the network learns to approximate the posterior predictive distribution under that prior.

Posterior predictive (what a PFN approximates):
    p(y* | x*, D) = ∫ p(y* | x*, φ) · p(φ | D) dφ

PFN training objective over synthetic tasks:
    L(θ) = E_{D ∪ {(x*, y*)} ~ prior} [ − log q_θ(y* | x*, D) ]

Row-attention cost for n context rows and m query rows:
    cost ≈ O(n² + n · m) per layer, per feature group

Here D is the set of labelled context rows, x* a new row and y* its unknown target, φ the hidden mechanism that generated the data, and p(φ | D) how plausible each mechanism is given the context. In the training objective, θ are the network weights, q_θ is the network's predicted distribution, and the expectation runs over datasets sampled from the prior. Minimising this loss makes q_θ approach the true posterior predictive for data like the prior. The last line says why context size matters: when context rows attend to each other, cost grows with the square of the number of context rows n.

The strength of this approach is unlimited, clean, labelled data with known structure. Its weakness is that real tables may differ from the prior, so prior design is the central engineering problem.

Pretraining on real tables

The alternative is to pretrain on large collections of real tables. This exposes the model to genuine column semantics, messy distributions and real text. It is harder: good tables are limited, licences vary, labels are inconsistent, and benchmark datasets must be removed carefully to avoid contamination. XTab, CARTE, TabDPT and TabSTAR all draw on real data. Many teams now mix both sources.

In-context learning on tables

At prediction time, training rows with their labels and test rows without labels are fed together. Attention lets each test row look at the training rows and their targets. No gradient step is taken. To respect the fact that row and column order carry no meaning, models avoid positional encodings across rows, and inference often averages over several random column orders, feature transforms or class relabellings, a cheap ensemble. See in-context prediction for a detailed walk-through.

Attention over rows and columns

SAINT alternates attention across columns within a row and attention across rows within a batch ("intersample attention"). FT-Transformer uses only column attention plus a summary token. TabPFN's larger version uses two-way attention per cell: each cell attends to other cells in its row and to the same column in other rows. TabICL separates the two stages: column-then-row attention builds row embeddings, then a Transformer does in-context learning over rows. These choices trade expressiveness against memory.

Masked cell modelling

Masked cell modelling hides random cells and asks the model to reconstruct them from the rest of the row and from other rows. It is the tabular counterpart of masked language modelling and gives a natural pretraining signal that does not need a target column. It also makes the model an imputer for free. See masked cell modelling for the objective in detail.

Outputs beyond classification

For regression, several models predict a full distribution rather than a single number. A common choice divides the target range into many bins and predicts a probability for each, a piecewise-constant "bar distribution" introduced with PFNs. This gives medians, quantiles and intervals, and is scored with proper scoring rules such as log-likelihood or the continuous ranked probability score. Imputation predicts missing cells, ideally with uncertainty. Anomaly detection can use how surprising a row or cell is under the model, for example its negative log-likelihood or reconstruction error.

Context-size limits and scaling

Because attention over context rows is quadratic, memory caps how many rows fit. The original TabPFN worked with about 1,000 rows; the 2025 version with about 10,000; TabICL is pretrained on synthetic datasets of up to 60,000 rows and reports handling up to 500,000 rows on affordable hardware. Practical strategies for larger tables include subsampling the context, retrieving the nearest training rows for each query, splitting the data and ensembling, and compressing rows before row attention. Width matters too: hundreds of columns increase cost, and very wide tables may need feature selection or grouping.

How they are benchmarked

Common suites include OpenML-CC18 (72 classification datasets), OpenML-CTR23 (35 regression datasets, 2023), the Grinsztajn benchmark, TabZilla (McElfresh and colleagues, 2023) and the TALENT collection. TabArena curates 51 datasets from 1,053 candidates drawn from 14 earlier benchmarks, with training sets between 500 and 250,000 rows, and ranks 16 models by Elo computed from ROC AUC (binary), log-loss (multiclass) and RMSE (regression). Its published findings include that boosted trees remain strong, that tuned and ensembled deep models have caught up, that foundation models perform best on smaller datasets, and that validation protocol and ensembling strongly affect results. See benchmarks for how this site's model documentation reports scores.

Key concepts and terms

A worked example

An illustrative clinic has a table of 2,400 patients with 18 columns (age, lab values, medication codes, a smoking category) and wants to predict a binary outcome. Fifteen per cent of lab values are missing.

1. Split 80/20 by patient into 1,920 context rows and 480 test rows.
2. Baseline: a gradient-boosted model with five-fold tuning over 50 configurations. Test ROC AUC is about 0.82 after roughly 20 minutes of tuning.
3. Foundation model: pass the 1,920 labelled rows as context and score the 480 test rows in one call, with no tuning. Missing values are handled natively. Test ROC AUC is about 0.84, with better-calibrated probabilities.
4. Imputation: ask the same model to reconstruct the missing lab values, returning a distribution for each, which a clinician can inspect.
5. Check: repeat both methods over five random splits. The gap is smaller than the spread on two of five splits, so the honest conclusion is "comparable or slightly better", not a clear win.

The numbers are illustrative. A minimal in-context call with the open TabICL package looks like this:

from tabicl import TabICLClassifier

clf = TabICLClassifier()          # pretrained weights load on first use
clf.fit(X_context, y_context)     # stores the context; no gradient training
proba = clf.predict_proba(X_test)

The scikit-learn style fit here mainly stores and preprocesses the context; the learning happens inside the forward pass.

Strengths and limitations

Strengths:

• Strong accuracy on small and medium tables with no per-dataset training or tuning.
• Fast to apply: seconds instead of hours of hyperparameter search.
• Often well-calibrated probabilities and full predictive distributions for regression.
• One model can serve classification, regression, imputation and anomaly scoring.
• Text-aware variants use column names and free text that trees ignore.

Limitations:

• Context size is bounded by memory; very large tables need subsampling or retrieval.
• Inference is heavier than a trained tree model, often needing a GPU.
• Performance depends on how well the pretraining prior or corpus matches your data.
• Less transparent than trees, though attribution methods can be applied.
• On large datasets, tuned gradient-boosted trees often remain as good or better and cheaper to serve.
• Benchmark claims are sensitive to protocol; contamination and validation leakage are real risks.

How it compares with alternatives

A sensible default in practice: try a foundation model and a tuned boosted model, compare them across several splits, and consider ensembling the two, which benchmarks suggest often beats either alone.

Practical guidance and best practices

1. Check the model's documented limits on rows, columns and classes before you start.
2. Always compare against a properly tuned gradient-boosted baseline on the same splits.
3. Evaluate over several random or time-based splits and report the spread, not one number.
4. Keep preprocessing light; many models handle missing values and scaling internally.
5. For large tables, try retrieval of similar rows as context or several subsampled contexts averaged together.
6. Choose a column-aware model when headers or text carry meaning, and a value-only model for coded data.
7. For regression, use the predictive distribution and score it with proper scoring rules.
8. Watch for leakage: drop identifier columns and anything recorded after the outcome.
9. Measure inference latency and memory early if the model will run in production.

Common mistakes

• Comparing a foundation model against an untuned tree baseline and calling it a win.
• Feeding far more rows than the model was designed for and blaming the model for poor results.
• Leaving ID or timestamp columns in the context, which the model can exploit spuriously.
• Reporting a single split where differences are within noise.
• Assuming results on public benchmarks transfer unchanged to a very different domain.
• Ignoring the predictive distribution and using only point predictions for regression.

Frequently asked questions

Do tabular foundation models replace gradient-boosted trees?

Not across the board. Current evidence shows them strongest on small and medium tables, while tuned boosted trees remain excellent and cheap on large ones. Many practitioners use both and ensemble them. The right choice depends on table size, latency budget and whether you need distributions or imputation.

What does "pretrained on synthetic data" mean?

The model never sees real tables during pretraining. Instead it sees millions of artificial tables generated from random causal graphs, random functions and noise models. Learning to predict across all of them teaches a general procedure for inferring patterns from labelled rows, which then transfers to real data that resembles the prior.

Why is context size limited?

Context rows attend to each other, so memory and compute grow roughly with the square of the number of rows. Doubling the rows roughly quadruples the cost of that attention. Architectural tricks, such as compressing rows first, and practical ones, such as retrieving relevant rows, push the limit but do not remove it.

Can these models use column names?

Some can. Column-aware models read names and text values with a language encoder, which helps transfer meaning between tables. Column-name-independent models ignore headers and rely on values, which is safer for coded or anonymised data.

How are they evaluated fairly?

Use curated benchmark suites, tuned baselines, several splits, and metrics that match the task: ROC AUC or log-loss for classification, RMSE or proper scoring rules for regression. Check that benchmark tables were not in the pretraining data. Maintained benchmarks such as TabArena publish protocols and leaderboards to make comparisons consistent.

Can I fine-tune a tabular foundation model?

Several models support fine-tuning, either on one large table or on a family of related tables. It can help when the data differs from the pretraining distribution or exceeds the context limit. Treat it like any fine-tuning: start from pretrained weights, hold out data, and watch for overfitting.

Glossary

• Tabular foundation model: a model pretrained across many tables for reuse on new ones.
• PFN: a prior-data fitted network trained on synthetic datasets.
• Structural causal model: a random graph of cause-and-effect rules used to generate synthetic tables.
• Context: the labelled rows passed with the query at prediction time.
• Intersample attention: attention between different rows, as in SAINT.
• Feature tokeniser: the layer that turns each feature into a vector, as in FT-Transformer.
• Cross-table transfer: reusing knowledge learned from other tables.
• Bar distribution: a binned predictive distribution for a numeric target.
• CRPS: a proper scoring rule for predictive distributions of numbers.
• Living benchmark: a benchmark that is maintained and updated with new models.

Summary

Tabular foundation models bring pretraining to tables, the data type where deep learning long trailed boosted trees. They represent each cell as a vector, build row representations with attention across columns, and let test rows learn from training rows through attention across rows, often by pure in-context learning. Pretraining comes either from synthetic tables drawn from a designed prior, which approximates Bayesian prediction, or from large collections of real tables, which bring genuine semantics and text; many systems mix both. Beyond classification they produce regression distributions, imputations and anomaly scores. Their main constraint is context size, set by quadratic attention over rows. Benchmarks such as TabArena and the OpenML suites show them strongest on small and medium tables, with tuned gradient-boosted trees still excellent on large ones, so careful comparison and ensembling remain the practical path.

Related

It belongs to Large tabular foundation models in my stack, under Deep learning. Others in the same group: