Tabular foundation models

A tabular foundation model is a neural network pre-trained once on a very large and varied collection of tables, so that it already knows how structured data behaves. At use time it reads your table as it is, and it makes predictions or returns understanding without being trained on that table.

That is the whole idea in two sentences. The rest of this page explains what it replaces, what the alternatives are, how the main families differ, and why the last family, models that read raw cell values directly, is the one this site is about.

If you are new to the topic, read the first three sections and the landscape table. If you already know the field, jump to the raw-table argument and the section on where this model sits.

The problem: tables are everywhere and awkward

Most of the data that runs organizations lives in tables. Orders, patients, sensors, transactions, accounts, shipments, claims, tickets. Rows are records and columns are attributes.

Text has language models. Images have vision models. Audio has audio models. Each of those modalities has a foundation model that understands it natively. For a long time, tables did not.

Instead, the standard workflow around a table looked like this: a person cleans the data, fills the gaps, rescales the numbers, encodes the categories, engineers new features, picks a model, tunes it, and trains it on that one table. When the next table arrives, the whole process starts again.

The paper behind the model on this site describes this as a bottleneck. Preprocessing consumes most of the effort of an AI project, commonly cited at 60 to 80 percent, pipelines break when upstream data changes, each decision throws information away, and the work takes domain, statistical and engineering knowledge together. See Research.

Why tables are hard for neural networks

Tables look easy. They are tidy grids. Five properties make them hard for a neural network to read.

• Heterogeneous types. One table mixes continuous numbers, categories, dates, currencies, identifiers and free text. Each has different statistics and meaning.

• Semantic ambiguity. A column called value can hold money, a sensor reading or an abstract score. Without outside knowledge the meaning is underspecified.

• Variable structure. Images have a fixed pixel grid and text has one axis. Tables vary in both rows and columns, and neither axis has a natural order.

• Missing data. Values go missing through collection errors, privacy rules or inapplicability, and the pattern of what is missing often carries information.

• Implicit relationships. Columns relate in ways their values do not show.

A foundation model for tables has to cope with all five at once, in one network.

What makes a model a foundation model

The phrase foundation model has a specific meaning. It is not simply a big model.

A foundation model is trained on broad data, usually with a self-supervised goal, so that it learns general structure. It is then adapted to many tasks with little or no extra training. The adaptation is cheap, and the general knowledge is reused.

Applied to tables, four properties define the class.

1. Pre-trained once, on many tables. The model has seen a great variety of tables before it sees yours.

2. Weights fixed at use time, or nearly. For a prediction, nothing is trained on your table. The model reads the labelled rows as context.

3. One model, many tables. The same weights serve a table with six columns and a table with sixty, and tables from different industries.

4. More than one task. The same body supports prediction, imputation, understanding and matching, through small heads.

Compare it with the usual approach. A gradient-boosted tree is trained from scratch for each table, and it knows nothing about any other table. A tabular foundation model arrives with knowledge and applies it.

An everyday analogy

Think of two ways to staff a clinic. In the first, you hire a person with no medical training for each new clinic and train them on that clinic's patients. In the second, you hire a doctor who has seen thousands of patients across many clinics, and you hand them the new clinic's short charts.

The second doctor does not start from zero. They recognize patterns, and they need only a few local examples to adjust. A tabular foundation model is the second doctor.

How a tabular foundation model works

Three ideas carry most of the weight. They appear in every serious model of the class, in different forms.

Idea one: pre-train on a prior over tables

There are not enough real, public tables to teach a large network what tables look like. So most tabular foundation models are pre-trained on large numbers of synthetic tables, often together with real ones.

A synthetic generator is a recipe that produces random tasks. It encodes beliefs about how columns relate, how noise behaves and how targets depend on features. After pre-training, the network has absorbed the recipe. That set of beliefs is called the prior. See synthetic data generation.

The generator is not neutral, so the prior is part of the model. If it has a flaw, the model inherits the flaw. I met this directly, when the training prior turned quantile-band targets into XOR-like patterns and nine different decoders failed in the same way. See Pre-training.

Idea two: learn by hiding cells

The self-supervised goal that suits tables is simple. Hide cells and ask the model to recover them from the rest of the table. The table is its own teacher, and no labels are needed. See masked cell modeling and self-supervised learning.

Recovering a hidden cell forces the model to use two kinds of evidence, the other cells in the row and the other values in the column. It also makes imputation a natural by-product of training, because filling a gap is exactly the training task.

Idea three: predict in context

At use time the labelled rows are placed in the model's input as examples, and the model reads off an answer for new rows by comparing them with those examples. No weights change. This is in-context learning, and it is the reason a prediction takes seconds and not an hour of tuning. See in-context prediction and few-shot learning.

Together, these ideas give the class its character. A broad prior supplies general knowledge. Masked-cell training makes the knowledge usable. In-context prediction applies it to your table.

The landscape of approaches

Tabular machine learning has several families, and they are not rivals in every case. Each has a place. Here they are, in roughly historical order.

Family one: classical models trained per table

Gradient-boosted decision trees and random forests are the workhorses of tabular prediction. They fit many small decision trees and combine them. The best-known implementations are XGBoost, LightGBM and CatBoost, and the older ensemble method is the random forest.

They are fast, robust and strong on many tables. They need a preprocessing pipeline around them, and they are trained from scratch for each table and each task.

They carry no knowledge from one table to another. A tree ensemble trained on one retailer's orders knows nothing about another retailer's.

Their strength is real. On large tables with plenty of labels and hand-made features, a tuned tree ensemble is still very hard to beat, and any honest comparison keeps that in view.

Family two: AutoML stacks

AutoML systems automate the search. They try several models, tune settings, build ensembles and stack them, often within a time budget.

A stack of tuned models is a strong baseline, and on many benchmark boards the second place behind a foundation model is a tuned AutoML pipeline. The price is time, because hours of tuning per dataset is normal.

AutoML does not remove preprocessing. It automates the model choice, and the data still has to be shaped to fit.

Family three: deep networks trained per table

Neural networks designed for tables, such as TabNet, choose which features to look at and are trained end to end with gradient descent. They offer built-in feature selection and interpretability, and they can be pre-trained in a self-supervised way.

They are still trained per table, and for years they struggled to beat boosted trees on typical tabular data. Their importance is as stepping stones. Many ideas in later models, such as attention over features and self-supervised pre-training on tables, came out of this family.

Family four: prior-fitted, in-context tabular models

This is where the first true tabular foundation models appeared. TabPFN is a transformer pre-trained on millions of synthetic tables so that it predicts on a new table in one forward pass. The labelled rows are the context, the unlabelled rows are the queries, and no training happens on your data.

The first version handled small classification tables. A later generation extended the approach to regression, missing values, categorical features and larger tables, and it let attention run across rows and across columns.

TabICL pushed the same direction toward larger tables. It splits the work into stages, first summarizing each column, then turning each row into a compact vector, and only then doing in-context learning over those row vectors.

This family established the central trick: pre-train on a prior and predict in context. Its inputs are typically already cleaned and encoded numbers.

Family five: tabular models with a language component

Some tables contain free text beside numbers and categories. A product table has a title and a description. A job table has a posting.

TabSTAR reads such text with a pretrained language encoder, turns every cell into a meaningful vector and reasons over those vectors. It is pre-trained on real tables and is typically fine-tuned on the target table, so it sits between in-context use and per-table training.

The language component brings world knowledge, such as the fact that two different product names mean nearly the same thing.

Family six: large language models on serialized tables

A tempting route is to turn each row into a sentence and give it to a large language model.

This works at small scale and for tables that are mostly text.

• Joint structure is lost. Flattening discards the way columns relate across rows. The model must rediscover, from a string, that two numbers belong to the same record and that a column is a column.

• Numbers become pieces of text. Every digit is paid for as a separate token, and numeric closeness is not represented well.

• Length explodes. A table of many rows and columns becomes a very long text.

• Order matters when it should not. Rows and columns have no order, and a text version forces one.

The paper behind the model calls the serialization route a bottleneck of its own. It replaces preprocessing with another transformation, and the transformation loses information.

Family seven: data language models

A Data Language Model reads the table itself, directly from raw cell values. There is no serialization into text and no preprocessing pipeline. Cells become vectors, attention runs along rows and columns, and a small set of latent vectors summarizes the whole table.

The name is an analogy with language models, and the analogy is exact in one respect. A language model learns the structure of language. A Data Language Model learns the structure of tables. See Research and Architecture.

The first model of this class had 140 million parameters and was trained on more than 2.3 million synthetic and real-world datasets. The model on this site is its successor, with more than 1.5 billion parameters.

The families side by side

The table simplifies. Real systems vary, and some models blend traits. It is still a useful map of where each family puts its effort.

When each family is the right choice

No family wins everywhere. These are the rules of thumb I would give a colleague.

• Large table, many labels, stable schema, time to tune. A boosted tree, or an AutoML stack, is a safe and strong choice.

• Small or medium table, few labels, no time. A prior-fitted in-context model is attractive, because it needs no tuning and gives a good answer in seconds.

• Many tables with unknown or messy columns. A model that reads raw values and does not depend on names removes the mapping work.

• A table dominated by text. A model with a language component, or a language model, may use the text better.

• A need for more than prediction. If you also want the industry of the table, the role of each column, the filled gaps, or matches across tables, a model with several heads on one body fits.

• A stream of new tasks. A model with an update path, such as adapters with replay, avoids retraining everything. See Continual learning for tables.

The sensible habit is to compare on your own split. See Evaluation.

Why reading the raw table matters

Here is the argument at the heart of this site. It has five parts.

Part one: every preprocessing step discards something

Cleaning, imputing with a mean, scaling, encoding categories as numbers, and building features all make choices. Each choice throws some information away, and the losses compound.

A mean fill erases the fact that a value was missing, and that fact may be informative. A one-hot encoding loses the similarity between related categories. A min-max scaling loses the absolute scale. None of this is wrong in a given case. It is lossy in general.

A model that reads the raw cell sees what was there before anyone decided what mattered.

Part two: pipelines are brittle

A preprocessing pipeline encodes assumptions about a particular table. When an upstream system adds a new category, changes a format or shifts a distribution, the pipeline breaks or silently does the wrong thing.

A model that reads raw values and learned general rules of tables is less tied to one schema. A column that holds a new kind of value is read as a column of values.

Part three: the meaning is in the values and the relations

Think of a detective handed a spreadsheet with all the headers torn off. A good detective can still tell which column is a phone number, which is a date of birth and which is a price, from the shape of the values and from how the columns move together.

That is how this model reads. Column embeddings are always zeroed, so the encoder never sees a header. This is why a column called c3 that holds phone numbers is understood as phone numbers, and why random or missing names change nothing. See Column name independence and Metadata independence.

Why does this matter? Because names are the least reliable part of many real tables. Legacy exports have codes like val_B and metric_14. Privacy rules strip headers. Two systems call the same thing by different names, or call different things by the same name.

A model that leans on names inherits all of that unreliability. A model that reads values does not.

Part four: missing cells are inputs, not errors

In real tables the pattern of what is missing carries information.

A model that reads raw cells can encode a missing cell explicitly, as its own kind of input. It then reads around gaps and treats a gap as a possible signal. It does not need a separate imputation step before anything else can run.

That also explains why imputation is not bolted on. Masked cell modeling is the training task, so filling a gap is the natural act of the model. See Pre-training.

Part five: tables relate to tables

A lot of the value in real data lies between tables. The same customer sits in five systems under five identifiers. The same column is called three things.

A model that has learned what columns and rows look like can compare them across tables without shared keys. That is how cross-table matching works. Each match carries a confidence and a verbatim example from each side, and it goes to a review queue.

None of this works if the data first has to be flattened into a form that destroys the structure the comparison needs.

What the argument does not claim

I want to be careful here. Reading the raw table does not make the model correct everywhere. It does not replace a tuned tree ensemble on a large, clean table with many labels. It does not turn a biased table into a fair one.

The claim is narrower. For the common case in which you have many tables, messy columns and few labels, a model that reads raw cells removes a large amount of work and avoids a class of silent errors. The Limitations page lists the cases where it does not help.

The architecture that makes raw reading possible

Raw reading is not magic. It needs a network designed for grids. This section gives the short version, and the Architecture page gives the long one.

Cells become vectors

Every cell is mapped to a vector that combines its value, its type and its position. Numbers, categories, dates and text all land in the same space, so one network can read them together. See Cell encoding.

Attention alternates along rows and columns

Axial attention splits the work into two cheaper pieces, one along each axis. A row layer lets the cells of one record find each other. A column layer lets the values of one attribute find each other.

Stacked, the layers let information travel between any two cells within a couple of steps. See axial attention.

Latents give a view of the whole table

A Perceiver-style block lets a small fixed set of latent vectors read all the cells and write back to them. The latents are what make table-level questions possible, such as which industry a table belongs to. See Perceiver IO.

Small heads, one body

The body is shared, and small heads read from it. One reads the missing cells, one names sectors and column roles, one predicts, and one matches across tables. Because they share a body, a pass over a table returns one bundle of outputs.

What the class can do

A tabular foundation model of this kind returns more than a prediction. These are the outputs this model produces in one pass. Each is covered in Capabilities.

The sector output is unusual. No earlier tabular model returns it, because earlier models need a person to say what domain they are looking at.

Where this model sits

The model on this site belongs to the data language model family. It has more than 1.5 billion parameters, and once every head and adapter is merged with the core into one checkpoint the total is about 2.18 billion.

Here is how it differs from the other families, in plain terms.

• It does not depend on names. Column embeddings are zeroed.

• It treats missing cells as inputs. Gaps are encoded, not cleaned away.

• It is more than a predictor. Understanding, imputation, matching and prediction come out of one pass.

• It can learn new tasks without a full retrain. Adapters that do not depend on the number of columns, replay of raw rows and regularization let it take on new tasks. See Continual learning for tables.

• It ships as one object. One checkpoint, which can also be compressed. See Compression and serving.

On the benchmarks it was measured against strong rivals, including hand-tuned AutoML pipelines and published tabular foundation models, from one sealed configuration. The results and their caveats are on the Benchmarks pages. Regression and multiclass boards are behind the leaders, and I say so.

How the class is evaluated

A tabular foundation model is judged at two levels. On your own table, a held-out split shows how it handles unseen rows. On public benchmarks, it is compared with strong baselines under each benchmark's protocol.

Because the class makes broad claims, the evaluation has to be careful about leakage and fairness. A model pre-trained on millions of tables might have seen a benchmark table, so contamination is checked with an identifier blocklist and a near-duplicate check. A model that runs one default configuration is compared with rivals that are tuned for hours, and that fact is stated.

Read Evaluation before you trust any number, including mine.

A short history of the idea

The class did not appear overnight. Four steps led to it.

1. Prior-data fitted networks. A 2021 paper showed that a transformer trained on data sampled from a prior can approximate Bayesian prediction in a single forward pass. The idea was general, and tables were one place to use it.

2. A first tabular model. In 2022 a transformer was applied to small tabular classification problems, solving them in about a second without training on the target table. It was limited to small numeric tables, and it already matched strong baselines on many small benchmarks.

3. A second generation. A later article in 2025 described a model that handled regression, missing values, categorical features and larger tables, and that ran attention across both rows and columns.

4. Scale and breadth. Later work pushed toward larger tables, text-rich tables, and, in the data language model line, toward reading raw cells with no preprocessing and returning understanding as well as predictions.

The pattern is a familiar one in machine learning. A narrow, clever demonstration is followed by a more general system that copes with the messy properties of real data.

One task, three ways

A concrete comparison makes the families easier to hold in mind. Take a simple job. You have a table of customers with a column that says whether each one churned, and you want to predict churn for new customers.

The boosted tree way

You clean the table and fill gaps. You encode the categories and perhaps build features such as tenure in months. You choose settings, run a search, train, check on a validation split and pick the best setting.

The result is a strong model for this table. It has no idea what a customer table is, and it does nothing for the next table.

The in-context way

You give the labelled customers and the new customers to a prior-fitted model together. It reads them in one forward pass and returns predictions for the new ones.

There is no search and no training. The cleaning step is often still on you, because such models usually expect numbers that are already encoded.

The raw-reading way

You give the table as it is, with missing cells, mixed types and whatever headers it has. The model encodes each cell with its value, type and position, reads along rows and columns, and uses the labelled rows as context.

You get predictions, and the same pass also returns a profile of each column, the missing cells filled with a confidence, and the industry of the table. A held-out block says how well it did on rows it did not see. The point is that the cleaning step has moved inside the model.

The table is a sketch and not a measurement. Real workflows differ, and a good analyst can make the first column shorter. It shows where the effort sits in each family.

Vocabulary of the class

A few terms recur. Here they are, in plain words.

• Prior. The set of beliefs about what tables look like that is built into the data generator and absorbed by the model.

• Support set. The labelled rows the model reads as context before predicting other rows.

• Query rows. The rows the model is asked to predict.

• Masked cell modeling. Hiding cells and training the model to recover them.

• Axial attention. Attention along one axis at a time, rows and then columns.

• Latents. A small fixed set of vectors that summarize the whole table.

• Head. A small module on top of the shared body that produces one kind of output.

• Adapter. A small trainable module attached to the body for one task.

• Sealed configuration. The one fixed setting used for a benchmark, with no per-dataset tuning.

• Contamination. Evaluation data that also appears in the training corpus.

What to ask of any tabular foundation model

Claims in this field vary in quality. These questions help separate solid ones from loose ones, for my model and for any other.

• What does it read? Raw cells, encoded numbers, or text? What preprocessing does it assume?

• What was it trained on? How much of that was synthetic, and what is known about the prior?

• How was it compared? Were the rivals tuned and was the model sealed? Were rival figures published values or re-runs?

• What is the protocol? How many folds, how many seeds, which averaging rule?

• Was contamination checked? How?

• Where is it weak? A credible source can name the boards where it is behind.

• How does it handle gaps, names and new tasks? Those are the places where real tables bite.

If a source cannot answer these, treat its headline number with caution.

Common misunderstandings

A few mistakes come up often when people first meet the class.

"It is a language model reading CSV text."

It is not. A language model on serialized text reads a string. A data language model reads cells as cells, with their types and positions, and attends along rows and columns.

"No training means no learning."

The learning happened before, during pre-training. At use time, no weights change, and the model adapts through the context it reads. It is closer to a skilled worker reading a briefing than to a new hire being trained.

"It will always beat a tuned tree ensemble."

It will not. On large, clean tables with many labels and good features, a tuned ensemble may win. The evidence on this site says the model leads on several boards, is behind on others, and has narrow leads in places.

"It needs a data dictionary."

It does not. It reads values and relations. A dictionary can still help a human reader, and it is not required by the model.

"One foundation model removes the need to look at data."

It does not. Representativeness, leakage, label noise and distribution shift are all still yours to check.

Open problems for the class

The field is young, and honest notes on what is unsolved help set expectations.

Models read a support set of rows and not every row of a huge table, and choosing that sample well matters.

• Strong regression and multiclass results. The classification and binary boards are where models of this kind are strongest. Other task types need more work.

• Calibrated uncertainty. A model can rank well and still be badly calibrated. Intervals needed split-conformal calibration to cover as often as they claimed.

• Prior design. The synthetic prior shapes everything downstream. Better priors are an active area.

• Fair benchmarking. Harness mistakes, tuning asymmetries and contamination have all distorted results. Careful, symmetric protocols are still the exception.

• Distribution shift. A model reads the table it is given. When the world moves, the context goes stale, and nothing inside the model notices.

Further reading on this site

• Architecture explains the network part by part.

• Cell encoding shows how a raw cell becomes a vector.

• In-context prediction explains how it predicts without training.

• Pre-training covers how the weights were learned.

• Capabilities lists each output.

• Solutions lists the six questions a model like this answers.

• Evaluation and Limitations say how to read the results.

Frequently asked questions

What is a tabular foundation model?

It is a neural network pre-trained once on a large and varied collection of tables, so that it understands how structured data behaves. At use time it reads your table and returns predictions or understanding without being trained on that table. The class includes prior-fitted in-context models such as TabPFN and TabICL, models with a language component such as TabSTAR, and data language models that read raw cell values.

How is it different from XGBoost or a random forest?

Boosted trees and forests are trained from scratch for each table and each task, and they carry no knowledge from other tables. A tabular foundation model arrives pre-trained and predicts in context from the labelled rows. Trees are still excellent on large tables with plenty of labels and good features, so a fair comparison on your own split is the right way to choose.

What does in-context mean for a table?

The labelled rows are placed in the model's input as examples, and the model compares each new row with them to read off an answer. No weights are updated, which is why there is no training run for your table. The method is explained on the in-context prediction page.

Why not just give the table to a large language model?

Serializing rows into text discards the joint structure between columns, spends tokens on every digit, makes long inputs, and forces an order on rows and columns that have none. A model that reads cells as cells keeps that structure from the start.

What is a data language model?

It is a foundation model that understands tables natively, directly from raw cell values, with no serialization and no preprocessing. The name draws an analogy with language models: where a language model learns the structure of language, a data language model learns the structure of tables. The idea comes from the paper described on the Research page.

Does the model need column names?

No. Column embeddings are always zeroed, so the encoder sees values and their relations to other columns and never a header. A column called c3 that holds phone numbers is understood as phone numbers. When a name is the only clue, the model has nothing extra to lean on, so unusual formats can lower its confidence.

How is the model trained?

It is pre-trained with masked cell modeling, which hides cells and asks the model to recover them, on a blend of real and synthetic tables. The synthetic generator's prior is part of the model, so it is audited. Training is distributed across GPUs with data and tensor parallelism and mixed precision. See Pre-training.

Can it do more than predict?

Yes. In one pass it names the industry sector of a table, profiles each column, maps columns and records across tables, fills missing values with a row-level confidence, and makes classification, regression and anomaly predictions. Each output comes with its own confidence. See Capabilities.

Is it better than every other tabular model?

No. It leads on several published boards under one sealed configuration, and it is behind the leaders on regression and multiclass boards. Some leads are narrow. The benchmark pages say which is which, and the standing rules explain how the comparisons were made.

When should I not use one?

Be cautious when you have a very large, clean table with abundant labels and strong engineered features, when the target depends on a rule that no similar row reveals, or when a decision about a person depends on a score and no human reviews it. See Limitations.

Summary

A tabular foundation model is pre-trained once on many tables and then applied to new tables without per-table training. It replaces the cycle of cleaning, encoding, engineering and training with a single model that already knows how tables behave.

The landscape runs from boosted trees and AutoML stacks, through per-table deep networks and prior-fitted in-context models, to models with a language component, language models on serialized rows, and data language models. Each has a place, and none wins everywhere.

Reading the raw table matters because every preprocessing step discards information, pipelines are brittle, names are unreliable, gaps carry signal, and tables relate to each other in ways that flattening destroys. The model on this site is built on that premise, with cells as vectors, attention along rows and columns, a latent view of the whole table, and small heads for each output. For the details, continue to Architecture, Cell encoding and In-context prediction.