# What is a tabular foundation model? A tabular foundation model is a model pre-trained on a very wide range of tables so that it can work on a new table it has never seen. Instead of fitting a fresh model for every dataset, you give it the table and it applies what it learned across all the tables it saw during pre-training. ## Why tables needed their own foundation model Language models and image models became foundation models because one pre-trained network could be reused for many tasks. Tables were slower to follow. Gradient-boosted trees such as [XGBoost](https://www.ozercemkelahmet.com/tools/xgboost) and [CatBoost](https://www.ozercemkelahmet.com/tools/catboost) remained the default for tabular prediction because they are fast, robust, and strong on small and medium data. But every dataset still needs its own training run, its own tuning, and its own preprocessing. A table is also a different kind of object from text. Its rows have no fixed order. Its columns can be numbers, categories, dates, or free text, and their meaning changes from one dataset to the next. A tabular foundation model has to respect that structure instead of forcing it into a sequence of words. ## How it works Most tabular foundation models share four ideas. - **Pre-train once, on many tables.** Training uses a large collection of real or synthetic tables, with objectives such as predicting a masked cell or a held-out label. See [pre-training](https://www.ozercemkelahmet.com/model/pretraining). - **Treat rows and columns symmetrically.** Attention runs along rows and along columns, so the model sees both which rows are alike and which columns are related. See [architecture](https://www.ozercemkelahmet.com/model/architecture). - **Encode every cell as a vector.** A cell is represented by its value, its type, and its position, with an explicit marker for a missing value. See [cell encoding](https://www.ozercemkelahmet.com/model/cell-encoding). - **Predict in context.** The labeled rows of your table are given to the model as context and it predicts the unlabeled rows in a forward pass, often with no gradient step on your data. See [in-context prediction](https://www.ozercemkelahmet.com/model/in-context-prediction). ## The landscape of tabular foundation models Several families exist. [TabPFN](https://www.ozercemkelahmet.com/tools/tabpfn) showed that a transformer trained on a very large number of synthetic datasets can predict on small real tables in a single forward pass. [TabICL](https://www.ozercemkelahmet.com/tools/tabicl) extends in-context learning to larger tables. [TabSTAR](https://www.ozercemkelahmet.com/tools/tabstar) targets tables that contain text columns. Earlier deep models such as [FT-Transformer](https://www.ozercemkelahmet.com/tools/ft-transformer) and [SAINT](https://www.ozercemkelahmet.com/tools/saint) are not foundation models, because they are trained per dataset, but they introduced attention over table columns. A [Data Language Model](https://www.ozercemkelahmet.com/articles/what-is-a-data-language-model) is a tabular foundation model that goes one step further: it reads the raw table directly, with no serialization to text, no preprocessing, and no dependence on column names. ## What one model can do Because the model builds a general representation of a table, the same network can serve several tasks: - predicting a column (classification and regression); - filling in missing values ([imputation](https://www.ozercemkelahmet.com/model/bench-imputation)); - flagging rows that do not fit ([anomaly detection](https://www.ozercemkelahmet.com/model/bench-anomaly-detection)); - estimating a full predictive distribution ([probabilistic regression](https://www.ozercemkelahmet.com/model/bench-probabilistic-regression)); - matching records and columns across tables ([entity and schema matching](https://www.ozercemkelahmet.com/model/bench-entity-schema-matching)). ## Where tree ensembles still win A tabular foundation model is not a replacement for every pipeline. Gradient-boosted trees are cheap, easy to explain, and hard to beat when a team has time to tune a single large dataset. Foundation models have costs of their own: memory for large tables, a pre-trained checkpoint to host, and the need to check that evaluation data was not seen in pre-training. The [limitations](https://www.ozercemkelahmet.com/model/limitations) page lists what I know does not work yet. ## How to judge one Read the protocol before the score. Check which datasets were used, whether the rivals were tuned, whether data could have leaked from pre-training, and whether the numbers come from one fixed configuration. The [evaluation](https://www.ozercemkelahmet.com/model/evaluation) page and the [benchmark stories](https://www.ozercemkelahmet.com/model/benchmarks) describe how I did it, and [A score is only as good as its report](https://www.ozercemkelahmet.com/articles/a-score-is-only-as-good-as-its-report) explains what a trustworthy report contains. ## Frequently asked questions What is a tabular foundation model in one sentence? A model pre-trained on many tables that can predict, impute, and detect anomalies on a new table without being trained again on that table. How is it different from XGBoost? XGBoost is trained from scratch on each dataset. A tabular foundation model is trained once and applies that experience to every new table it is given. Is a tabular foundation model a large language model? No. A language model reads tokens of text. A tabular foundation model reads the cells of a table and keeps its row and column structure. Does it need column names? Not necessarily. Some models use column names as text. A Data Language Model such as the one documented here works from values and relations alone, so it also works on tables with no headers.