# Architecture How a tabular foundation model is built: cell encoding, axial attention, Perceiver-style global reasoning, task heads, adapters, and one merged checkpoint. > Informational documentation, based on the paper Data Language Models: A New Foundation Model Class for Tabular Data (arXiv:2605.06290, https://arxiv.org/abs/2605.06290). Reference material, not a hosted service; hosts, keys, and ids are placeholders. The model is a transformer designed for grids, not for sequences. It reads every cell of a table as a vector, lets cells talk to each other along rows and along columns, summarizes the whole table in a small set of latent vectors, and then answers with small task heads. This page walks through those parts in the order a table moves through them, and says why each choice was made. If you only want the short version: the encoder never uses a column's name or identity, only its values and how they relate to other columns. That one decision explains most of the behavior you see in the benchmarks. ## The design at a glance A table goes in. One shared representation comes out of the body. Several small heads read from it. Nothing in the path needs a preprocessing step. ```text raw table | v cell encoding value + type + position (+ explicit "missing") | v axial attention row layer, column layer, row layer, column layer ... | v global reasoning latent vectors read the whole table, then write back | v shared representation | +--> imputation network +--> semantic (sector and column role) head +--> prediction head (classes, quantiles, anomaly scores) +--> cross-table head (columns and entities) +--> adapters and memory (continual learning) ``` Think of a hospital team. One person reads the whole chart (the body). Then different specialists each look at that same reading and answer their own question (the heads). Nobody re-reads the raw chart from scratch. The first model of this class was a 140M parameter model trained on more than 2.3 million synthetic and real-world datasets, described in the research paper. The model on this site is its successor and has more than 1.5 billion parameters. Once every head and adapter is merged with the core into one checkpoint, the total is about 2.18 billion parameters. ## Why tables need their own architecture Text is a line of tokens. An image is a fixed grid of pixels. A table is neither. It has two axes, and both vary from one table to the next. Four properties of tables shape the design. - **Rows have no order.** Shuffle the rows of a customer table and it is still the same table. A model that cares about row order is learning something false. - **Columns have no order either.** Swap column 3 and column 7 and nothing about the data changed. - **Sizes vary.** One table has 40 rows and 6 columns. Another has 200,000 rows and 90 columns. One network must handle both. - **Types are mixed.** A single row can hold a number, a category, a date, and a piece of text. A plain language model would force the table into a string, which throws away the joint structure between columns. A plain image model would force a fixed size. Neither fits. The model here is built so that the structure of a grid is a built-in assumption, not something it has to rediscover from data. See tabular foundation models for how this class of model differs from per-table training. ### Symmetry is a feature Because rows and columns carry no order, a good tabular model should give the same answer when you shuffle them. Attention is naturally friendly to this. It treats its inputs as a set unless you add position information. The position signal used here marks which cell sits in which row and which column, so cells in the same row or column can find each other. It does not say "this is the third column of the file, so it must be the age." ## 1. Cell encoding Every cell becomes a vector. This is the first step and the one that makes everything else possible. For a deeper look, see cell encoding. A cell's vector combines three things. | Part | What it carries | |---|---| | Value | The number, the category, the date, or the text, mapped into a shared space | | Type | Whether the cell is numeric, categorical, a date, or text | | Position | Which row and which column the cell belongs to, so the model can group cells | Numbers, categories, dates, and text all land in the same vector space. That is what lets one network read them together. A row such as a price, a product category, a purchase date, and a free-text note is turned into four vectors of the same width. ### Missing cells are inputs A missing cell is not an error to be cleaned away. It is encoded explicitly, as its own kind of input. The reason is practical. In real tables the pattern of what is missing often carries information. Because missing cells are first-class, the same machinery that reads present cells also reads absent ones. That is also how imputation works later: the model has already seen, during pre-training, a great many tables with holes in them. See Pre-training. ### Column identity is never used This is the most important design decision in the encoder. Column embeddings are always zeroed. The model never gets a column's identity as an input. It gets the values in the column and how those values relate to the values in other columns. An everyday analogy: imagine a detective who is handed a spreadsheet with all headers torn off. A good detective can still tell which column is a phone number, which is a date of birth, and which is a price, from the shape of the values and from how the columns move together. The model is trained to be that detective. What this buys you: - A column called `c3` that holds phone numbers is understood as phone numbers. - Renaming every column to random strings does not change what the model sees. - Removing the names entirely does not change what the model sees. - Misleading names, such as a column called `value` that is really a date, do not mislead it. When a name is the only clue, the model has nothing extra to lean on. It reads the data, and unusual or mixed formats can still lower its confidence. See Limitations and column name independence. ## 2. Axial attention Full attention over every cell is too expensive. If a table has R rows and C columns, there are R times C cells. For a table with 100,000 rows and 50 columns that is hopeless. Axial attention splits the work into two cheaper pieces, one along each axis. See axial attention for the general method. ```text layer n : attend within each row (cells of one record) layer n+1 : attend within each column (values of one attribute) layer n+2 : attend within each row layer n+3 : attend within each column ... ``` In a row layer, each cell looks at the other cells of its own row. This is how the model learns that a high balance and a long tenure go together in one customer. In a column layer, each cell looks at the other values of its own column. This is how the model learns what a typical value looks like for that attribute, and which values are strange. ### Why alternating layers still reach everywhere You might worry that a cell can only see its row and its column. It can, in one layer. But the layers are stacked. After a row layer, every cell carries information about its row. After the next column layer, each cell collects those row summaries from its whole column. After two layers, information has travelled from any cell to any other. Think of passing a note in a classroom laid out in a grid. In one move you can pass it along your row. In the next move you pass it along a column. Two moves reach any desk. Each is far below the square of R times C. In practice the row axis is short (tens of columns) and the column axis is long (many rows), so the long axis is where care is needed. This is one reason the model is designed to read a **support set** of rows, rather than every row of a large table at once. That idea returns in the prediction head below. ### Permutation behavior Row attention without row-order information is insensitive to column order. Column attention without order information is insensitive to row order. Together they give the model the right symmetry: shuffle the rows or the columns and the output moves with them, not apart from them. ## 3. Global reasoning Axial attention is excellent locally. It is weaker at one thing: forming a view of the whole table. The table as a whole is what tells you the industry it comes from, or whether it is related to another table. For that, the model uses a Perceiver-style block. See Perceiver IO for the general method. The block works in two moves. 1. **Read.** A small, fixed set of latent vectors cross-attends over all the cells. Each latent asks the table a question and collects an answer. 2. **Write back.** The cells then cross-attend to the latents, so each cell is updated with the table-level view. ```text cells (many, variable) latents (few, fixed) | ----- read -----> | | <---- write back ---- | ``` The key property is that the number of latents is fixed. A table with ten rows and a table with a million rows are both squeezed through the same narrow channel. An analogy: an editor skims a long report and writes a one-page brief. Anyone can then ask the brief a question without rereading the report. The latents are the brief. This is what makes whole-table tasks possible. ### What the latents are used for - **Sector identification.** The industry of a table is a property of the whole table, not of any cell. The latents carry that. - **Cross-table work.** To decide that two columns in two different tables mean the same thing, you need a compact description of each column in context. The latents help build it. - **Table-level confidence.** A summary of how typical or unusual a table is feeds into how confident the outputs are. ## 4. The shared representation After the axial layers and the global block, every cell has a vector that has seen its row, its column, and the whole table. There are also latent vectors that describe the table as a whole. Together these are the shared representation. Everything the model can do is a different reading of this one representation. That is why the outputs come in one bundle and have one structure. See Capabilities for each output. This is the core idea of a foundation model: learn the general reading once, then attach thin task-specific parts. The thin parts are small compared with the body. They are cheap to train and, in several cases, cheap to swap. ## 5. Task heads Small heads sit on top of the shared representation. Each is described here in terms of what it reads and what it returns. | Head | Purpose | |---|---| | Imputation network | Reconstructs missing numeric and categorical cells | | Semantic classification head | Assigns the sector and the role of each column | | Prediction head | Classification, regression, and anomaly scoring in context | | Cross-table head | Aligns columns and matches entities across tables | ### Imputation network The imputation network reads the shared representation of a masked or empty cell and returns a value for it. It was trained with masked cell modeling, which is explained on Pre-training, and it covers both numbers and categories. See imputation networks for the general idea. For numbers, the output is a number. For categories, the output is a distribution over the class labels, and the filled value is a class label. Confidence is reported per row. On the hardest imputation benchmark I ran, the final system was not a single head. It was a chain of three arms: - a **ridge arm**, a plain linear model that is hard to beat and gives a verified floor, - an **in-context arm**, which uses the model's own reading of the table, - a **neighbours arm**, which borrows values from similar rows. The arm used for each column is chosen on masked holdouts, with fixed proportional margins against the plain ridge arm. The reason for the margins is statistical honesty. When only about 30 holdout cells decide among 77 candidates, the winner is partly luck, so a candidate must beat the baseline by a set margin, not just by a hair. For categories the chain returns the expected value instead of the single most likely class when the metric is squared error, because the mean is the optimal answer for squared error. See Imputation. ### Semantic classification head This head does two jobs. - **Column role and type.** For every column it returns a role such as email, phone, name, date, address, identifier, measure, category, currency, or geo, along with a type and a missing share. - **Sector.** For the whole table it returns a top guess and a short list with a confidence on each. The sector head is hierarchical. It first decides a coarse family and then a specific sector inside it. This was a deliberate fix. A flat fingerprint of each column on its own could not separate neighbouring sectors such as developer tools and data analytics. What did separate them was the joint relationship between pairs of columns. So the head is fed pair-level signal from the body, and the final blocks of the body were fine-tuned for it on a set of tables kept out of the test set. See Blind sector identification. ### Prediction head The prediction head makes classification, regression, and anomaly scores **in context**. That phrase has a precise meaning: the model reads the labelled rows that are already in the table (the support set), and predicts the target for other rows by comparing them with the supports. Nothing is trained for your table. See in-context prediction and few-shot learning for the general idea. In practice the head works by retrieval over the supports. In the early work I tried nine similarity-based decoders, and they all hit the same wall. The support sets were physically too small, only 24 supports where 128 or more were needed. A deeper cause was the training prior, which turned quantile-band targets into XOR-like patterns that no similarity rule can separate. After a cluster-style prior removed the wall, error-correcting output codes added only bit noise, so I kept direct retrieval over the supports. Then I made the retrieval mode leaner and tested each change on the whole board, not on one dataset. Three flavors share the head. - **Classification** returns class probabilities. - **Regression** can return quantiles, so you read a range rather than a single number. A 999-quantile head trained with a CRPS loss gives a full distribution. The distribution is then calibrated with split-conformal calibration and an isotonic correction so that the intervals cover as often as they claim. See probabilistic regression. - **Anomaly scoring** runs unsupervised, with no target. See below. #### Anomaly detection as three channels The anomaly scorer uses what the model has learned about the table, not only distances. It has three model-based channels: 1. the **residual of the imputation head**: how surprising a cell is given everything else in its row, 2. **nearest neighbours in the model's embedding space**, 3. **nearest neighbours in the raw space**. An isolation forest (isolation forest) joins as an auxiliary channel. Which channel to trust is decided without labels, by a pseudo-anomaly selector. Fake anomalies are built from the training rows by shuffling features and injecting uniform values. Each channel is measured on them, and the channels are weighted with a softmax. The ranks are then fused. When a table has no missing values, the imputation head would have nothing to do, so 15 percent random missingness is injected before fitting. See anomaly detection and Anomaly detection benchmark. ### Cross-table head The cross-table head decides which columns and which rows in different tables describe the same thing. For **schema matching** (which column in one table is which column in another), the matcher combines three channels: - adapter embeddings of columns and cells, - a channel of 28 column statistics, - a name channel. It scores pairs, normalizes the scores, and keeps only mutual best one-to-one matches. A pair survives only if each column picks the other as its best match. For **entity matching** (which rows in two tables are the same real-world entity), the head works with no keys. The recipe that worked was a short training run, a two-model ensemble, and a calibrated threshold. Longer training hurt, and the ensemble cut the run-to-run variance. See Entity and schema matching. The output is a set of column alignments, entity matches, and a unified schema, each with a confidence. Each aligned pair is grounded in a verbatim example from each side. ## 6. Adapters and memory for continual learning A foundation model that is frozen cannot learn from your next task. A foundation model that is simply fine-tuned forgets the last one. The architecture has a path in between. See continual learning for tables and adapters. ### The adapter An adapter is a small module attached to the body. It is trained for a task while the body stays mostly fixed. The rule that matters most: the adapter does not depend on the number of columns. Its weights are the same size whether the table has 6 columns or 60. That is what lets the weights carry from one task to the next. The first continual-learning runs forgot badly, and the cause was an architectural bug. The adapter was re-created from scratch for every task, so each task erased the last. After the fix the adapter was created once and carried across tasks. ### Replay and regularization Memory comes from two tools. - **Experience replay** keeps a small buffer of raw rows from earlier tasks and mixes them into later training. The recipe that stayed uses 48 raw rows per task. See experience replay. - **Elastic weight consolidation** penalizes changes to weights that mattered for earlier tasks. See EWC. Other options were tested and dropped. An L2 anchor to the old weights failed. A pseudo-rehearsal variant had a higher mean but a terrible worst case, so I rejected it. I also found that a missing random seed alone moved results by 15 to 20 percent, so runs are seeded. Retention here comes from replay and from the adapter design, not from the weights alone, and the buffer grows linearly with the number of tasks. Read the numbers in Continual learning with that in mind. ## 7. One checkpoint A result counts only if one model can reproduce all of it. So the core and every head and adapter were merged into one unified checkpoint of about 2.18 billion parameters, with the code embedded. One line of bootstrap gives access to every capability. The merged model was then tested in isolation. See One checkpoint. This has two benefits. First, a benchmark number is not a separate research artifact. It comes from the same model that serves requests. Second, a head that works only in a special build cannot sneak into a claim. ### Compression The checkpoint can be stored at half precision, 8-bit, and 4-bit. Fidelity was measured with TurboQuant. See TurboQuant and compression and serving. Post-training quantization is the right fit for a model that is trained once and served many times. ## What was tried and did not help Honest architecture notes include the dead ends. They shaped what stayed. | Idea | What happened | |---|---| | More similarity-based decoders | Nine of them hit the same wall; the support set and the prior were the cause | | Error-correcting output codes | Added only bit noise; direct retrieval over supports stayed | | An L2 anchor for continual learning | Failed | | Pseudo-rehearsal | Higher mean, terrible worst case; rejected | | Test-time augmentation for sector identification | Destroyed the signal in tables, though it helps in vision | | A CRLS fine-tune | Made both metrics worse; dropped | | Standardizing heavy-tailed targets by standard deviation | Made distributions too narrow; a log transform fixed it | | Longer entity-matching training | Hurt; a short run with an ensemble worked | The pattern is useful. Several failures were not architecture failures at all. They were a prior, a harness, a seed, or a scale choice. Before changing the network, check the ruler. ## How the parts map to outputs | Output of a pass | Parts that produce it | |---|---| | Sector | Latents, hierarchical semantic head | | Column profile | Cell encoding, column attention, semantic head | | Cross-table map | Latents, adapter embeddings, cross-table head | | Imputation | Masked-cell training, imputation network, three-arm chain | | Predictions | Support set, prediction head, quantile head, anomaly channels | | Held-out block | Same prediction head, scored on rows kept out of context | ## Design principles in one list - **Read values, not names.** Column identity is zeroed. - **Respect the grid.** Alternate row and column attention. - **Keep heads small and the body shared.** One reading, many answers. - **Predict in context.** Use the labelled rows that are there. Train nothing. - **Make missing a first-class input.** Holes carry information. - **Remember on purpose.** Adapters that do not depend on column count, replay, and regularization. - **Ship one object.** One checkpoint, one contract, one bundle. ## Reading the rest of the documentation - What it is explains the idea and the inputs and outputs. - Pre-training covers how the weights were learned. - Capabilities covers each output. - Evaluation covers how results are measured. - Benchmarks shows how each capability was developed. - Limitations says what the model does not do. ## Frequently asked questions ### What kind of neural network is the model? It is a transformer built for two-dimensional data. Cells are encoded as vectors, attention alternates between the rows and the columns of a table, a Perceiver-style block gives it a table-level view, and small heads produce the outputs. It is called a Data Language Model because it reads the raw values of tables the way a language model reads text. ### Why use axial attention instead of full attention? ### Does the model use column names? No. Column embeddings are always zeroed, so the encoder sees values and their relations to other columns, never a column's identity. That is why it is robust when headers are random, stripped, or misleading. A name channel exists in the schema-matching scorer as one of three channels, but the encoder itself does not use names. ### How many parameters does it have? The model has more than 1.5 billion parameters. When the core and every head and adapter are merged into one checkpoint, the total is about 2.18 billion. The first model of this class had 140M parameters. ### What is the role of the Perceiver-style block? It compresses a table of any size into a small, fixed set of latent vectors and lets them talk back to the cells. ### How does the model predict without training on my table? It predicts in context. The labelled rows in the table act as a support set, and the prediction head compares each new row with them. No weights are updated. The method and its limits are covered in in-context prediction. ### How does the model avoid forgetting old tasks? Adapters that do not depend on the number of columns carry their weights across tasks, a small replay buffer of raw rows (48 per task in the recipe that stayed) is mixed into later training, and regularization such as elastic weight consolidation limits drift. ### Why one merged checkpoint? So that every reported capability comes from one object that can be tested in isolation. The merged checkpoint has the code embedded and one bootstrap line gives access to all capabilities, and it is the same model that serves requests. ### Can the model be compressed? Yes. It was dumped at half precision, 8-bit and 4-bit, and fidelity was measured with a dedicated quantization method. The trade-offs are described in compression and serving. ### What parts of the design are still weak? Regression and multiclass boards are behind the leaders, interval coverage was weak before calibration, and continual-learning retention depends on replay. The Limitations page lists these plainly. ## Summary The model is a transformer made for grids. Cells become vectors that carry value, type, and position, with missing cells encoded openly and column identity zeroed. Small heads for imputation, semantics, prediction, and cross-table matching read one shared representation, adapters with replay and regularization let it learn new tasks without forgetting, and one merged checkpoint of about 2.18 billion parameters holds all of it. The design choices came from tests, including the ones that failed, and the sibling pages on Pre-training, Capabilities, and Benchmarks show how they paid off.