Cell encoding

Cell encoding is the first step a table takes inside the model, and it makes everything after it possible. Every cell, one value in one row and one column, becomes a vector. Numbers, categories, dates and text all land in the same space, so one network can read them together.

A cell's vector combines three things: its value, its type, and its position. A missing cell is not dropped or patched. It is encoded openly, as its own kind of input. And the encoder never receives a column's name or identity, because column embeddings are always zeroed.

This page explains why the cell is the unit, what goes into a cell's vector, how each kind of value is handled in principle, why missing cells are inputs, why names are left out, and what that means for you in practice. It is the deep dive behind the first step of Architecture.

Why the cell is the unit

A table is a grid of cells. Text is a line of tokens and an image is a grid of pixels, and a table is a grid in which each position holds a value of a different kind.

Most older approaches pick a bigger unit and lose something. Some treat a whole row as one vector, which squeezes the columns together before the model has looked at them. Some turn the table into a sentence, which throws away the grid. Some treat each column as a separate feature with its own name and its own scaling, which ties the model to one schema.

The cell is the smallest unit that still means something on its own. It has a value, it belongs to one row and one column, and it can be present or missing. If the model encodes cells well, rows, columns and whole tables can be built up from them.

An everyday picture

Think of a seating chart for a banquet. Each seat has a guest, and each guest has a few facts: a name, a table number, a dietary need. You can reason about the whole room by reasoning about seats and their neighbours.

If you rolled the chart into a long paragraph of text, you would have to rediscover who sits next to whom. If you squeezed each table into one summary, you would lose the seat-level detail. Cell encoding keeps the seats.

The three ingredients of a cell vector

The architecture page states the rule in one table. A cell's vector combines three parts.

A fourth case sits beside them. A cell may have no value at all, and then the vector says so, explicitly.

The three parts are combined into one vector of the same width for every cell. A row with a price, a product category, a purchase date and a free-text note becomes four vectors of the same size.

Having the same width is what lets one attention layer treat them as members of one family. Nothing else in the network has to know which kind of cell it is looking at, except through what the vector carries.

The value

The value is the content of the cell. The hard part is that values come in very different kinds, with very different statistics.

The documentation of this model states the principle and not the internals. Each value is mapped into a shared space where numbers, categories, dates and text can sit beside each other. The sections below explain, in general terms, what any good encoder has to get right for each kind, and where the choices come from.

Numbers

Numbers are the most common cell. They are also tricky for three reasons.

• Scale. A column of ages runs from 0 to 100. A column of incomes runs from zero to millions. A column of probabilities sits between zero and one. A network cannot treat these on the same footing unless the scales are handled.

• Shape. Many real columns are heavy-tailed, with a few huge values and many small ones. House prices, runtimes and car prices are examples.

• Meaning of closeness. Seven is close to eight and far from seven million. A good encoding keeps that closeness, so similar numbers have similar vectors.

I met the second problem in my own work on probabilistic regression. The target was standardized by its standard deviation, and for heavy-tailed targets that made the predicted distribution too narrow. A log transform for heavy tails fixed it. The lesson carries over to the input side: how a number is scaled is not a detail, it shapes what the model can see. See Probabilistic regression.

A useful way to picture a numeric encoding is a ruler that is read relative to its own column. The model does not need to know that a value is measured in dollars. It needs to know where the value sits among the other values of its column, and how it relates to the other cells of its row.

Categories

A category is a label from a small set.

The values of a category have no natural order or distance. Gold is not between silver and bronze in any numeric sense. The encoding has to make equal values look equal and different values look different, without inventing a false ordering.

There is a further subtlety. Because the model does not use column names, a category value like gold is not interpreted through a dictionary of English words by the encoder. It is understood by how it behaves, by which rows share it, how often it appears, and how it co-occurs with other columns.

That is the same way a person can understand a column of unfamiliar codes, A7, B2, A7, C1, by noticing which codes go with which other values. A code does not need a definition to be a useful label.

Dates and times

A date is a number with structure. It has an order, a distance to other dates, and cycles inside it: day of week, month, season, hour of day.

A date written as text, 2024-11-03, hides all that unless the encoder reads the structure. A good encoding keeps the order, so that earlier dates are different from later ones, and keeps enough of the cycles that a Monday can resemble other Mondays.

Dates also relate to other dates in the same row. A shipping date after an order date is normal. A shipping date before it is a hint of an error. The row attention in the next stage can only learn that if the two dates are encoded in a way that makes their difference readable.

Text

Some cells hold free text, such as a note, a title or a short description. The documentation states that text is one of the kinds mapped into the shared space.

Text is the least tidy kind, because it can be anything. In practice most text cells in business tables are short and repetitive: a status note, a product name, a city. For those, treating the cell as a value with a recognizable shape works well.

Long free text, where meaning depends on grammar and world knowledge, is a harder case, and it is where models with a dedicated language component have a natural advantage. See Tabular foundation models for how the families differ.

Identifiers

Identifiers are values that name things, such as customer numbers, order numbers and codes. They look like numbers or like text, and they do not behave like either. Adding two customer numbers is meaningless.

A good encoder has to avoid treating an identifier as a measurement. The column profile output has a role called identifier for exactly this reason, and a column whose values are all different and carry no magnitude is likely to be read that way. See Capabilities.

The type

The type tells the network what kind of cell it is looking at: numeric, categorical, a date or text.

You might ask why the type is needed at all, if the value already carries it. There are two reasons.

First, the same raw value can be read in different ways. The digits 20240101 could be a date, a count or an identifier. Knowing the type removes the ambiguity.

Second, the type is a cheap and strong hint for the later layers. A column of numeric cells can be compared by magnitude, a column of categorical cells by equality. Telling the network the type saves it from rediscovering that fact in every layer.

The type is a property of the cell as read, and not of the column's name. The column profile that comes out of the model reports a type for each column, along with a role, and it does so without looking at a header.

Role is not type

The two are easy to mix up. Type is how the values are stored and compared. Role is what the column means in the table.

A column can be numeric in type and a measure, an identifier or a currency in role. A column can be categorical in type and an email, a phone number or a name in role. The encoder carries type, and the semantic head works out role from the shared representation later. See Architecture.

The position

Position says which row and which column a cell belongs to. It is what lets the model group cells.

The key point is how position is used. It marks that this cell sits in row 12 and column 3, so that cells of the same row or column can find each other during attention. It does not say that this is the third column of the file and therefore must be the age.

Why position matters for axial attention

The next stage of the model alternates attention along rows and along columns. In a row layer each cell looks at the other cells of its own row. In a column layer each cell looks at the other values of its own column. See axial attention.

For that to work, the cells have to know which row and which column they belong to. Position is the signal that makes the grouping possible.

No order is imposed

Rows have no order. Shuffle the rows of a customer table and it is the same table. Columns have no order either. Swap column 3 and column 7 and nothing about the data changed.

A model that cared about row order or column order would be learning something false. Attention treats its inputs as a set unless you add position information, and the position used here marks membership in a row and a column, not an absolute rank.

The consequence is the right kind of symmetry. Shuffle the rows or the columns and the output moves with them, not apart from them.

original              columns swapped
r1:  34  gold  Ana    r1:  Ana  gold  34
r2:  51  free  Bo     r2:  Bo   free  51
r3:  29  gold  Cy     r3:  Cy   gold  29

same cells, same row and column memberships, same answers

What position does not do

Position does not carry meaning on its own. It does not tell the model that the first column is usually an identifier, which is a habit some files have. If you have a rule of that kind, you should not expect the model to guess it from a column's place in the file.

Missing cells are inputs

A missing cell is not an error to be cleaned away. It is encoded explicitly, as its own kind of input.

The practical reason is that the pattern of what is missing often carries information. A blank date of death means something different from a blank date of birth.

If the encoder replaced a missing value with a mean before the model ever saw it, that information would be gone. The model would see a plausible number and have no way to know it was made up.

One machinery for present and absent cells

Because a missing cell is a first-class input, the same machinery that reads present cells also reads absent ones. During pre-training the model sees tables that already have holes, as well as holes that were added on purpose by masking. See Pre-training.

That is also how imputation works later. A real gap is treated like a masked cell, and the model fills it from the row and the column around it.

The three mechanisms of missingness

Why a cell is missing matters, and the model has seen all three standard mechanisms.

An explicit missing marker lets the model learn the patterns: gaps that cluster in one column, gaps that follow another column's value, gaps that line up with the hidden value. The widest margin in the imputation benchmark came under not-at-random missingness, which is the hardest case. See Imputation.

Robustness to injected gaps

The same design explains behaviour in the robustness test. Ten datasets get masks of three types and several rates, and AUC is measured after injecting missing values into the features. The model reads around the gaps instead of choking on them. The mean AUC is above the published method, by a narrow lead that I report as one. See Missing-data robustness.

Honest and dishonest placeholders

Real data often marks missing values in disguise. A value of -999, an empty string, the text N/A, a date of 1900-01-01 and a zero in a column where zero is impossible are all placeholders.

If these are read as real values, the model sees a false pattern. The honest encoding is to turn a placeholder into a true missing cell where you can. Where you cannot, expect lower confidence, because the model has to work out that the placeholder is not a measurement.

Column identity is never used

This is the most important design decision in the encoder. Column embeddings are always zeroed.

The model never gets a column's identity as an input. It gets the values in the column and how those values relate to the values in other columns.

Why leave names out

Imagine a detective handed a spreadsheet with all headers torn off. A good detective can still tell which column is a phone number, which is a date of birth and which is a price, from the shape of the values and from how the columns move together.

The model is trained to be that detective. If the name were available, the shortcut would be tempting: learn that a column called age holds ages, and stop looking at the values. That shortcut breaks the moment a table arrives with a column called c3.

By closing the shortcut during training, the model is forced to learn what a phone number, a date or a price looks like from the values and from how columns relate.

What this buys you

• A column called c3 that holds phone numbers is understood as phone numbers.

• Renaming every column to random strings does not change what the model sees.

• Removing the names entirely does not change what the model sees.

• Misleading names, such as a column called value that is really a date, do not mislead it.

In the metadata test, twenty numerical classification datasets are run under three naming conditions, full names, random strings and names removed. Models that read headers lose roughly seven AUC points when headers vanish. The model is the only one in the comparison whose accuracy does not fall. See Metadata independence and Column name independence.

When a name is the only clue, the model has nothing extra to lean on.

A column of the digits 1 to 5 might be a rating, a severity grade or a count of children. A header would settle it in a second. The model has to infer from the values and the neighbours, and its confidence reflects the doubt.

On pure numeric classics without semantics, a stronger prior-based rival still leads on the metadata benchmark, usually by a few thousandths. The benefit of ignoring names is largest where meaning matters.

Names still exist elsewhere

Zeroing the column embedding is a statement about the encoder. A name channel does exist in the schema-matching scorer, as one of three channels, so the matching step can use names when they help. The encoder that produces the shared representation does not.

This split is deliberate. The body stays a general reader of tables, and the one task that can benefit from names gets them through a separate channel.

What the encoder must be insensitive to

A good way to understand an encoder is to list what should not change its answer.

The first four rows are guarantees by design. The last three are the points where the model is meant to react.

A subtler item is scale. A column measured in metres and the same column in centimetres should lead to the same understanding of its role, even though the raw numbers differ. Reading each value relative to its own column is one way to get that, and it is why the column evidence in attention matters so much.

From cell vectors to the rest of the model

Cell encoding is the front door. What happens after it is easier to follow with the sequence in front of you.

raw cell
   |  value + type + position  (or explicit "missing")
   v
cell vector                same width for every cell
   |
   v
axial attention            row layers and column layers alternate
   |
   v
global reasoning           latents read the whole table and write back
   |
   v
shared representation  -->  heads: imputation, semantics, prediction, cross-table

The shared representation is a vector per cell that has seen its row, its column and the whole table. Everything the model returns is a reading of it.

A good cell encoding therefore matters in a special way. Any information it fails to carry cannot be recovered later, because the later layers only see what the encoder passed on.

How encoding supports each capability

The same encoded cells feed every output. It helps to see which part of the encoding each output leans on.

Column profile

The profile reports a role, a type, a missing share and a personal-data flag for each column. It leans on the value statistics of the column, on the type, and on the missing markers. Because names are not used, the profile still works when headers are cryptic. See Capabilities.

Sector

The industry of a table is a property of the whole table. A marginal fingerprint of each column on its own could not separate neighbouring sectors such as developer tools and data analytics. What did separate them was the joint relationship between pairs of columns.

That is only possible if cells from different columns are encoded in a space where relationships can be compared, which is what the shared value space provides. See Blind sector identification.

Imputation

Imputation reads the encoded neighbours of a missing cell, the other cells in its row and the other values in its column. The explicit missing marker is what turns an absent cell into a question the model can answer. See Imputation.

Cross-table matching

To decide that two columns in two tables mean the same thing, the model needs a description of each column in context. The schema matcher combines adapter embeddings of columns and cells, a channel of 28 column statistics and a name channel. The cell-level encoding is the base on which the adapter embeddings are built.

It then scores pairs, normalizes the scores and keeps only mutual best one-to-one matches. See Entity and schema matching.

Prediction

The prediction head reads the encoded rows of the support set and compares new rows with them. If the encoding places similar rows near each other, the comparison works. If the encoding were noisy, for example by treating an identifier as a magnitude, the neighbours would be wrong. See In-context prediction.

Anomaly detection

One of the anomaly channels is nearest neighbours in the model's embedding space, and another is the residual of the imputation head. Both depend on what the encoding and the later layers have learned about a normal cell. See Anomaly detection.

Cell encoding compared with the alternatives

There are other ways to prepare a table for a model. Comparing them shows what cell encoding keeps.

The last row is the one this page describes.

The paper behind the model calls the preprocessing route a bottleneck and the serialization route a bottleneck of its own. Cell encoding is the way out of both. See Research.

A worked example, one small table

Let us follow a tiny table through the encoder. The numbers and values are invented for illustration, and the vectors are described in words and not in numbers.

        col A     col B     col C        col D
row 1   34        gold      2024-11-03   "late delivery"
row 2   51        free      2024-11-05   (missing)
row 3   29        gold      2024-11-09   "ok"
row 4   (missing) free      2024-11-10   "refund asked"

Step one: split into cells

There are sixteen cells. Fourteen hold a value and two are missing. The model does not see headers, because there are none to see, and it would not use them if there were.

Step two: tag the type

Column A reads as numeric. Column B reads as categorical, with two distinct values. Column C reads as a date. Column D reads as text, short and repetitive.

These are readings of the cells, made from what the values look like. No label told the model so.

Step three: map the values

The numbers in column A are placed relative to the other values in their column, so 51 is recognizably larger than 34 and 29. The two categories in column B are given distinct vectors with no order between them. The dates in column C are placed along a line of time with their cycles. The short texts in column D become vectors that reflect what they say.

Step four: add position

Each of the fourteen present cells gets a mark for its row and its column. Cells in row 2 know they are neighbours. Cells in column B know they belong together.

Step five: encode the gaps

The two missing cells receive the explicit missing encoding, together with their row and column marks. The model now knows that row 2 has a blank text cell and that row 4 has a blank number.

Step six: hand over to attention

All sixteen cell vectors have the same width. A row layer lets the four cells of row 4 look at each other, so the blank number in column A can be guessed from the date, the category and the text beside it. A column layer lets the values in column A look at each other, so the blank can also be guessed from the typical values of that attribute.

That is the whole encoder. It is simple in outline, and its value is that nothing was thrown away on the way in.

How the encoding is tested

An encoding cannot be judged by looking at it. It is judged by what the model does with it. These are the tests that bear most directly on cell encoding.

The first three have benchmark pages with full protocols: Metadata independence, Missing-data robustness and Imputation.

A lesson from the imputation harness

Early imputation results were poor, and one of the causes was an encoding problem in the harness and not in the model. The semantic channel had been fed placeholder column names. The imputer had been shown only a fifth of each table, the test fold. The fix was to let it see train and test together, with both masked.

The lesson applies to cell encoding in general. When a model seems to read cells badly, check what you actually gave it before you blame the encoder. See Evaluation.

Why not give the encoder the names as well

It is fair to ask whether names would help, since they carry real information when they are good.

There are three reasons the encoder leaves them out.

1. The shortcut problem. If the model can read a name, it can learn to rely on it, and it stops looking at the values. That breaks on the first table with a cryptic header.

2. Names are unreliable at scale. Across many systems, names are inconsistent, abbreviated, translated, coded or missing. The more tables the model sees, the less a name tells it.

3. Fairness to the task. In cross-table matching, a model that reads names finds easy matches and misses hard ones. A model that is name-blind has to find the hard ones from the data, and it beats every name-aware matcher on all four schema scenarios while running name-blind.

None of this makes names worthless. It makes them a separate channel that a task can opt into, as the matcher does, and not a built-in dependency of the encoder.

What stays the same across table sizes

A table with 40 rows and 6 columns and a table with 200,000 rows and 90 columns pass through the same encoder. Nothing in the cell encoding depends on the shape of the table.

That is a design requirement, not an accident. Because the encoder looks at one cell at a time, with its value, its type and its row and column marks, the shape of the grid only matters later, in the attention layers.

This matters for adapters too. An adapter that does not depend on the number of columns can carry its weights from a 6-column task to a 60-column task, which is what lets continual learning work. See Continual learning for tables.

Practical advice for your own tables

You do not have to prepare data for the model, and a few cheap habits still help.

Keep types honest

If a column of dates is stored as text in several formats, normalize them where it is cheap. If a column of numbers has thousands separators or currency symbols stuck to it, strip them. The model is robust to many oddities, and unusual or mixed formats can still lower confidence.

Turn placeholders into real gaps

Replace -999, N/A, empty strings and impossible dates with a true blank where you can. A real gap is an input the model understands. A fake value is a pattern it may learn by mistake.

Do not drop the empty cells

It can feel tidy to remove rows with gaps. Resist it. The pattern of what is missing is information, and the model is built to read it.

Leave identifiers as identifiers

Do not convert customer numbers into floats or sort and re-index them. If an identifier column is not useful for your task, the model will treat it as an identifier. If it leaks the answer, remove it yourself.

Do not worry about names

You do not need a data dictionary, and renaming columns to be tidy changes nothing for the encoder. If a name carries a business rule that the values cannot show, write the rule down for the humans who read the result.

Do not worry about order

Rows and columns can be in any order. Do not sort to help the model.

Quick reference

For readers who want the page in a form they can scan.

• Unit: the cell, one value in one row and one column.

• Vector parts: value, type, position. A missing cell has an explicit encoding.

• Shared space: numbers, categories, dates and text are all mapped into it.

• Width: the same for every cell, so one attention layer can read them all.

• Names: never used by the encoder. Column embeddings are always zeroed.

• Order: not imposed. Position marks membership of a row and a column.

• Gaps: inputs, not errors. Real gaps are treated like masked cells.

• Next stage: axial attention along rows and columns, then a Perceiver-style block.

• Your job: keep types honest, turn placeholders into true blanks, leave identifiers alone.

Edge cases and open questions

Honest notes on where cell encoding is hard.

• Very high-cardinality categories. A column with a different value in almost every row looks a lot like an identifier. The model reads it through its behaviour, and the line between a category and an identifier can blur.

• Mixed types in one column. A column with numbers in most rows and words in a few is read cell by cell. It works, and the type signal is noisier.

• Heavy-tailed numbers. Scale matters, and extreme values can dominate if they are not handled. This was a live issue on the output side of regression.

• Locale and format. 1.234,5 and 1,234.5 are the same number in different places. Reading them as different is an error that no model can fully avoid without help.

• Long free text. Cells with paragraphs of prose are a poor fit for a numeric-and-category-first encoder, and a model with a language component may do better.

• Names as the only clue. If the only information that settles a column's meaning is its header, the model does not have it.

These are the places where I would look first if a table behaved oddly. See Limitations for the wider list.

Frequently asked questions

What is cell encoding?

It is the step that turns each cell of a table into a vector, combining the cell's value, its type and its position. A missing cell gets its own explicit encoding. Numbers, categories, dates and text all land in the same space, so one network can read them together.

Why encode cells and not rows?

The cell is the smallest unit that still means something. Rows and columns can be built up from cells through attention, along rows and along columns. Encoding a whole row as one vector would squeeze the columns together before the model looked at them, and turning the table into text would throw away the grid.

Does the model use column names?

No. Column embeddings are always zeroed, so the encoder never gets a column's identity. It gets the values in a column and how they relate to the values in other columns. A column called c3 that holds phone numbers is understood as phone numbers. A name channel exists only in the schema-matching scorer.

How are missing values handled?

They are encoded explicitly, as their own kind of input, and not as an error. The model sees tables with holes during pre-training, learns to read around them, and learns that a gap can itself be a signal. Imputation then treats a real gap like a masked cell.

Does row or column order matter?

No. Position marks which row and which column a cell belongs to, and it does not impose a rank. Shuffle the rows or the columns and the output moves with them, so the answers stay the same.

How are numbers, categories and dates different?

Each has different statistics and meaning. Numbers need their scale and shape handled so that closeness is preserved. Categories need equal values to look equal without an invented order. Dates have order and cycles. The type tag tells the network which kind of cell it is reading, and all kinds are mapped into one shared space.

Why is the type needed if the value already shows it?

Because the same raw value can be read in different ways, such as digits that could be a date, a count or an identifier. The type removes the ambiguity and gives later layers a cheap, strong hint about how to compare values in a column.

What happens with placeholders like -999 or N/A?

If they are read as real values, they create a false pattern. Turn them into true blanks where you can. Where you cannot, expect lower confidence, because the model has to work out that the placeholder is not a measurement.

Does ignoring names make the model worse?

On tasks where meaning matters it makes the model better and more robust, and on pure numeric classics without semantics a stronger prior-based rival still leads by a few thousandths. When a name is the only clue, the model has nothing extra to lean on. The trade is described in Column name independence.

Can I change how cells are encoded?

The encoding is part of the pre-trained model, so you cannot change it for a single table. What you can do is give it better inputs: honest types, true blanks for placeholders, and identifiers left as identifiers.

Summary

Cell encoding turns every cell into a vector of the same width. The vector combines the value, mapped into a shared space for numbers, categories, dates and text, the type, and the position that says which row and column the cell belongs to. A missing cell is encoded openly, because the pattern of what is missing carries information.

Column embeddings are always zeroed, so the model reads values and their relations and never a header, and it is robust to random, missing or misleading names. Position marks membership, not rank, so row and column order do not matter.

Everything the model returns is built on these vectors, from the column profile and sector to imputation, matching and prediction. For the next step read Architecture, for how predictions use the encoded rows read In-context prediction, and for the wider field read Tabular foundation models.