Row-level prediction benchmark: how the tabular foundation model was built and measured

The row-level prediction benchmark asks one plain question.

Given a table with one column hidden, how well can a model predict that column for rows it has never seen?

This page explains how the board works, how I built the model to answer it, what made the difference, and what is still open.

The short version is this.

The model runs one default configuration and ranks first on the classification and binary boards at every board size I tested.

It is behind the leaders on regression and multiclass.

The nearest rival on most boards is a hand-tuned AutoML pipeline, not another foundation model.

There are no scores on this page on purpose.

Scores depend on the harness, the fold count and the snapshot date, so they live in the model card next to their protocols.

If you are new to the topic, start with Tabular foundation models and the Architecture page.

If you want the full list of benchmarks, see Benchmarks.

What this benchmark measures

Row-level prediction is the bread and butter of tabular machine learning.

You have a table.

One column is the target.

The other columns are the inputs.

You want to predict the target for new rows.

There are two flavours.

• Classification. The target is a label, such as churned or not churned. Binary means two labels. Multiclass means more than two.
• Regression. The target is a number, such as a price or a duration.

The board does not test one dataset.

It tests many public datasets at once.

A method that is great on one dataset and poor on the rest will not rise.

A method has to be good across the field.

Why "held-out" matters

Every score on the board is computed on rows that the method did not see while it learned.

The model reads the labelled rows as context.

The rows it is scored on were never part of that context.

This is the same logic as the held-out evaluation described in Evaluation.

Think of a school exam.

If the student has seen the exact questions in advance, the grade says nothing.

A held-out split is the sealed exam paper.

The TabArena-Lite style

I ran the board in the TabArena-Lite style.

That means one fold per dataset.

A fold is one way of cutting a dataset into a learning part and a scoring part.

One fold is cheaper than five.

It is also noisier.

For that reason I only trust a result when it holds across many datasets, and I check it at several board sizes.

I come back to this in the protocol section.

How methods are ranked: the Elo idea

A leaderboard needs a way to turn many dataset-by-dataset results into one order.

Here the order comes from an Elo-style score.

Elo is the system used in chess.

Every player has a rating.

When two players meet, the winner takes points from the loser.

A win against a strong player moves the rating more than a win against a weak player.

On a tabular board, a "match" is a pair of methods on one dataset.

The better score wins that match.

After many matches over many datasets, each method has a rating.

The rating tells you how often it beats others and how strong those others were.

Why Elo and not a simple average

An average of raw scores has a problem.

Datasets have different scales.

A big error on one dataset can swamp the rest.

Elo only cares about who won, not by how much, so no single dataset can dominate.

There is a second reason.

Elo works with a shared field.

Methods that were run on only some datasets can still be compared, as long as they meet each other often enough.

This matters because not every rival ran on every dataset.

What this means for reading the board

• A higher Elo means the method wins more head-to-head matches across datasets.
• The gap between two ratings is a statement about win probability, not about accuracy points.
• A method can be first by Elo and still lose on a particular dataset. That is normal.

For a related way to think about ranking, see Learning to rank, which covers ordering items by pairwise comparison in a different setting.

Who the model is compared with

The shared field includes two groups.

1. Hand-tuned AutoML pipelines. These are systems that search for a good model and good settings for each dataset. They often spend hours of compute per dataset. They are strong because they adapt to the data they see. Gradient-boosted trees such as XGBoost are a typical building block.
2. Published tabular foundation models. These are models pre-trained on many tables, such as TabPFN. They predict in context with no training on the new dataset.

The model on this page belongs to the second group.

It differs in one important way.

It runs one default configuration.

There is no search per dataset and no tuning knob that I turned for a given table.

Competitor numbers are never re-run by me.

They are third-party published values, cited to their primary source, with the snapshot date pinned when a board is live.

How I built it

I did not start with the model. I started with a written protocol.

Step 1: write the rules before the runs

On 13 June I wrote a benchmark protocol.

It set out the phases, the pass gates and the buyer-facing model card.

A pass gate is a rule that says: you may not move on until this check is green.

This sounds like paperwork.

It saved me.

Later, several of my biggest "improvements" turned out to be errors in the ruler.

Having the gates written down meant I could not talk myself into believing a number that had not passed.

Step 2: find out where the failure really is

The first real finding was about where the failure was not. It was not in the decoder.

A decoder is the part that turns the model's internal picture of a row into a prediction.

When early results were poor, the obvious suspect was the decoder.

So I tried nine different similarity-based decoders.

A similarity-based decoder predicts a row by finding similar labelled rows and borrowing their labels.

It is the tabular version of asking your neighbours.

All nine hit the same wall.

When nine different designs fail in the same way, the problem is not the design.

It is upstream.

Step 3: the supports were too small

The first cause was physical. The supports were too small.

A support set is the group of labelled rows the model can look at when it predicts a new row.

In the failing runs there were 24 supports.

The task needed 128 or more.

An everyday picture helps.

Imagine guessing the neighbourhood of a stranger by comparing them with 24 people you know.

You may find nobody similar.

With 128 or more people you probably will.

The decoder was not wrong.

It had too little to look at.

Step 4: the training prior

The deeper cause was the training prior.

A prior here means the kind of synthetic tables the model sees while it learns.

If the practice tables have a particular shape, the model gets good at that shape.

See Pretraining for how synthetic data is used.

Some benchmark targets are quantile-band targets.

The label is a band, such as "low, medium, high", cut from a number.

My training prior turned these into XOR-like patterns.

XOR is a rule where the answer depends on two inputs together, and neither input alone tells you anything.

Models find XOR patterns hard to learn from few examples.

A cluster-style prior removed the wall.

In a cluster-style prior, rows form groups, and the label follows the group.

That is closer to how many real tables behave, and it is easier to read from a small support set.

Step 5: direct retrieval over the supports

I also tried error-correcting output codes.

This is a trick from communications.

You spread a label over several bits so that a few wrong bits can be fixed.

In my setting it added only bit noise.

Each extra bit was one more thing to get wrong, and nothing fixed the errors.

So I kept direct retrieval over the supports.

The model looks at the supports and answers from them.

I report this because a failed idea is useful.

If you build something similar, you can skip it.

Step 6: make the retrieval mode leaner

With the wall gone, I made the retrieval mode leaner.

I changed it in small steps.

After each step I tested on the full board, not on one dataset.

That rule is worth stating twice.

A change that helps one dataset by luck is easy to find.

A change that helps across the board is rare.

The full-board test is the only test that counts.

What made the difference

Two things stand out.

One default configuration

The model runs one default configuration. It does not tune per dataset.

This matters because the rivals do.

A tuned AutoML pipeline gets hours of search on each dataset.

The model gets none.

That it wins on the classification and binary boards under this handicap is the main claim of this benchmark.

It also matters for a practical reason.

A user who has a new table does not have hours to spare.

The default setting is what they will get.

First on classification and binary, at every board size

The model is first on the classification and binary boards at every board size I tested.

Board size means how many datasets are included.

A small board can be won by luck.

A large board is harder to win by luck.

Testing at several sizes shows whether a lead is real or an accident of which datasets were picked.

The overall board needed a harness fix

The overall board only became valid after I found a harness error.

This is the most instructive part of the story.

A regression run had been filed under a different method name.

So the harness saw one method name with classification results and a different name with regression results.

For the first name, the regression datasets were missing.

The harness handled the missing results by filling them in with a penalty imputation.

In plain terms, it gave the method a bad score on datasets it never ran.

That dragged the overall rating down for the wrong reason.

I fixed it by merging the metadata so that both runs counted as one method.

Then I checked that no row was imputed.

That second check is the important one.

Fixing a bug is not enough.

You also verify that the bug is gone.

Check the ruler first

This episode is a good example of a rule I follow on every benchmark.

The largest gains often come from fixing the harness, not the model.

Before I trust a comparison, I check that the folds, the masks, the averaging rules and the method names are all consistent.

The same habit shows up in Missing-value imputation and Missing-data robustness.

How the model reads a row

It helps to know a little about how the model gets from a table to a prediction.

The full story is in Architecture and Cell encoding.

Here is the short version.

1. Every cell is turned into a vector that carries its value, its type and its position.
2. Axial attention lets cells in the same row talk to each other, then cells in the same column.
3. A Perceiver-style block, see Perceiver IO, gives the model a table-level summary.
4. A prediction head reads the labelled rows as context and answers for the new rows.

The process is described in In-context prediction.

No weights change when you give it a new table.

The labelled rows are the only teaching signal.

This design is also why the support-size finding mattered.

In-context prediction is only as good as the context.

If the context has too few useful rows, no decoder can recover.

Reading the results honestly

Where the model is strong

• Classification boards.
• Binary boards.
• Every board size tested.
• Under one default configuration.

Where the model is behind

Regression and multiclass are behind the leaders.

I do not hide this.

The summary line of the timeline says the model is first on the published boards "except where an entry says otherwise".

This is one of those places.

For regression, the better story is on a different benchmark.

That one scores a full predicted distribution, not a single number.

See Probabilistic regression.

Who is second

The second place on most boards is a tuned AutoML pipeline, not another foundation model.

This is a useful fact for two reasons.

First, it shows that the foundation-model approach has a real lead over other foundation models.

Second, it shows that careful tuning with hours of compute still matters.

A reader who needs the last drop of accuracy on one dataset, and has time, may still prefer a tuned pipeline.

I would rather say this plainly than let a reader find it out later.

The protocol, step by step

Every benchmark on this site follows the same standing rules.

Here is how they apply to row-level prediction.

Rule 1: competitor figures are third-party values

I never re-run a rival.

I take its published value, cite the primary source, and pin the snapshot date for live boards.

If I re-ran a rival badly, I could make it look worse than it is.

If I re-ran it well, I would just be copying the authors.

Using their own numbers avoids both problems.

Rule 2: one sealed configuration

My figures come from one sealed configuration per benchmark.

Sealed means I decide the settings before the final run and do not change them afterwards.

This guards against a quiet form of cheating.

If I could change settings after seeing the result, I would tend to pick the ones that look best.

The board would then reflect my choosing, not the model.

Rule 3: no per-dataset tuning

I do not tune per dataset.

The configuration that runs on the first dataset is the same one that runs on the last.

Rule 4: do not blend different measurements

Results measured under different datasets, metrics or masking schemes are never blended into one number.

A classification board and a regression board are separate.

A one-fold board and a five-fold board are separate.

Rule 5: check for contamination

Each run is checked for contamination against the training corpus.

Contamination means that test data leaked into training.

If the model had seen the test rows during pre-training, its score would be inflated.

I used an identifier blocklist and a distributional near-duplicate check.

The first catches known dataset identifiers.

The second catches copies that have been renamed or lightly changed.

The contamination check is described in more detail in Unified checkpoint.

A table of the rules

Common mistakes on prediction boards

I made some of these myself. Here is a list that may save you time.

Missing results filled with a penalty

As in the harness error above, a missing run can be silently filled in.

Always count the rows and check for imputation before you read a rating.

Comparing a one-fold board with a five-fold board

Five folds average out luck.

One fold does not.

A rating from one fold has a wider error bar.

Do not put the two side by side without saying so.

Winning on one dataset

It is easy to find a change that helps one dataset. Test on the whole board.

Trusting a decoder fix when the data is the problem

Nine decoders failed the same way in my case.

The data setup was at fault.

When many fixes fail alike, look upstream.

Forgetting that the field matters

An Elo rating is relative to the field.

If the field changes, so does the rating.

That is why the snapshot date is pinned.

Why the support set mattered so much

The support-set finding deserves its own section, because it is the idea I would carry to other projects.

In in-context prediction, the model does not learn weights on your table.

It reads labelled rows and compares new rows with them.

The labelled rows it can see are the supports.

If the supports are few, three things go wrong.

• Coverage. Some regions of the data have no nearby supports, so the model has nothing to compare with.
• Noise. With few supports, one odd row can sway the answer.
• Class balance. A rare class may have no supports at all.

With 128 or more supports, these problems shrink.

The model sees more of the shape of the data.

So the lesson is simple.

Before you blame the model, ask how much it is allowed to look at.

What stayed open

Honesty about open items is part of the protocol.

• Regression. The model is behind the leaders on regression boards.
• Multiclass. The model is behind the leaders on multiclass boards.
• Tuned pipelines. On most boards the second place is a tuned AutoML pipeline. A user with time to tune may do better than the default.
• One fold. The board is run with one fold per dataset. That is cheaper and noisier than five folds. I counter this by testing at several board sizes.

These are the places to look for future work. I do not claim to have closed them.

What this means for a user

If you have a classification or binary problem and no time to tune, the default configuration of the model is a strong start.

You give it a table with labelled rows and it predicts the rest.

See Capabilities for the outputs and Solutions for the questions it answers.

If you have a multiclass or regression problem and you need every last point, test the model against a tuned pipeline on your own data.

The Limitations page lists what the model does not claim.

Either way, read your own held-out score.

A board score tells you how a method does across many public datasets.

Your own table is the only table that matters to you.

Frequently asked questions

What is the row-level prediction benchmark?

It is a set of public datasets on which methods predict a held-out target column from the other columns.

It covers classification and regression, and methods are ranked against a shared field.

The aim is to see which methods generalize across many tables, not just one.

What does Elo mean on a tabular board?

Elo is a rating system borrowed from chess.

Each pair of methods on each dataset counts as a match, and the better score wins.

After many matches, each method has a rating that reflects how often it wins and how strong its opponents were.

It is a way to rank methods without letting one dataset dominate.

Why does the model not tune per dataset?

A foundation model is meant to work out of the box.

Tuning per dataset would give an advantage that a new user cannot match without hours of compute.

Running one default configuration shows what a user would actually get, and it keeps the comparison honest.

Where is the model strongest?

It is strongest on the classification and binary boards, where it is first at every board size I tested.

It does this with one default configuration against rivals tuned for hours per dataset.

That lead held at each board size.

Where is the model weaker?

Regression and multiclass boards are behind the leaders.

For regression there is a separate benchmark that scores whole distributions, and the story there is different.

See the probabilistic regression page for details.

What was the biggest bug you found?

The overall board was wrong because a regression run was filed under a different method name.

The harness treated the missing datasets as failures and filled them with a penalty imputation.

I merged the metadata and checked that no row was imputed.

After that the overall board was valid.

Why did the support set matter?

In-context prediction can only use the labelled rows it can see.

With 24 supports the model often had no close neighbours.

With 128 or more it did.

Nine different similarity-based decoders failed the same way until the supports were large enough, which showed the problem was upstream.

Why are there no scores on this page?

Scores depend on the harness, the fold count and the snapshot date.

A number without those details is easy to misread.

The model card carries the scores together with their protocols, so each number can be checked.

Who is second on most boards?

A tuned AutoML pipeline, not another foundation model.

Hours of per-dataset search still buys real accuracy.

I state this openly so a reader can choose the right tool for the job.

How do I know the test data was not in training?

Each run is checked against the training corpus with an identifier blocklist and a distributional near-duplicate check.

The first catches known datasets by name.

The second catches renamed or lightly edited copies.

Both are described in the unified checkpoint page.

Summary

The row-level prediction benchmark ranks methods by Elo on many public datasets, one fold per dataset, against hand-tuned AutoML pipelines and published tabular foundation models.

The model runs one default configuration and is first on the classification and binary boards at every board size I tested.

It is behind the leaders on regression and multiclass, and the second place on most boards is a tuned pipeline.

The work that mattered most was not clever decoding.

It was finding that supports were too small, that a training prior created XOR-like patterns, and that a mislabelled run made the harness impute a penalty.

Fix the data, then fix the ruler, then trust the number.

Read next: Probabilistic regression, Anomaly detection, Continual learning, and the full benchmarks timeline.