Limitations

Every model has edges, and a model that hides them is harder to trust.

The short answer is that the model reads the table you give it and predicts in context from the labelled rows it finds there. Everything it does well follows from that, and so does everything it cannot do. If the table is a poor picture of the rows you care about, no amount of model quality repairs it.

The limits fall into six groups. They are limits that come from your data, limits that come from the method, limits of the evaluation, limits of particular tasks, limits of specific capabilities, and limits in how the results should be used. They are listed in that order below, and a table near the end puts them side by side.

For the evidence behind the strengths, read Evaluation and Benchmarks. For the design that causes the trade-offs, read Architecture.

How to read this page

The limits here are of three kinds, and it helps to know which kind you are looking at.

• Hard limits. Things the model does not do at all, such as predicting a class that never appears in the table.

• Soft limits. Things that get worse as conditions get worse, such as accuracy on a tiny sample.

• Measured gaps. Places where the benchmarks show the model behind a rival or less certain, such as the regression and multiclass boards.

I try to say which kind each item is. A hard limit needs a different workaround from a soft one, and a measured gap needs a different response from either.

Limits that come from your data

It reads the table you give it

The model's quality depends on how representative the table is of the rows you care about. This is the most important limit, and the most common cause of disappointment.

A foundation model does not know your business. It knows how tables behave, and it reads your table to learn what is true in this one.

They will do a good job on what they saw. The forecast is still limited by the week.

If the table over-represents one region, one season or one customer group, the predictions will lean the same way. The model cannot correct a sampling mistake it cannot see.

Predictions are in context

Predictions reflect the labelled rows provided. The model compares each new row with the labelled rows in the support set, and it reads an answer off that comparison. See in-context prediction.

That is why no training run is needed. It is also why very small or heavily biased samples limit what can be learned from them.

There are three practical consequences.

• Few labels, weak neighbours. With very few labelled rows, a new row has few similar rows to compare against.

• Rare classes may be starved. A class with a handful of examples gives the model a handful of neighbours, so recall on that class will be low.

• Bias passes straight through. If the labels are biased, the support set is biased, and the prediction follows it.

An early lesson from my own work shows how much the support set matters. When I tried nine similarity-based decoders for row-level prediction, they all hit the same wall. One cause was that the support sets were physically too small, 24 supports where 128 or more were needed. The fix was not a cleverer decoder, it was giving the method enough supports. The same logic applies to a table with too few labelled rows.

Labels can be wrong

A label is a measurement made by someone or something, and it can be mistaken. Mislabelled rows in the support set mislead the model, and mislabelled rows in the held-out part make the score look worse or better than the truth.

The model has no way to know that a label is wrong unless the rest of the row contradicts it. Cleaning labels, or at least sampling a few to check, is cheap and often more valuable than any change to the model.

Leaky columns look like skill

If a column contains the answer in disguise, such as a refund-processed flag in a table that predicts churn, the model will find it and score well. The score is real on the table and useless in production, because the column will not exist at the moment of prediction.

The model cannot tell a leaky column from an honest one. You can, because you know when each column becomes available.

Duplicates and near-duplicates

If the same row appears in the context and in the held-out part, the model will recognize it and the score will be inflated. Exact duplicates are easy to remove. Near duplicates, such as the same customer with a slightly different address, are harder.

A held-out score on a table with many near duplicates measures memory more than generalization.

A different distribution is a different problem

A held-out score describes unseen rows of the same table. It does not describe a different distribution.

It does not cover:

• a new market with different customers,

• the same customers after a policy change,

• a sensor after a recalibration,

• or any table whose columns mean something subtly different from the table you tested on.

When the world changes, the support set goes out of date. Refreshing the data and re-measuring is the remedy. The model has no way to notice the change by itself.

Classes and categories absent from the table

The model predicts from the labelled rows it can see. A class that never appears among them cannot be predicted as a class, and a category value it has never met is read from its context and not from a dictionary.

If new classes appear over time, the table must be refreshed to include them. This is a hard limit.

Rows with a missing target are not predictable labels

In classification, rows whose target is missing are counted as dropped. They never become a predictable label. If a large share of your target is missing, the model is learning from a subset, and that subset may not be typical.

Limits that come from the method

Reading a support set, not every row

Attention along rows and columns is cheaper than attention over every cell, and the long axis of a table is still its rows. That is one reason the model is designed to read a support set of rows and not every row of a very large table at once.

In practice this means that for large tables the model works from a sample of labelled rows. A well-chosen sample is usually enough, and a poorly chosen one is not. If your table is huge and highly structured, check that the sample covers the structure.

Very wide tables, with hundreds of columns, strain the column side. Very long tables strain the row side. The Perceiver-style latent block keeps the whole-table step linear in the number of cells, and the axial layers remain the part to watch. See axial attention and Perceiver IO.

Attention treats a table as a grid, not a graph

The model understands rows and columns inside one table, and it understands relationships between tables through its cross-table head. It does not reason about an arbitrary graph of linked tables with many levels of foreign keys unless you hand it the relevant tables.

If a fact lives three joins away, bring it into the table first.

The prior is part of the model

The model was pre-trained on a blend of real and synthetic tables. A synthetic generator encodes beliefs about what tables look like, and the model absorbs them. If the prior has a flaw, the model inherits it.

I saw this directly. The training prior turned quantile-band targets into XOR-like patterns, patterns in which neither input alone predicts the output, so similarity-based decoders failed in the same way again and again. A cluster-style prior removed the wall.

The lesson for users is modest. If your tables are very unlike anything a generator would produce, such as strange hierarchies or highly engineered feature crosses, expect lower confidence. See Pre-training.

Retrieval has a ceiling

The prediction head works by retrieval over the supports, and I kept it that way after error-correcting output codes added only bit noise. Retrieval is strong when similar rows predict similar outcomes.

It is weaker when the outcome depends on a rule that no neighbour reveals, such as a parity-like combination of columns. Gradient-boosted trees and other learned models can capture some rules of that kind from a large table, and an in-context method may not.

No explanation of the causal kind

The model finds patterns and does not establish causes. A feature that predicts churn is not necessarily the reason for churn. Do not turn a predictive pattern into an intervention without a causal argument.

Limits that come from the evaluation

A held-out score is local

It describes unseen rows of the same table, drawn the same way as the context rows. If you will use the model on later data, split by time. If rows cluster by person or site, split by group. See Evaluation.

A random split on a time-dependent table can flatter the model by letting it peek at the future through neighbouring rows.

Lite protocols are lighter tests

On the row-level boards I ran the lite style of the protocol, with one fold per dataset. A single fold gives less information than a repeated cross-validation, and the result carries that caveat.

Competitor figures are third-party values

I do not re-run rivals. Their figures are published values, cited to their sources, with the snapshot date pinned for live boards. That keeps me honest and also means my comparison is only as current as their publication.

If a rival improved after a snapshot, the benchmark page does not know.

One sealed configuration is a strict test and also a limited one

The model gets a single default configuration, with no per-dataset tuning. That makes the result closer to what you would see on a table I never saw. It also means that on a specific table, a tuned configuration might do better, and I have not claimed otherwise.

Narrow leads are narrow

In missing-data robustness the mean AUC is above the published method, and the lead is narrow. A further ladder of tuning gave nothing, so I stopped. It does not mean the model wins on every dataset.

Results are never blended

Results measured under different datasets, metrics or masking schemes are never blended into one number. That means there is no single headline figure for the whole model, by design. If you want one, you will have to choose the benchmark that resembles your task.

Contamination checks are strong, not absolute

Each run is checked against the training corpus with an identifier blocklist and a distributional near-duplicate check. That catches the obvious cases. It cannot prove that nothing about a benchmark reached the corpus in some unrecognized form.

Harness errors are a real risk

Several of the largest swings in my own results came from measurement mistakes, not from the model. I fixed those I found. I cannot promise there are none left, and neither can anyone else who builds a harness. That is why I align plain baselines to their published values and verify masks before running a grid.

Limits of specific capabilities

Row-level prediction

The model is first on the classification and binary boards at every board size I tested. Regression and multiclass are behind the leaders.

On most boards the second place is a tuned AutoML pipeline, not another foundation model. That tells you something about the competition: a pipeline tuned for hours per dataset is a strong opponent.

If your problem is regression or multiclass and the table is large, compare against a tuned tree ensemble on your own split. See Row-level prediction.

Probabilistic regression

Interval coverage was the weak metric before calibration. After split-conformal calibration with an isotonic correction, the intervals cover as often as they claim on the benchmark. Mean absolute error and RMSE were not the target of the distribution head, so do not read it as the best point predictor.

Conformal calibration gives coverage on average, under the assumption that new rows resemble calibration rows. It does not promise coverage for a specific row, and it does not hold if the data drifts.

CRLS punishes a distribution that is too narrow. Heavy-tailed targets such as house prices, runtimes and car prices needed a log transform, and a target with an unusual shape may still need care. See Probabilistic regression.

Imputation

Imputed values are estimates. The model fills a gap from the joint structure of your own data and attaches a confidence at the row level, and the original cells are never overwritten.

Confidence is not certainty. Low-confidence rows deserve review. Do not treat an imputed value as an observation, and do not feed imputed values into a downstream decision about a person without a flag that says they are imputed.

The benchmark results are strong on the real-data board, with the widest margin under not-at-random missingness. They are not uniform.

• On two synthetic settings, the best published MICE figure stayed ahead of my own runs, and I left those rows as reported.

• On a few datasets with few columns or few rows, the chain has little context and the gain is small.

• Categorical imputation has no published competitor, so it has no external reference point.

Under not-at-random missingness, the reason for the gap is the very thing you cannot see. No method can fully recover a value whose absence depends on the value itself. The model is better than the alternatives, and it is not magic. See Imputation.

Anomaly detection

The benchmark is semi-supervised. The detector is fitted on normal rows only and scored on a mixed test set. If your history contains many undetected anomalies, the model will treat them as normal.

An anomaly score says a row is unusual, not that it is bad. A genuinely rare but legitimate customer will score high. The threshold is a choice, and it trades false alarms against missed cases.

The pseudo-anomaly selector chooses channels without labels by building fake anomalies, with shuffled features and uniform values. That is a good proxy, and it is still a proxy. If your real anomalies are of a very different kind, such as a subtle drift over weeks, the proxy may not rank channels well. See Anomaly detection.

Cross-table matching

The cross-table map is a proposal, not a command. It carries a confidence and evidence, a verbatim example from each side, and it goes to a review queue. It should never be applied automatically to records that matter.

The schema matcher keeps only mutual best one-to-one matches. A pair survives only if each column picks the other as its best match. That is a good rule for clean correspondences. It misses cases where one column in one table corresponds to two columns in another, such as a full name split into first and last.

Entity matching works without keys, and the margin is widest where text matching gives out, for records that describe the same entity in different words. It remains a probabilistic judgement.

One published competitor was trained on the benchmark data, which is a leakage caveat for them. That caveat does not remove the need to check the model's matches on your own data. See Entity and schema matching.

Sector identification

The sector of a table is returned as a top guess and a short list, each with a confidence. The remaining errors are sibling-sector confusions such as banking and insurance, and 24 datasets are ambiguous even for human labellers, so they are reported separately.

I confirmed the ceiling in four ways: eighteen failed attempts to push past it, the structure of the signal, a three-judge audit of the labels that corrected three of them, and the literature on semantic type detection.

Two practical limits follow.

• If a table belongs to two industries at once, the model will give a ranked list and not a single clean answer.

• The sector is a property of the whole table. A small extract of a few columns carries less signal than a full table.

The label set also matters. The score is against a fixed list of sectors, and a table from an industry outside that list will land on the nearest neighbour. See Blind sector identification.

Continual learning

The model can take on new tasks through adapters, a replay buffer and regularization. Be clear about what that gives.

Retention comes from replay, not from the weights alone. The recipe that stayed keeps 48 raw rows per task, and the buffer grows linearly with the number of tasks.

An earlier generation reports a higher number on a different task sequence, so the two cannot be compared directly. There is no published competitor to benchmark against, because tree ensembles retrain fully for each task and frozen foundation models have no update path.

Results are sensitive to seeds. A missing random seed alone moved results by 15 to 20 percent. If you run your own continual learning, seed it and report the seed. See Continual learning for tables.

Name independence

Names help but are not required. The model reads values, and unusual or mixed formats can still lower confidence.

Because column embeddings are always zeroed, the encoder cannot use a header even when the header is the only clue. A column called risk_tier that holds the digits 1, 2 and 3 is read as a small integer category, not as a risk tier.

On pure numeric classics without semantics, a stronger prior-based rival still leads on the metadata benchmark, usually by a few thousandths. The benefit of name independence is largest where meaning matters. See Column name independence.

Synthetic data

Synthetic data is a stand-in. It is useful for testing, for sharing without exposing real records, and for filling thin data. It is not a substitute for real records in a final decision.

Synthetic rows inherit the generator's beliefs and the source table's patterns. They can look plausible and still miss a rare real-world case. Treat them as scaffolding, and say plainly that they are synthetic whenever you share them.

Compression

The model can be stored at half precision, 8-bit and 4-bit. A 4-bit weight takes a quarter of the space of a 16-bit weight, and the rounding adds error.

Fidelity was measured against the full-precision model on the real tasks, and it is still worth re-checking on your task if you serve a compressed copy. See Compression and serving.

Limits in how results are used

Confidence is not certainty

Imputed values carry a confidence at the row level, and predictions carry probabilities. Those numbers are the model's own estimate of how much it knows.

They are useful, and they are not guarantees. Calibration can drift when the data does. A ninety percent claim should be checked by binning predictions and comparing with outcomes.

Decisions about people

A model trained on past decisions reproduces past patterns, including patterns that were unfair.

Held-out accuracy does not measure fairness. Break results down by group, and ask whether the errors fall evenly.

Personal data stays personal

The model can flag columns that hold personal data, and it reads the values you give it. Reading is not a privacy review. Decide what may be sent, and mask what must not leave.

The replay buffer in continual learning stores raw rows, so it needs the same care as any other copy of the data.

Not a substitute for domain review

The model can tell you what a column looks like and how rows relate. It does not know your regulations, your contracts or your unwritten rules.

It cannot know that one of them is legally forbidden as an input. Domain review is a separate job.

Not a database

The model returns understanding and predictions. It does not store, version or secure your data. Treat it as an analysis layer, and keep the system of record where it belongs.

Worked examples of limits in practice

Abstract limits are easy to nod at and forget. These short scenarios show how each one looks when it happens. They are illustrations, with invented details, and not results.

The quiet week

A team gives the model a table of orders from one slow week and asks it to predict which orders will be returned. The held-out score is good.

Then a holiday sale begins, the mix of products changes, and the return rate doubles. The model is not wrong about the week it saw. The table was not a picture of the sale.

The fix is not a better model. It is a table that includes sale periods, and a time-based split that tests the model on a later period than the one it reads.

The helpful column

A table that predicts loan default contains a column called collections_status. The held-out score is excellent.

That column is only filled in after a loan goes bad. In production it is empty at the moment of the decision, and the model has nothing to lean on.

The model cannot know when a column becomes available. You do, so the review of columns for leakage belongs to you.

The rare class

A table of equipment failures has 20,000 normal rows and 15 rows of one particular failure. Overall accuracy is very high, because the normal rows dominate.

Per-class results tell the true story: recall on the rare failure is low, because a new row has few labelled neighbours from that class. This is why reports show per-class results and a baseline, and why the weakest slice is named.

Collecting a few more examples of the rare class helps more than any tuning would.

The renamed column

A legacy extract arrives with columns called val_B and metric_14. The model reads the values and works out the role of each column from their shape and from how the columns move together. That is what it was built for.

Then a column holds the digits 1 to 5, and in the original system they were severity grades with a strict order. The model reads a small set of numbers. Whether the order matters is something the values alone may not settle, and the confidence reflects that.

If the meaning is a business rule, write it down and pass it along. Names help, and so do notes.

The merged customers

Two systems share no keys. The cross-table map proposes that a row in one belongs to the same person as a row in the other, with a confidence and the attributes it matched on.

Most proposals are correct. A few join two relatives who share an address and a surname. The proposal goes to a review queue, and a person decides.

That is the design working as intended. The limit is that the model proposes, and it does not decide.

The filled gap

A health table has a blank for a lab value. The model fills it with an estimate and a row-level confidence.

A downstream report averages the filled value together with measured values and presents the average as a clinical fact. The estimate has quietly become an observation.

The remedy is a flag that travels with every imputed cell, so that reports can separate what was measured from what was inferred.

The very wide table

A table has several hundred columns and a modest number of rows. Column attention has a lot to do, and the support of rows for any pattern is thin.

The model still reads it, and the result is more uncertain than for a table with a dozen well-filled columns. Dropping columns that are constant, duplicated or irrelevant often helps more than expected.

The comparison that did not transfer

A team reads that the model leads a benchmark and expects the same lead on their own regression table. On their split, a tuned tree ensemble does better.

Nothing is wrong. The benchmark claims what it measured, under its protocol, and the regression boards are a place where the model is behind the leaders. Their own held-out comparison is the right guide for their table.

Limits by role

Different readers meet different limits first. This section gathers the ones that matter for each role.

If you are an analyst

Your main limits are representativeness, leakage and the split. Check the table before you check the model.

Read per-class results and the weakest slice. Report the baseline alongside every score, so that nobody mistakes a majority-class guess for skill.

If you are an engineer

Your main limits are scale, formats and monitoring.

Keep the system of record elsewhere, and keep a version of the data that matches each report. A held-out score without its snapshot cannot be repeated.

If you are a data steward

Your main limits are personal data, replay buffers and review queues. The model flags columns that look personal, and flagging is not a policy.

The continual-learning buffer holds raw rows, and cross-table proposals should never be applied without review. Decide who owns each queue.

If you are a decision maker

Your main limit is the difference between a score and a promise. A held-out score is local, a benchmark result is protocol-specific, and confidence is not certainty.

Ask for the baseline, the split, the slices and the weakest case. A report that cannot answer those is not ready.

Questions the model cannot answer

It helps to know the kind of question that is out of scope, so that you do not try to force an answer.

• Why did this happen? The model finds patterns, and does not establish causes.

• What will happen if we change the policy? A policy change moves the world away from the table. The model predicts for the world the table describes.

• Is this legal or fair? Neither can be read off the values. Both need review by people who know the rules.

• Which of these columns is true? The model can say that two columns disagree. It cannot know which one is the correct record.

• What does this table mean to the business? It can name the industry and the role of each column. The meaning to your business lives in your head.

Limits that are about honesty

Some limits are less about the model and more about how results are reported. These are the habits I follow and the weaknesses they guard against.

• I say when a lead is narrow. A narrow lead is not dominance.

• I leave rows as reported when a rival is ahead. I do not drop the awkward setting.

• I separate clean subsets from full sets. After a colleague objected that a perfect score on an easy subset proves nothing, I made the full 504 datasets the headline for sector identification and kept the clean subset as a sanity check only.

• I drop wins that sit inside seed noise. A result that cannot beat its own variation is not a result.

• I name the caveats of rivals too. One published matching competitor was trained on the benchmark data, and I note it next to the result.

• I do not blend protocols. There is no single magic number.

These habits do not remove the limits. They make sure the limits are visible when you read a result.

All the limits side by side

A review checklist before you rely on a result

Use this list as a quick pass. Each item matches a limit above.

• Is the table representative of the rows I will score, in time, place and customer mix?

• Do I have enough labelled rows for every class I care about?

• Have I checked a sample of labels, and removed columns that will not exist at prediction time?

• Did I remove duplicates between the context and the held-out part?

• Does the split match the use, by time or by group where that applies?

• Is the baseline written down, and is the lift real over it?

• Have I read the weakest slice and not just the average?

• Do low-confidence rows go to a person?

• Are imputed values flagged as imputed downstream?

• Is a human the final step for matches and decisions that matter?

• Will I measure live accuracy after launch, and compare it with the held-out number?

What would change these limits

Some limits are fixed by the method. Others move with work, and I try to be careful about which are which.

• More labels move the soft limits. More and better labelled rows help every in-context prediction.

• Fresher data moves the drift limit. Refreshing the table and re-measuring is the direct remedy for a changed world.

• Better priors move the inherited-prior limit. A cluster-style prior removed one wall, and a prior is a thing that can be improved. Any change to it is tested on the whole board before it is kept.

• More published rivals move the comparison limits. As more methods publish on a board, the comparison improves. I cannot control that.

• Some limits are structural. A model that only reads what you give it will not know what you did not give it. No version of the method will change that.

I do not promise dates or results for any of these. The honest position is that the limits above hold today, and that each is written next to its evidence.

Frequently asked questions

Does the model work on any table?

It reads any table, with any column names, and returns an understanding of it. How well it predicts depends on how representative the table is of the rows you care about. A small, skewed or stale table limits the result, and the model cannot correct a sampling mistake that it cannot see.

What does a held-out score not tell me?

It does not tell you how the model performs on a different distribution, after the world changes, or on classes that never appear in the table. It describes unseen rows of the same table. Live accuracy from labels that arrive after launch is a separate measurement and should be tracked on its own.

Can I trust the confidence numbers?

Use them as a guide and not as a guarantee. Imputed values and predictions carry a confidence, and low-confidence rows deserve review. Calibration can drift when data changes, so bin the predictions occasionally and compare with outcomes.

Is the model worse at regression than classification?

On the row-level benchmark boards, the model is first on classification and binary tasks and behind the leaders on regression and multiclass. On most boards the second place is a tuned AutoML pipeline. For probabilistic regression, the picture is different: the model has the lowest median CRPS and the best mean rank on the merged board. Compare on your own split if the choice matters.

Why does the model not use column names?

Column embeddings are always zeroed, so the encoder reads values and their relations to other columns and never a header. That makes it robust to random, missing or misleading names. The price is that when a name is the only clue, the model has nothing extra to lean on, and unusual formats can lower its confidence.

Are imputed values safe to use as real data?

Treat them as estimates with a confidence. Original cells are never overwritten, and low-confidence rows should be reviewed. Under not-at-random missingness no method can fully recover a value whose absence depends on the value itself. Flag imputed values downstream so that nobody mistakes them for observations.

Can the model match records automatically with no human check?

It can propose matches with a confidence and evidence, and the proposals go into a review queue. The schema matcher also keeps only one-to-one matches, so a column split into two will be missed.

Does continual learning mean the model never forgets?

No. Retention comes from replay and from an adapter design that does not depend on the number of columns, not from the weights alone. Results are also sensitive to seeds.

Is synthetic data as good as real data?

No. It is useful for testing, for sharing without exposing real records, and for filling thin data, and it is not a substitute for real records in a final decision. It inherits the beliefs of its generator and can miss rare real cases.

What are the biggest open problems?

Regression and multiclass boards behind the leaders, sibling-sector confusion, the dependence of continual-learning retention on replay, imputation on tables with few rows or columns, and the plain fact that a score is local to its data. Each is written next to its evidence in the benchmark pages.

Summary

The model reads the table you give it and predicts in context from the labelled rows present. Its quality is bounded by how representative that table is, and a held-out score is local to the table, the split and the snapshot.

Measured gaps are stated plainly: regression and multiclass boards behind the leaders, interval coverage weak before calibration, a published imputation figure ahead on two synthetic settings, sibling-sector confusions, and retention that depends on replay. Narrow leads are called narrow.

Confidence is not certainty, synthetic data is a stand-in, names help but are not required, and matches and decisions about people need a human at the end. For the evidence read Evaluation and Benchmarks, and for what the model can do, read Capabilities.