# Benchmarks

What each benchmark measures, how I built the capability that answers it, and what made the difference, in the order the work was done.

> Informational documentation, based on the paper Data Language Models: A New Foundation Model Class for Tabular Data (arXiv:2605.06290, https://arxiv.org/abs/2605.06290). Reference material, not a hosted service; hosts, keys, and ids are placeholders.


This page is a timeline, not a leaderboard. Each entry says what the benchmark is, how I built the capability that answers it, and what made the difference against the rivals. Scores are left out on purpose: they depend on the harness, the fold count and the snapshot date, and the model card carries them with their protocols. The result is that the model is first on the published boards for these tasks, except where an entry says otherwise.
**Standing rules for every entry.** Competitor figures are never re-run by me; they are third-party published values cited to their primary source, with the snapshot date pinned for live boards. My figures come from one sealed configuration per benchmark, with no per-dataset tuning. Results measured under different datasets, metrics or masking schemes are never blended into one number. Each run is checked for contamination against the training corpus.
## The timeline
June to July 2026
## Row-level prediction boards
Classification and regressionWhat it measuresHow well a model predicts a held-out target from the other columns, on many public datasets at once. Methods are ranked with an Elo-style score against a shared field that includes hand-tuned AutoML pipelines and published tabular foundation models. I ran the TabArena-Lite style: one fold per dataset.How I built itI started by writing a benchmark protocol on 13 June: phases, pass gates and the buyer-facing model card. The first real finding was that the failure was not in the decoder. Nine similarity-based decoders hit the same wall, because the support sets were physically too small (24 supports, where 128 or more were needed) and, deeper, because the training prior turned quantile-band targets into XOR-like patterns. A cluster-style prior removed the wall. Error-correcting output codes added only bit noise, so I kept direct retrieval over the supports. Then I made the retrieval mode leaner and tested each change on the full board, not on one dataset.What made the differenceThe model runs one default configuration and wins against rivals that were tuned for hours per dataset. It is first on the classification and binary boards at every board size I tested. The overall board only became valid after I found a harness error: a regression run filed under a different method name made the harness fill the missing datasets with a penalty imputation. I merged the metadata and re-checked that no row was imputed.What stayed openRegression and multiclass are behind the leaders. The second place on most boards is a tuned AutoML pipeline, not another foundation model.13 July 2026
## Continual learning
Sequential tasksWhat it measuresRetention across a long sequence of fine-tunes: after learning task after task, how much of the first tasks is still there. There is no published competitor. Tree ensembles retrain fully for each task, and frozen foundation models have no update path at all.How I built itThe first runs forgot badly, and the cause was an architectural bug: the adapter was re-created from scratch for every task, so each task erased the last. After the fix I checked that the adapter did not depend on the number of columns, so its weights carry across tasks. Then I ran controlled experiments: warm starts, an epoch brake, an L2 anchor that failed, and seeded runs once I found that a missing random seed alone moved results by 15 to 20 percent. The recipe that stayed is experience replay with 48 raw rows per task. A pseudo-rehearsal variant had a higher mean but a terrible worst case, so I rejected it.What made the differenceThe model is the only one in the comparison that can take on new tasks without a full retrain. Retention is measured with a fixed formula, and I also report backward transfer.What stayed openRetention here comes from replay, not from the weights alone, and the buffer grows linearly. An earlier generation reports a higher number on a different task sequence, so the two cannot be compared directly.13 July to 3 August 2026
## Anomaly detection
Semi-supervised ADBenchWhat it measuresRanking rows that do not fit the normal distribution. The model is trained on normal rows only and scored on a test set, the setting in which the published methods are measured. Ten classical detectors and published model pools are the rivals.How I built itMy first chain used the wrong engine, the weaker imputer. I switched to the imputation engine and then fixed its integration bugs one at a time: the output scale (a per-column linear calibration), an empty head when the data has no missing values (I inject 15 percent random missingness before fitting), and import-time side effects. The final detector has three model-based channels: the residual of the imputation head, nearest neighbours in the model's embedding space, and nearest neighbours in the raw space. Isolation forest joins as an auxiliary channel. Which channel to trust is decided without labels, by a pseudo-anomaly selector: I build fake anomalies from the training rows by shuffling features and injecting uniform values, measure each channel on them, and weight the channels with a softmax. The ranks are then fused.What made the differenceNo labels are needed to choose how to combine the signals, and the signals come from what the model has learned about the table rather than from distances alone. The detector is first among thirteen methods by both average rank and AUROC.What stayed openThe earlier uniform fusion was kept in the history. I also defended the final claim with an independent pass.17 July to 5 August 2026
## Probabilistic regression
ScoringBenchWhat it measuresPredicting a whole distribution, not a single number, and scoring it with proper scoring rules: CRPS and CRLS. The board carries 102 regression datasets and the published outputs of 50 models.How I built itI trained a 999-quantile head with a CRPS loss, following the benchmark paper's own finding that you win on the scoring rule you train for. That gave first place on CRPS. CRLS lagged, and the diagnosis was precise: the target was standardized by its standard deviation, which for heavy-tailed targets (house prices, runtimes, car prices) made the predicted distribution too narrow, and CRLS punishes a narrow distribution hard. The fixes were a log transform for heavy tails, a CRLS fine-tune that I later dropped when both metrics got worse, and finally split-conformal calibration with an isotonic correction. I ran the official runner locally with one sealed configuration, five folds on all datasets, merged my outputs with the published ones, and recomputed every statistic over the whole pool with the board's own ranking procedure. I also found that the rival TabICL had been run weakened by the benchmark wrapper (8 estimators instead of 32, 200 quantiles instead of 999), re-ran it fairly and confirmed that the comparison did not change.What made the differenceThe model has the lowest median CRPS and the best mean rank on the merged board, with one sealed configuration and no per-dataset tuning.What stayed openInterval coverage was the weak metric before calibration. Mean absolute error and RMSE were not the target of this head.Protocol[arXiv:2603.29928](https://arxiv.org/abs/2603.29928)18 July 2026
## Metadata independence
Prediction without column namesWhat it measuresTwenty numerical classification datasets under three naming conditions: full names, random strings, and names removed. The metric of record is accuracy with no names and the drop from full names. Models that read headers lose roughly seven AUC points when headers vanish.How I built itThe design choice came first: column embeddings are always zeroed, so the model never uses a column's identity, only its values and its relation to other columns. The rest was measurement discipline. I fixed two measurement bugs, then made the protocol symmetric: rivals run on the same folds with five seeds, 300 measurements in all, one fixed configuration for me, no source selection. A win that sat inside its own seed noise was dropped.What made the differenceIt is the only model in the comparison whose accuracy does not fall when names are removed or replaced with noise. In practice, no renaming, mapping or data-dictionary work is needed before the model is useful.What stayed openMy wins are concentrated on datasets where meaning matters. On pure numeric classics without semantics a stronger prior-based rival still leads, usually by a few thousandths.ProtocolThe protocol of the first model's paper, [arXiv:2605.06290](https://arxiv.org/abs/2605.06290)18 to 19 July 2026
## Blind sector identification
No published competitorWhat it measuresName the industry sector of an unseen dataset from raw cell values. 504 real datasets, column names zeroed, no labels, no metadata, scored top-1 and top-5 against a uniform random baseline. Ground-truth labels come from independent judges, never from the model itself.How I built itThe first insight was about signal: a marginal fingerprint of each column could not separate neighbouring sectors such as developer tools and data analytics, but joint pair correlation between columns could. I added a hierarchical sector head with coarse families, then trained a delta on the last blocks of the body using 1,401 tables that are excluded from the test set. I tried eighteen other ideas, and two helped. Test-time augmentation, a favourite from vision, destroyed the signal in tables. I then confirmed the ceiling four ways: the eighteen failed attempts, the structure of the signal, a three-judge audit of the labels that corrected three of them, and the literature on semantic type detection. I froze the test set.What made the differenceA new capability with no earlier tabular model that performs it. After a colleague objected that a perfect score on an easy subset proves nothing, I made the headline the full 504 datasets and kept the clean subset as a sanity check only.What stayed openThe remaining errors are sibling-sector confusions such as banking and insurance, and 24 datasets are ambiguous even for human labellers, so they are reported separately.29 July to 2 August 2026
## Missing-value imputation
Numeric and categoricalWhat it measuresNumeric imputation hides cells and measures reconstruction error (NRMSE) on 20 real datasets under three missingness mechanisms: completely at random, at random, and not at random. Categorical imputation measures macro-F1 on masked categorical cells and has no published competitor.How I built itThis was the longest hunt, and most of it was fixing my own measurement. I found fifteen errors in the harness and the model chain. The imputer was given only a fifth of each table (the test fold), so I let it see train and test together, with both masked. The semantic channel had been fed placeholder column names. The nearest-neighbour arm read real values on its own validation cells, which let it cheat. I also replaced argmax with the expected value for categories, because NRMSE measures squared error and the mean is optimal for it. The final design is a chain of three arms (ridge, in-context, and neighbours), chosen per column on masked holdouts. When only 30 holdout cells decide among 77 candidates, the winner is partly luck, so I set fixed proportional margins against the plain ridge arm.What made the differenceFirst place on the real-data board, ahead of language models and the classical imputers in every setting, with the widest margin under not-at-random missingness. A plain ridge chain matched MICE exactly, which gave me a verified floor.What stayed openOn two synthetic settings the best published MICE figure stayed ahead of my own runs, and I left those rows as reported. On a few datasets with few columns or few rows the chain has little context and the gain is small.Protocol[arXiv:2603.22332](https://arxiv.org/abs/2603.22332)31 July to 5 August 2026
## Entity and schema matching
Keyless and name-blindWhat it measuresEntity matching: six multi-table datasets, find which rows describe the same real-world entity with no keys and no shared identifiers. Schema matching: the Valentine scenarios, which column in one table corresponds to which in another, scored as F1. Every published competitor sees column names. My model does not.How I built itFor entity matching, the mistake was over-training: the Music runs trained for 8,000 steps, which hurt, while 1,200 steps with a two-model ensemble and a calibrated threshold fixed it and cut the run-to-run variance. For schema matching the matcher combines adapter embeddings of columns and cells, a channel of 28 column statistics, and a name channel, scores pairs, normalizes them, and keeps only mutual best one-to-one matches. Scoring is micro-averaged. The macro average gave a different answer, so the averaging rule must be fixed before scoring. When the Valentine data was missing I rebuilt it from its public archive. A checkpoint that was not preserved at the time of sealing taught me to copy every sealed adapter at the moment of sealing.What made the differenceThe model matches records with no keys, beats the published state of the art on the entity benchmarks, and beats every name-aware matcher on all four schema scenarios while running name-blind. The margin is widest where text matching gives out: records that describe the same entity in different words.What stayed openOne published competitor was trained on the benchmark data, which is a leakage caveat for them, not for me.Protocol[arXiv:2604.21238](https://arxiv.org/abs/2604.21238) and [arXiv:2507.10897](https://arxiv.org/abs/2507.10897)1 to 2 August 2026
## Missing-data robustness
Prediction under injected missingnessWhat it measuresTen datasets, masks of three types and several rates, with AUC measured after injecting missing values into the features. It tests whether a model's predictions survive gaps.How I built itMy first move was suspicion of the ruler. A plain XGBoost scored about twelve points lower on my harness than in the published table, so I built an aligned protocol and verified the masker with drift gates: six shift checks and four untouched validations. A long string of failed MAR and MNAR runs turned out to be a shell quirk that swallowed a less-than sign in a pasted comparison, not a model problem. I closed two datasets first (eight of eight settings ahead), then ran the full grid, 750 of 750 cells, with the arbitration choosing the imputation family per dataset.What made the differenceThe mean AUC is above the published method. It is a narrow lead, and I report it as one.What stayed openA further churn-style lever ladder gave nothing, so I stopped tuning.Protocol[arXiv:2507.08280](https://arxiv.org/abs/2507.08280)4 to 7 August 2026
## One checkpoint, then compression
Packaging the whole resultWhat it measuresA benchmark result counts only if a single model can reproduce all of it. This step checks that all of the capabilities live in one object, and that compressing it does not change them.How I built itI merged the core and every head and adapter into one unified checkpoint of about 2.18 billion parameters, with the code embedded, so one line of bootstrap gives access to every capability, then tested the merged model in isolation. I then dumped the model at half, 8-bit and 4-bit precision and measured fidelity with TurboQuant. The model card was published on 7 August, after a contamination audit of the evaluation sets against the training corpus using an identifier blocklist and a distributional near-duplicate check.What made the differenceThe numbers come from one sealed configuration per benchmark, run on the same model that serves requests.
## What makes the difference

- **It reads raw tables.** No shared keys, no labels, no preprocessing, and no column names needed. Every benchmark above was run under those conditions.- **Column identity is never used.** Zeroed column embeddings make the model robust to cryptic, missing or misleading headers, and make it a general reader of tables rather than a reader of names.- **One checkpoint, many tasks.** Classification, regression, distributions, imputation, anomaly detection, matching, sector identification and continual learning share one core.- **One sealed configuration.** Rivals get hours of tuning per dataset. The model gets a single default configuration.- **The ruler is checked first.** In several benchmarks the largest gains came from fixing the harness, not the model, and I verified every comparison on the same masks, folds and averaging rules before I trusted it.- **Weak spots are said aloud.** Regression and multiclass boards, interval coverage before calibration, synthetic imputation settings and narrow leads are all written next to the results.