# How a tabular foundation model took first place in probabilistic regression

A tabular foundation model is pre-trained once, mostly on synthetic tables, and at prediction time reads your training rows as context instead of being trained from scratch on each dataset. I led the development of one, and this year I worked on making it good at probabilistic regression: predicting a whole distribution of outcomes instead of one number.

We measured it on an independent benchmark, ScoringBench (Landsgesell, Knoll, Wenzel, [arXiv:2603.29928](https://arxiv.org/abs/2603.29928)): 102 datasets, 5-fold cross-validation, 50 competing models, and the benchmark's unmodified metric code. It ended first of 51 on both headline scoring rules. This article is about how: what a foundation model contributes, which changes moved the numbers, which ideas failed, which mistakes nearly fooled us, and what the win does not mean.

1st of 51CRPS: median, mean rank, trimmed mean1st of 51CRLS: median, mean rank, trimmed mean88 / 81datasets won of 102 (CRPS / CRLS)2×lower median CRLS than the strongest rival

## Step 0: what the foundation model brings, and what it does not

Three ingredients come from the foundation-model side.

- **A distribution-valued head, trained on the score itself.** Instead of predicting a mean, the head outputs 999 quantiles and is trained directly with the CRPS. It was pre-trained for 600K steps on synthetic tables generated from causal structures, including tree-ensemble and heavy-tail priors, then trained further on 100 real datasets with a penalty that keeps the weights close to the pre-trained ones.
- **In-context use with light fine-tuning.** On each new dataset the model reads the training rows as context and is fine-tuned briefly (600 steps) on the training portion only.
- **A backbone that is mostly not needed here.** The foundation model's large backbone helped only on text- and category-heavy datasets (roughly 19 to 30 of 157 folds in our tests). The probabilistic-regression score comes mostly from the small head and from how its output is selected, which is the subject of the rest of this article.

## Step 1: know exactly what the metric rewards

Two scoring rules do most of the work. **CRPS** measures how far the predicted distribution is from the outcome across the whole range, so it rewards a distribution that is both centred correctly and not wider than it needs to be. **CRLS** is a logarithmic cousin that punishes probability placed in the wrong region much more harshly.

A weather forecast is a good everyday picture: "rain tomorrow, 90% sure" scores very well when it rains and very badly when it does not, while "somewhere between dry and a storm" is never badly wrong and never impressive.

The first real progress came from reading the benchmark's metric code, not from modelling. CRLS pays for a distribution that is sharp and crosses the true value at the right place, and it penalises mass spread between and outside the right bins. That told us which knobs could matter and which could not.

## Step 2: the levers, in order of impact

LeverWhat it didMeasured effect
Allow much sharper distributions: lower bound of the spread-search from 0.5 to 0.15The single biggest lever. The selection step was clamped so it could never shrink a distribution below half its width; many tables wanted far less.isolet CRLS 4.11 → 0.96
Put CRLS into the selection objective (weight 2, normalised)Choosing the spread by CRPS alone left CRLS on the table. A combined objective served both rules.abalone CRLS 3.34 → 1.64
Calibrate on a slice the model never fittedEarly calibration reused data the model had already seen and looked excellent for the wrong reason. A held-out slice fixed it.90% coverage 0.237 → 0.889; CRPS −31% (sanity set)
Match the benchmark's fold protocol (2 → 5 folds)Not a model change. Our first run used half the training data per fold; competitors used 5-fold.median CRLS 0.560 → 0.496; mean rank 5.42 → 3.73

Levers that moved the numbers (development measurements on probe sets, except the protocol row)isolet CRLSbefore: 4.11after: 0.964.11 → 0.96abalone CRLSbefore: 3.34after: 1.643.34 → 1.64median CRLS (2 to 5 folds)before: 0.56after: 0.4960.56 → 0.496mean rank (2 to 5 folds)before: 5.42after: 3.735.42 → 3.73Levers that moved the numbers (development measurements on probe sets, except the protocol row). Grey is before, blue is after.
Calibrating on a slice the model never fitted: 90% interval coverage (ideal 0.90)90% coveragebefore: 0.237after: 0.8890.237 → 0.889Calibrating on a slice the model never fitted: 90% interval coverage (ideal 0.90). Grey is before, blue is after.

## Step 3: what did not work

We closed three tempting knobs by testing them on a probe set rather than arguing about them.

- **Smoothing the distribution** did nothing (probe-set CRLS 3.45 → 3.49).
- **Adding an extra tail margin** made it worse (3.45 → 4.79).
- **Sharpening the distribution aggressively** drove CRPS to a degenerate value near zero while CRLS stayed poor (3.78), which is a warning sign about the metric rather than a result.

Each of these would have looked reasonable on paper.

## Step 4: the brakes that kept it honest

Aggressive sharpening is dangerous, so every sharpening step got a brake.

- **Overfit brake.** A step that re-centres predictions on very close training neighbours produced a validation gain of +80% on one dataset and a loss of +249% on the test fold. The rule now is: if the validation gain looks too good (above 25%), revert.
- **Large-scale gate.** On datasets whose targets are in the thousands, the log-based term misbehaved. A gate that switches it off above a target-scale threshold fixed one dataset's systematic +6.8% degradation; together with the brake, another dataset's CRLS fell 92.9% (29,381 → 2,091).

One configuration, no per-dataset tuning, no separate mode for the nine largest datasets. If you cannot run the same recipe everywhere, the result is a collection of tuned runs, not a method.

## Step 5: evaluation discipline, or how we nearly fooled ourselves

- **A phantom regression.** A "baseline is broken" alarm cost days. We were comparing a fold-averaged score (11.38) with a single-fold score (14.x). Always compare fold means with fold means.
- **Running extras without the sealed recipe.** We ran nine extra large datasets without the final settings, got a poor rank (7.23) and concluded we were second. Re-run with the sealed settings, the same board gave 5.42, first. Every additional run must use the same configuration or the comparison is meaningless.
- **Silent failed folds.** 13 folds on 8 datasets had failed under GPU contention and were not caught by a row-count check. Check result files for NaN, not for length.
- **A test ladder.** Nothing went to the full benchmark until it passed a ladder: 1 dataset, then 4, then 12, then 16, with a rule that no previously helped dataset may get worse.

## Step 6: the final board

Place is given by median, mean rank, and trimmed mean.

CRPS boardModelMedianMean rankTrimmedPlace
Ours0.04801.512.511/1/1
nori 30m0.21203.4211.822/2/2
nori0.24534.1812.503/3/3
tabpfn v30.379211.0215.604/4/27
finetune tabpfn realv2 5 crls0.385712.7714.725/5/6
finetune tabiclv20.401813.7615.0527/6/25

CRLS boardModelMedianMean rankTrimmedPlace
Ours0.49623.7316.381/1/1
forest diffusion flow0.94515.9137.562/2/2
finetune tabiclv21.298210.6348.4731/3/4
catboost quantile1.155610.8548.923/4/5
tabiclv21.328812.7950.7632/5/12
tabpfn v31.265415.3652.1420/6/22

CRPS board: median score, lower is better (first of 51)oursours: 0.0480.048nori 30mnori 30m: 0.2120.212norinori: 0.24530.2453tabpfn v3tabpfn v3: 0.37920.3792finetune tabpfn realv2finetune tabpfn realv2: 0.38570.3857finetune tabiclv2finetune tabiclv2: 0.40180.4018CRPS board: median score, lower is better (first of 51)
CRLS board: median score, lower is better (first of 51)oursours: 0.49620.4962forest diffusion flowforest diffusion flow: 0.94510.9451catboost quantilecatboost quantile: 1.1561.156tabpfn v3tabpfn v3: 1.2651.265finetune tabiclv2finetune tabiclv2: 1.2981.298tabiclv2tabiclv2: 1.3291.329CRLS board: median score, lower is better (first of 51)
Median CRPS is 4.4× lower than the runner-up and median CRLS 1.9× lower. Against each of the four strongest rivals, the paired Wilcoxon test over 102 datasets gives p ≤ 7 × 10−14, with 91 to 101 wins out of 102 on CRPS and 92 to 97 on CRLS.

Figure 1. Per-dataset score ratio to the best of the 50 other models (log10, negative = we win).

## The price

A first place on two scoring rules is not a first place at forecasting. On other metrics from the same harness and the same predictions the result is poor, and it is the direct cost of the sharpening levers above.

OursTypical rivalPlace of 51
90% interval coverage (ideal 0.90)0.4640.9051st on coverage error
PIT-KS statistic (calibration)0.3040.0551st
90% interval score8.813.00 (best)51st
MAE / R20.628 / 0.6670.55 / 0.82 (best)38th / 42nd
Time per fold (median)102 s0.3–0.5 s–

The price: where the same predictions rank on other metrics90% interval coverage90% interval coverage: 0.464 (ideal 0.90)ideal / best: 0.900.464 (ideal 0.90)PIT-KS calibration statisticPIT-KS calibration statistic: 0.304 (typical 0.05)ideal / best: 0.050.304 (typical 0.05)90% interval score90% interval score: 8.81 (best 3.00)ideal / best: 3.008.81 (best 3.00)MAEMAE: 0.628 (best 0.55)ideal / best: 0.550.628 (best 0.55)The price: where the same predictions rank on other metrics. Orange is ours, the blue line marks the ideal or best value.
In plain terms the model is confident and mostly well placed, but its intervals are too narrow, so you should not use it where valid prediction intervals matter. I also noticed that its unweighted CRPS is first while the centre- and tail-weighted CRPS variants rank 44th to 48th. Those quantities should move together for an exact distribution, so I read the CRPS lead as "measured by the benchmark's harness" until I have recomputed it on the exact distribution.

## Takeaways

- A foundation model gives you a strong, distribution-valued starting point; the last mile is choosing the output with the metric in mind.
- Read the metric code. The biggest gain came from understanding what the score rewards and removing a clamp that blocked it.
- Put every rule you care about into the objective that selects the final distribution.
- Brake every aggressive step with a rule that reverts it when the gain looks too good.
- Calibrate on data the model never touched, match the benchmark's protocol exactly, and check result files for NaNs.
- Publish the metrics where you are weak next to the ones where you win.

See also the [probabilistic regression benchmark story](https://www.ozercemkelahmet.com/model/bench-probabilistic-regression), [What is a tabular foundation model?](https://www.ozercemkelahmet.com/articles/what-is-a-tabular-foundation-model), and [A score is only as good as its report](https://www.ozercemkelahmet.com/articles/a-score-is-only-as-good-as-its-report).

## Frequently asked questions

Which benchmark was used?
ScoringBench by Landsgesell, Knoll, and Wenzel (arXiv:2603.29928): 102 datasets, 5-fold cross-validation, 50 competing models, with the benchmark's unmodified metric code.

What do CRPS and CRLS measure?
CRPS measures how far the whole predicted distribution is from the outcome. CRLS is a logarithmic score that punishes probability placed in the wrong region much more harshly.

Does first place mean the model forecasts best?
No. On interval coverage, calibration, and interval score the same predictions rank last, so it should not be used where valid prediction intervals matter.