Probabilistic regression benchmark: predicting a whole distribution, scored by CRPS and CRLS

The probabilistic regression benchmark asks a model to predict a whole distribution for a number, not one guess.

This page explains how that is scored, how I built the capability, what made the difference, and what stayed open.

The short version is this.

The model has the lowest median CRPS and the best mean rank on the merged board, with one sealed configuration and no per-dataset tuning.

The board carries 102 regression datasets and the published outputs of 50 models.

Interval coverage was the weak metric before calibration.

There are no scores on this page on purpose.

The model card carries them with their protocols.

For related material, see Row-level prediction, Evaluation and the full Benchmarks timeline.

The benchmark paper is arXiv:2603.29928.

What this benchmark measures

A normal regression model says: the price will be 300.

A probabilistic model says: the price is most likely near 300, it could be anywhere from 250 to 400, and here is how likely each value is.

That second answer is a distribution.

It carries the uncertainty.

The benchmark

ScoringBench scores predicted distributions with proper scoring rules.

The board carries 102 regression datasets and the published outputs of 50 models.

What is a proper scoring rule

A scoring rule gives a number to a forecast once the truth is known.

A rule is proper if the forecaster gets the best expected score by reporting their honest belief.

A proper rule cannot be gamed by hedging or by exaggerating.

Think of a weather forecaster who says 70 percent chance of rain.

If the rule is proper, the forecaster scores best on average by saying exactly what they believe.

Saying 100 percent to look bold, or 50 percent to play safe, would score worse in the long run.

CRPS

CRPS is the continuous ranked probability score.

It compares the whole predicted distribution with the one value that actually happened.

Here is the picture.

Draw the predicted cumulative distribution as a curve that rises from 0 to 1.

The real outcome is a step: 0 before the true value, 1 after.

CRPS is the area between the curve and the step.

A smaller area is better.

CRPS rewards two things at once.

• Sharpness. A narrow distribution is better, if it is right.
• Calibration. The distribution must actually contain the truth at the stated rates.

If the model is sharp but wrong, CRPS punishes it.

If the model is right but vague, CRPS punishes it too.

CRPS has a pleasant property.

When the forecast is a single number, CRPS reduces to the absolute error.

So it is a generalization of mean absolute error to distributions.

CRLS

CRLS is the continuous ranked logarithmic score.

It is the log-score cousin.

It looks at how much probability the predicted distribution puts on the value that happened, and takes a logarithm.

The log score has a harsh edge.

If the distribution puts almost no probability on the truth, the log of that tiny number is a large negative value, so the penalty is large.

A distribution that is too narrow gets hit hard.

I come back to this below, because it was the key to the second half of the work.

Why two metrics

CRPS and CRLS reward different things.

CRPS is forgiving about the tails.

CRLS is strict about them.

A method that wins on both is robust to how you choose to judge it.

Interval coverage

Coverage is a check on calibration.

If a model says its 90 percent interval contains the truth, then over many rows about 90 percent of the true values should fall inside.

If the coverage is much lower than the stated level, the model is overconfident.

If it is much higher, the model is too cautious.

See Evaluation for how quantile calibration is read.

Ranking on the board

Methods are compared with the board's own ranking procedure.

For each dataset the methods are ranked by score.

Then the ranks are summarized as a mean rank, and the scores are summarized as a median.

The model has the lowest median CRPS and the best mean rank on the merged board.

Who the model is compared with

The board holds the published outputs of 50 models.

These are third-party values.

I never re-run a competitor, with one narrow exception described below, where I ran a rival fairly because the benchmark wrapper had weakened it.

The field includes tabular foundation models and classical probabilistic regressors.

The model on this page is one more entry in that pool.

How I built it

Train for the rule you are scored on

I trained a 999-quantile head with a CRPS loss.

This followed the benchmark paper's own finding that you win on the scoring rule you train for.

A quantile head predicts many points of the distribution.

The 50th quantile is the median.

The 10th quantile is the value below which 10 percent of the outcomes fall.

With 999 quantiles, the model describes the distribution in fine steps, from the 0.1 percent point to the 99.9 percent point.

A CRPS loss trains the head directly on the metric.

There is a neat link: CRPS equals the average of the pinball loss over all quantile levels, doubled.

So a head with many quantiles trained on pinball loss is trained on CRPS.

That gave first place on CRPS.

See Probability for background on distributions and quantiles.

CRLS lagged

CRLS lagged.

The model was first on one metric and not on the other.

That pointed to something specific, not a general weakness.

The diagnosis

The diagnosis was precise. The target was standardized by its standard deviation.

Standardizing means subtracting the mean and dividing by the standard deviation, so the target has a typical size of about one.

It helps neural networks learn.

But for heavy-tailed targets, such as house prices, runtimes and car prices, the standard deviation is dominated by a few huge values.

After dividing by it, the bulk of the data is squeezed into a tiny range.

The predicted distribution came out too narrow for that bulk.

And CRLS punishes a narrow distribution hard.

When the truth falls outside a too-narrow distribution, the log penalty is severe.

An everyday picture: a tailor who cuts all suits from one measure taken from the tallest customer.

The suit fits no one in the shop, but the tailor's average looks fine until you look at each customer.

Fix 1: a log transform for heavy tails

For heavy-tailed targets, I used a log transform.

Taking the logarithm pulls in the long tail, so the data are more even in shape.

The model predicts in log space and the result is mapped back.

Prices, durations and counts often grow by multiples, not by additions.

A log scale matches that.

A difference between 100 and 200 is similar in spirit to a difference between 1,000 and 2,000.

Fix 2: a CRLS fine-tune, later dropped

I tried a CRLS fine-tune. I trained the head further against the log score directly.

I later dropped it.

Both metrics got worse.

It was a case where chasing one metric hurt both.

I record this because the temptation to train on the metric that lags is strong, and here it did not work.

Fix 3: split-conformal calibration with an isotonic correction

The fix that stayed is split-conformal calibration with an isotonic correction.

Conformal prediction is a method for making intervals that cover the truth at a stated rate, with few assumptions.

The split version works like this.

1. Hold out a part of the data as a calibration set.
2. Predict those rows and measure how far off the model was.
3. Use those errors to adjust the intervals so they cover at the right rate.

It is like checking a thermometer against a reference in the same room.

You do not rebuild the thermometer.

You learn how much it reads high or low and correct for it.

The isotonic correction adds a monotone fix.

Isotonic regression fits a curve that only goes up.

Here it maps the stated quantile levels to the levels that are actually observed.

If the model's 90 percent point really covers 80 percent of outcomes, the map says: to get 90 percent, ask for the 96 percent point.

The map is monotone, so quantiles never cross.

Calibration repaired the weak interval coverage and the CRLS gap together.

The harness: five folds, one configuration

Running the official runner

I ran the official runner locally with one sealed configuration, five folds on all datasets.

Running the official code removes a whole class of doubt about whether my scoring matches theirs.

Merging my outputs with the published ones

I merged my outputs with the published ones.

The result is one pool in which my model sits beside the 50 published models.

Recomputing every statistic over the whole pool

I recomputed every statistic over the whole pool with the board's own ranking procedure.

This matters.

If I had computed my rank on my own, and compared it with their stored ranks, the numbers would not be on the same footing.

Adding a new entry changes everyone's rank.

So the statistics must be recomputed together.

The rival that was run weakened

I also found that the rival TabICL had been run weakened by the benchmark wrapper.

The wrapper used 8 estimators instead of 32, and 200 quantiles instead of 999.

An estimator here is one member of an ensemble.

Fewer members means a worse and noisier forecast.

Fewer quantiles means a coarser description of the distribution.

Both hurt a rival that is meant to be run with the larger settings.

I re-ran it fairly, with 32 estimators and 999 quantiles, and confirmed that the comparison did not change.

This is the fair thing to do, and it is also the safe thing to do.

A win over a handicapped rival is not worth much.

A win that holds against the rival at full strength is worth having.

I report both the finding and the result of the check.

This is the one place where I ran a competitor myself.

The reason is that the published output had been produced with a wrapper setting that did not match the rival's own recommendation.

Everywhere else, competitor values are third-party published values.

What made the difference

Train on the rule

The 999-quantile head with a CRPS loss won CRPS.

Matching the loss to the metric is the simplest big lever.

Understand why the second metric lagged

The CRLS gap was not mysterious.

The target was standardized by its standard deviation, and that made distributions too narrow for heavy-tailed targets.

Once the cause was named, the fixes were natural.

Calibrate

Split-conformal calibration with an isotonic correction repaired coverage and CRLS.

It also protects against the narrowness problem in general, not only for the datasets I looked at.

Check the ruler

The board's own procedure, run locally, one sealed configuration, five folds on all datasets, a fair re-run of a weakened rival.

The ruler was verified before I trusted the result.

One sealed configuration

The model has the lowest median CRPS and the best mean rank on the merged board, with one sealed configuration and no per-dataset tuning.

How the model gets to a distribution

For the general design, see Architecture and In-context prediction.

Here is the short version.

1. The table is encoded into cell vectors, see Cell encoding.
2. Axial attention mixes information within rows and columns, see Axial attention.
3. The labelled rows are the context. The model compares a new row with them.
4. The regression head outputs 999 quantiles.
5. After calibration, the quantiles are adjusted so that coverage is honest.

In the product, regression can return quantiles, so you can read a range rather than a single number.

See Capabilities.

The protocol

Third-party competitor values

Competitor figures are never re-run by me, apart from the one fair re-run described above.

They are third-party published values cited to their primary source.

One sealed configuration

My figures come from one sealed configuration per benchmark, with no per-dataset tuning.

For this benchmark: five folds on all 102 datasets, with the same settings everywhere.

Snapshot date

For live boards the snapshot date is pinned, because a board changes as new entries arrive.

A rating is a statement about a field at a given date.

No blending

Results measured under different datasets, metrics or masking schemes are never blended into one number.

CRPS and CRLS are reported as separate metrics.

Contamination

Each run is checked for contamination against the training corpus.

Rules at a glance

Lessons from the build

A metric that lags has a cause

CRLS lagged for a specific reason: narrow distributions on heavy-tailed targets.

Look for the cause before changing the training.

Do not chase the lagging metric blindly

The CRLS fine-tune made both metrics worse. I dropped it.

Calibration is cheap and strong

A held-out calibration set and a monotone map fixed what a longer training run could not.

Standardizing is not harmless

Dividing by the standard deviation is a default that many pipelines apply without thought.

For heavy tails, it can silently hurt.

Run rivals at full strength

If a rival is weakened by a wrapper, a win over it proves little.

Re-run it fairly and check.

Recompute ranks together

A rank depends on the whole field. Recompute everything over the merged pool.

What stayed open

Interval coverage before calibration

Interval coverage was the weak metric before calibration.

Calibration repaired it, but it is a repair on top of the trained head, not a property of the head itself.

If you use the raw head without the calibration step, expect intervals that are too narrow on heavy-tailed targets.

Mean absolute error and RMSE

Mean absolute error and RMSE were not the target of this head.

The head was trained for CRPS and calibrated for coverage.

If your use case is the best single-number guess, evaluate on your own metric.

See Row-level prediction, where regression is behind the leaders on the Elo boards.

A calibration set is needed

Split-conformal calibration needs rows set aside for calibration.

The guarantee is marginal

Conformal calibration gives coverage on average across rows.

It does not promise the same coverage inside every subgroup.

A rare region of the table can still be under-covered.

Out-of-distribution rows

Calibration assumes the new rows look like the calibration rows.

If the world shifts, the coverage can slip.

See Limitations.

What this means for a user

If you want a range, not a single guess, ask for quantiles.

Read the 10th and 90th quantile as an 80 percent interval.

If your targets are heavy-tailed, such as prices or durations, the log-transform and calibration steps matter.

They are why the intervals are honest.

Check coverage on your own held-out rows.

If your intervals say 90 percent, count how often the truth lands inside.

The result tells you whether to trust the ranges.

Frequently asked questions

What is probabilistic regression?

It is regression that predicts a full distribution for the target, not a single number.

The output tells you the most likely value and how wide the uncertainty is.

It is scored with proper scoring rules such as CRPS and CRLS.

What is CRPS?

CRPS is the continuous ranked probability score.

It measures the area between the predicted cumulative distribution and the step function of the true value.

A smaller area is better.

It rewards distributions that are both sharp and well calibrated.

Why did CRLS lag at first?

The target was standardized by its standard deviation.

For heavy-tailed targets such as house prices, runtimes and car prices, this made the predicted distribution too narrow.

CRLS punishes a narrow distribution hard, so it exposed the problem more than CRPS did.

What fixed it?

A log transform for heavy tails, and then split-conformal calibration with an isotonic correction.

A CRLS fine-tune was tried and dropped because both metrics got worse.

What is a 999-quantile head?

It is a prediction head that outputs 999 points of the distribution, from the very low end to the very high end.

Trained with a CRPS loss, it describes the distribution in fine steps.

The benchmark paper found that you win on the scoring rule you train for.

What is split-conformal calibration?

It is a way to make intervals cover at the stated rate.

You hold out some rows, measure the model's errors on them, and use those errors to adjust the intervals.

The isotonic correction adds a monotone map between stated and observed quantile levels.

How big is the board?

It carries 102 regression datasets and the published outputs of 50 models.

I ran the official runner locally with one sealed configuration, five folds on all datasets, merged my outputs with the published ones, and recomputed every statistic over the whole pool.

Was every rival run at full strength?

I found that TabICL had been run weakened by the benchmark wrapper, with 8 estimators instead of 32 and 200 quantiles instead of 999.

I re-ran it fairly and confirmed the comparison did not change.

All other competitor values are third-party published values.

Does the model also win on mean absolute error?

That was not the target of this head.

The head was trained for CRPS and calibrated for coverage.

On single-number regression the model is behind the leaders on the Elo boards, which is stated openly on the row-level prediction page.

Why are there no scores on this page?

Scores depend on the harness, the fold count and the snapshot date.

The model card carries them with their protocols so each number can be checked.

Summary

The probabilistic regression benchmark scores predicted distributions with CRPS and CRLS across 102 regression datasets and the published outputs of 50 models.

I trained a 999-quantile head with a CRPS loss and won on CRPS.

CRLS lagged because standardizing by the standard deviation made distributions too narrow on heavy-tailed targets.

The fixes were a log transform for heavy tails, a CRLS fine-tune that I dropped, and finally split-conformal calibration with an isotonic correction.

I ran the official runner locally with one sealed configuration and five folds, merged my outputs with the published ones, and recomputed every statistic over the whole pool.

A rival run weakened by the wrapper was re-run fairly and the comparison did not change.

The model has the lowest median CRPS and the best mean rank on the merged board.

What stayed open: interval coverage was weak before calibration, and mean absolute error and RMSE were not the target of this head.

Read next: Row-level prediction, Anomaly detection, Continual learning, and Benchmarks.