Missing-data robustness benchmark: predicting when the features have holes

The missing-data robustness benchmark injects holes into the features of a dataset and asks how well a model still predicts. It tests whether predictions survive gaps. The tabular foundation model described here has a mean AUC above the published method. The lead is narrow, and I report it as one.

This page explains the task, the three kinds of masks, why I distrusted the ruler first, how the protocol was aligned and verified, how a string of failed runs turned out to be a shell quirk, what made the difference, and what stayed open. Scores are left out on purpose. The model card carries them with their protocols. For all boards, see Benchmarks.

The short version

• Ten datasets, masks of three types and several rates, with AUC measured after injecting missing values into the features.
• My first move was suspicion of the ruler: a plain XGBoost scored about twelve points lower on my harness than in the published table.
• I built an aligned protocol and verified the masker with drift gates: six shift checks and four untouched validations.
• A long string of failed MAR and MNAR runs turned out to be a shell quirk that swallowed a less-than sign in a pasted comparison, not a model problem.
• I closed two datasets first, eight of eight settings ahead, then ran the full grid, 750 of 750 cells, with the arbitration choosing the imputation family per dataset.
• The mean AUC is above the published method. It is a narrow lead.
• A further churn-style lever ladder gave nothing, so I stopped tuning.

What the benchmark measures

You have a trained predictor and a clean test set. Now you remove some values from the features of the test rows and ask the predictor to answer anyway. A model that handles gaps well loses little. A model that depends on complete rows loses a lot.

Why this differs from imputation

The imputation benchmark scores how close a filled cell is to the true value. This benchmark scores something else: whether the final prediction is still good. A method could fill gaps poorly and still predict well, or fill them nicely and predict badly. What matters in practice is the decision at the end, so both are worth measuring.

AUC in plain words

AUC is the chance that the model ranks a randomly chosen positive row above a randomly chosen negative row. A value of 0.5 is a coin flip. A value of 1.0 is perfect ranking. It does not depend on where you draw the decision line, which makes it a steady yardstick for comparing models.

The three mask types

The pattern of missingness matters as much as the rate. The three standard mechanisms are described in Pre-training and Imputation.

Rates

Each mask is applied at several rates, from a few missing cells to a large share. A robust model degrades gently as the rate rises. A fragile one falls off a cliff.

The grid

Crossing ten datasets with three masks and several rates gives a grid of cells. The full grid has 750 cells, and each cell is one setting of the grid.

Real tables have holes

A model that only works on complete data is a model for a laboratory. In production, features go missing for dull reasons: a form field skipped, a sensor offline, a join that found nothing. Pipelines often mean-fill and hope.

The model treats a missing cell as an input, not an error. It is encoded explicitly, as its own kind of input, and the model has seen a great many tables with holes in them during pre-training. See Cell encoding and Pre-training.

• Missing cells are inputs.
• The pattern of what is missing can be signal.
• The same machinery reads present and absent cells.

My first move: suspicion of the ruler

Before I measured the model, I measured a plain XGBoost on my harness. It scored about twelve points lower than in the published table. See XGBoost for the method.

A twelve-point gap for a standard, well-understood baseline is not a finding about XGBoost. It is a finding about the harness. If a reference method cannot reproduce its published number, nothing measured by that harness can be trusted, including the model.

Possible reasons

• Different masks from the published protocol.
• Different rates or different handling of the rates.
• Different folds, or masking applied before the split.
• A different treatment of the missing cells at fit time.

I did not guess. I rebuilt the protocol.

An everyday analogy

If you weigh a known one-kilogram bag of sugar and your scale says 880 grams, you do not conclude the sugar has shrunk. You conclude the scale is off. I fixed the scale first.

Building an aligned protocol

The aligned protocol follows the published one closely enough that a plain baseline reproduces its published behaviour. See the protocol note at the end of the page.

Drift gates for the masker

I verified the masker with drift gates. A drift gate is a test that the masked data still looks like the clean data in the ways it should, and differs in the ways it should.

• Six shift checks. These confirm that masking shifted the data as intended: the right share of cells missing, in the right pattern for each mask type.
• Four untouched validations. These confirm that parts of the data that should not change did not change.

Ten gates in all. A masker that passes them is doing what it claims.

Why a masker needs verification

A masker is a small program, and small programs have bugs. A bug that masks too many cells, or the wrong cells, silently turns an easy test into a hard one or the reverse. Without gates, you cannot tell a model failure from a masker failure.

gate: share of masked cells matches the target rate          -> pass / fail
gate: MCAR mask is independent of every column               -> pass / fail
gate: MAR mask depends only on the observed driver column    -> pass / fail
gate: MNAR mask depends on the hidden value itself           -> pass / fail
gate: labels unchanged                                       -> pass / fail
gate: unmasked cells unchanged                               -> pass / fail

The list above is an illustration of the kinds of checks, in the spirit of the six shift checks and the four untouched validations.

The shell quirk

After the protocol was aligned, I hit a long string of failed MAR and MNAR runs. It looked like the model was breaking down on the harder masks. I spent time looking for a model problem.

It was not a model problem. A shell quirk swallowed a less-than sign in a pasted comparison. A condition that should have said "less than" was silently altered, so the check behaved wrongly and runs failed.

Why I tell this story

• Failures that appear only on certain settings can come from the tooling, not the model.
• A pasted command is code. Treat it with the same suspicion as any other code.
• The right response to a mysterious failure is to check the plumbing before touching the model.

An everyday analogy: a car that will not start on cold mornings may need a new battery, or you may have left the lights on. Check the lights first.

Closing two datasets first

With the ruler fixed, I did not launch the whole grid at once. I closed two datasets first.

For each of the two I ran every setting of mask type and rate. The model was ahead in eight of eight settings on them. Starting small made problems visible cheaply. If something had been wrong, I would have found it on two datasets rather than ten.

The full grid

Then I ran the full grid, 750 of 750 cells. The arbitration chose the imputation family per dataset. In plain words, for each dataset the system picked which family of imputation to apply, instead of using one family everywhere.

This is the same logic as the per-column choice on the imputation board, one level up. See Missing-value imputation.

What arbitration means here

• There are several imputation families to choose from.
• Different datasets favour different families.
• The choice is made by an arbitration step, per dataset, under the same sealed procedure.
• It is not a per-dataset hand tune. It is a rule applied the same way everywhere.

What made the difference

• Distrusting the ruler first. A plain baseline that cannot reproduce its published number is a warning.
• An aligned protocol. Close enough to the published one that baselines behave.
• Drift gates. Six shift checks and four untouched validations on the masker.
• Finding the plumbing fault. The shell quirk was not a model failure.
• Small first, then wide. Two datasets closed, then the full grid.
• Arbitration of the imputation family per dataset. A rule, not a hand tune.

The mean AUC is above the published method. It is a narrow lead, and I report it as one.

Why I call it narrow

A narrow lead is a small gap on average. It does not mean every setting. It does not mean every dataset. It means that across the grid, the average is a little higher. I would rather you read it as a modest edge, because that is what it is.

What stayed open

• The lead is narrow. A small average edge is not a blowout.
• A further lever ladder gave nothing. A churn-style lever ladder is a sequence of tuning steps tried one after another. This one gave nothing, so I stopped tuning.
• Ten datasets is a bounded sample. The claim covers the grid that was measured.
• Harder masks stay hard. Not-at-random missingness is the case where any method has the least to go on.

Why I stopped tuning

Continuing to adjust until something moved would break the standing rule: one sealed configuration per benchmark, with no per-dataset tuning. A lever ladder that gives nothing is evidence that the configuration is already where it should be. Chasing a fraction of a point past that is how a result becomes untrustworthy.

Protocol

• Ten datasets, masks of three types, several rates.
• AUC measured after injecting missing values into the features.
• An aligned protocol checked by reproducing a plain baseline.
• The masker verified with six shift checks and four untouched validations.
• The full grid of 750 cells.
• The imputation family arbitrated per dataset by a rule.
• One sealed configuration, with no per-dataset tuning.
• Competitor figures are third-party published values cited to their primary source, not re-run by me.
• Results under different datasets, metrics or masking schemes are never blended into one number.
• Each run is checked for contamination against the training corpus.
• The protocol follows arXiv:2507.08280.

See Evaluation for what a score claims.

Reading the grid

A grid of 750 cells is easier to read in layers.

What to look for

• A gentle slope as the rate rises, not a cliff.
• Small gaps between mask types on a given dataset.
• A mean that sits above the published method, even if narrowly.
• Any cells where the model is behind, which should be stated.

How it relates to the rest of the model

• Cell encoding. Missing cells are first-class inputs. See Cell encoding.
• Pre-training. The model sees tables with natural and injected holes. See Pre-training.
• In-context prediction. The labelled rows are read in context, gaps included. See In-context prediction.
• Imputation. The imputation family feeds the arbitration. See Missing-value imputation.
• Packaging. All of it ships in one checkpoint. See One checkpoint.

How the output looks

A prediction on a row with gaps looks like any other prediction, with a confidence.

{
  "prediction": {
    "target": "churned",
    "class": "retained",
    "probability": 0.87,
    "missing_features": ["income", "tenure"]
  }
}

The values are invented to show the shape. In use, a row with many missing features should be read with care, and the held-out report shows the weakest slice. See Evaluation and Limitations.

Lessons from this board

• Reproduce a known baseline before measuring anything new.
• Verify the tool that creates the test.
• Treat a pasted command as code.
• Go small before going wide.
• Report a narrow lead as narrow.
• Stop tuning when the ladder gives nothing.

Do and do not

Do:

• Check that a plain baseline matches its published number.
• Use drift gates on any masker.
• Test all three mask types at several rates.
• Run a small subset before the full grid.
• Keep one sealed configuration.

Do not:

• Trust a harness that cannot reproduce a baseline.
• Blame the model before checking the plumbing.
• Tune each dataset by hand.
• Present a narrow lead as a large one.
• Blend results from different masking schemes.

Questions to ask about any robustness claim

• Which mask types and rates were used?
• Did a plain baseline reproduce its published number on the harness?
• How was the masker verified?
• Were the rival figures re-run, or taken from the primary source?
• Is there one configuration, or one per dataset?
• Is the lead stated with its size?
• What was tried and found not to help?

Fact sheet

The result in six lines

• Ruler: suspected first, then aligned.
• Masker: verified with ten gates.
• Failed runs: a shell quirk, not the model.
• Two datasets: eight of eight settings ahead.
• Full grid: 750 of 750 cells.
• Mean AUC: above the published method, narrowly.

Common misreadings

"The model never loses accuracy to gaps"

All predictors lose something as gaps grow. The claim is about the mean across the grid against the published method.

"Twelve points lower means XGBoost is weak"

It means my first harness was unaligned. After the protocol was rebuilt, the baseline behaved.

"Arbitration is hidden tuning"

It is a rule applied the same way everywhere. A rule that selects among imputation families per dataset is part of the sealed procedure.

"The shell quirk is an excuse"

It is a finding about the harness. The failures disappeared when the comparison was fixed, and the model was not touched.

"A narrow lead is not worth reporting"

A narrow lead is still a lead, and it is reported with its size.

The work in order

A walk through one row

The row below is invented, to show what the benchmark asks.

feature      clean   MCAR-masked   MNAR-masked
age          54      54            54
income       71000   (blank)       (blank)
tenure       3       3             (blank)
region       west    west          west
label        1       1             1

1. With the clean row, the predictor has everything.
2. With the MCAR row, income is missing by chance. The other features still point toward the label.
3. With the MNAR row, income is missing because it was high, and tenure is missing too. Here the pattern of blanks is itself a clue.
4. The benchmark measures how well the predictor ranks rows like these across the whole grid, summarized as AUC.

Why the model should cope

• It reads missing cells as inputs, so a blank does not break the pass.
• It has seen many tables with holes during pre-training.
• It reasons about the joint structure of the row, so present cells inform absent ones.
• It predicts in context from labelled rows that may have gaps of their own.

Three habits that kept the benchmark honest

Reproduce before you compare

A plain baseline must reproduce its published behaviour on your harness. If it cannot, stop and fix the harness.

Verify the tool that makes the test

The masker is a program. Test it with gates, as you would test any program whose output you are about to trust.

Report the size of the lead

A narrow lead is reported as narrow. Language such as "wins" or "beats" without a size invites more confidence than the data supports.

Questions a skeptical reader might ask

• Could the model have been tuned to this grid? No: one sealed configuration, no per-dataset tuning.
• Could the masker have favoured the model? The gates check the masker, and the protocol was aligned to the published one.
• Could the shell quirk have hidden a model failure? It affected the comparison, and it was fixed without touching the model.
• Could the rival numbers be old? They are third-party published values, cited to their primary source.
• Could the tables be in the training data? Each run is checked for contamination.

What a reader can take away

• Predictions can survive holes in the features.
• A harness that cannot reproduce a baseline cannot be trusted.
• Failures on only some settings may be plumbing.
• A narrow lead is still a lead, stated with its size.

Where to read next

• Missing-value imputation for filling the gaps directly.
• Row-level prediction for the main prediction boards.
• Pre-training for how the model learns around holes.
• Evaluation for what a score claims.

The standing rules, restated

• Competitor figures are never re-run by me.
• They are cited to their primary source.
• My figures come from one sealed configuration.
• There is no per-dataset tuning.
• Different datasets, metrics or masking schemes are never blended.
• Each run is checked for contamination.

Do and do not, in short

• Do reproduce a baseline first.
• Do gate the masker.
• Do go small before wide.
• Do state the size of a lead.
• Do not blame the model before the plumbing.
• Do not tune per dataset.
• Do not blend masking schemes.

Quick answers

• Does it need complete rows? No.
• Does it treat blanks as errors? No, as inputs.
• Does it always win? The mean is above the published method, narrowly.
• Is the grid complete? Yes, 750 of 750 cells.
• Did you keep tuning? No, the lever ladder gave nothing and I stopped.
• Where are the scores? In the model card, with their protocols.

Mask types, one line each

• MCAR: a coin flip decides.
• MAR: something you can see decides.
• MNAR: the hidden value decides.

What the page did not claim

• It did not claim a large lead.
• It did not claim immunity to gaps.
• It did not claim results outside the ten datasets.
• It did not claim that tuning was unnecessary for rivals; their figures are their published ones.
• It did not claim the shell quirk was a model fault.
• It did not claim certainty for rows with many missing features.
• It did not claim that robustness replaces a held-out report on your own table.

Related reading

• Benchmarks

Terms used on this page

• Robustness. How little a prediction suffers when inputs are degraded.
• AUC. The chance that a random positive row is ranked above a random negative one.
• Mask. The rule that decides which cells are hidden.
• MCAR, MAR, MNAR. Missing completely at random, at random, and not at random.
• Drift gate. A test that masked data looks as it should.
• Arbitration. A rule that chooses among options, here the imputation family per dataset.
• Lever ladder. A sequence of tuning steps tried one after another.
• Cell. In the grid, one setting.

Frequently asked questions

What does the missing-data robustness benchmark test?

It tests whether a model's predictions survive gaps. Missing values are injected into the features of ten datasets with three mask types at several rates, and AUC is measured afterwards.

How did the model do?

The mean AUC is above the published method. The lead is narrow, and I report it as a narrow lead. Scores are in the model card with their protocols.

Why was a plain XGBoost twelve points lower?

Because my first harness was not aligned with the published protocol. A standard baseline that cannot reproduce its published number is a sign the ruler is off, so I rebuilt the protocol and verified the masker.

What are drift gates?

They are checks that the masked data looks as it should. I used six shift checks and four untouched validations to confirm the masker did what it claimed and left the rest alone.

What was the shell quirk?

A long string of failed MAR and MNAR runs turned out to be a shell quirk that swallowed a less-than sign in a pasted comparison. It was a plumbing fault, not a model problem, and the model was not changed to fix it.

What does 750 of 750 cells mean?

I ran the full grid, all 750 cells, where each cell is one setting of the grid. Before that I closed two datasets first, with the model ahead in eight of eight settings on them.

What does arbitration do?

It chooses the imputation family per dataset, by a rule applied the same way everywhere. It is part of the sealed procedure and not a hand tune of each dataset.

Why did you stop tuning?

A further churn-style lever ladder gave nothing. Continuing would risk fitting to the benchmark, which breaks the rule of one sealed configuration with no per-dataset tuning.

Is the missing-data result the same as the imputation result?

No. Imputation scores how close filled cells are to the truth. Robustness scores whether the final prediction survives gaps. They are related, and I measure them separately. See Missing-value imputation.

How should I treat a row with many missing features?

Read the prediction with care and use the confidence. The held-out report shows the weakest slice, which is where the model is least sure. See Limitations.

Summary

Missing-data robustness asks whether predictions survive holes in the features. Across ten datasets, three mask types and several rates, the model's mean AUC is above the published method, narrowly, and I say so.

The work was mostly about the ruler. A plain XGBoost scored about twelve points low on my first harness, so I built an aligned protocol and verified the masker with six shift checks and four untouched validations. A string of failed MAR and MNAR runs turned out to be a shell quirk. I closed two datasets first, eight of eight settings ahead, then ran the full 750-cell grid with the imputation family arbitrated per dataset. A churn-style lever ladder gave nothing, so I stopped tuning. Read next: Missing-value imputation, Architecture, Capabilities and Limitations.