Missingness shift: robustness when test data go missing differently (MIRRAMS)
Models are usually tested with the same missingness they were trained with. MIRRAMS asks what happens when that changes. Train at one rate, sweep the test-time rate, and see which strategies hold up.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
Each line is a model's accuracy on the same 400 unseen rows as more and more of their cells are hidden at test time; the vertical line marks the training rate. The panel shows how many accuracy points each model loses from 0% to 70% missing.
Things to try
- Train on clean data (0%) and run: which strategy loses the fewest points?
- Train at 40% and run again.
- Switch the test mechanism to MNAR.
How it works in real systems
Practical defences are models that handle missing values natively (gradient-boosted trees, tabular foundation models), missing-value indicators, and random masking during training; MIRRAMS-style evaluation reports performance across shifted missingness rather than at one rate.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Train at one missing rate, test at many
The same synthetic table as the simulation, so the numbers are comparable.
import numpy as np
rng = np.random.default_rng(7)
def churn(n):
t = rng.integers(1, 73, n); f = rng.integers(20, 121, n); c = np.minimum(9, rng.exponential(2.2, n).astype(int))
a = (rng.random(n) < 0.4).astype(int); u = rng.integers(1, 81, n)
z = 1.1 - 0.05 * t + 0.025 * (f - 60) + 0.45 * c - 1.6 * a - 0.015 * u + 1.1 * ((c > 5) & (a == 0))
y = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
return np.c_[t, f, c, a, u].astype(float), y
X, y = churn(60); X_test, y_test = churn(300)
X_tr, y_tr = churn(2000); X_te, y_te = churn(2000)
def hide(X, p, seed): r = np.random.default_rng(seed); X = X.copy(); X[r.random(X.shape) < p] = np.nan; return X
Step 2 · Four strategies
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier
p_train = 0.0
Xtr = hide(X_tr, p_train, 1)
aug = np.vstack([hide(X_tr, p, 10 + i) for i, p in enumerate(np.linspace(0, .7, 8))]); y_aug = np.tile(y_tr, 8)
models = {
"mean imputation": make_pipeline(SimpleImputer(), StandardScaler(), LogisticRegression(max_iter=1000)).fit(Xtr, y_tr),
"mean + indicators": make_pipeline(SimpleImputer(add_indicator=True), StandardScaler(), LogisticRegression(max_iter=1000)).fit(Xtr, y_tr),
"mask augmentation": make_pipeline(SimpleImputer(add_indicator=True), StandardScaler(), LogisticRegression(max_iter=1000)).fit(aug, y_aug),
"boosting, native NaN": HistGradientBoostingClassifier().fit(aug, y_aug),
}
Step 3 · Sweep the test-time rate
for p in (0, .2, .4, .6):
Xt = hide(X_te, p, 99)
print(f"{int(p*100):2d}% missing", {k: round(m.score(Xt, y_te), 3) for k, m in models.items()})
Add a tabular foundation model (TabPFN accepts NaN) to the dictionary to compare; with p_train = 0 the mean-imputation model degrades fastest, which is the failure MIRRAMS-style evaluation is designed to expose.