# Missingness shift: robustness when test data go missing differently (MIRRAMS)

Models are usually tested with the same missingness they were trained with. MIRRAMS asks what happens when that changes. Train at one rate, sweep the test-time rate, and see which strategies hold up.

<div data-sim="tfm" data-v="shift"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

Each line is a model's accuracy on the same 400 unseen rows as more and more of their cells are hidden at test time; the vertical line marks the training rate. The panel shows how many accuracy points each model loses from 0% to 70% missing.

## Things to try

- Train on clean data (0%) and run: which strategy loses the fewest points?
- Train at 40% and run again.
- Switch the test mechanism to MNAR.

## How it works in real systems

Practical defences are models that handle missing values natively (gradient-boosted trees, tabular foundation models), missing-value indicators, and random masking during training; MIRRAMS-style evaluation reports performance across shifted missingness rather than at one rate.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Train at one missing rate, test at many

The same synthetic table as the simulation, so the numbers are comparable.

```python
import numpy as np
rng = np.random.default_rng(7)
def churn(n):
    t = rng.integers(1, 73, n); f = rng.integers(20, 121, n); c = np.minimum(9, rng.exponential(2.2, n).astype(int))
    a = (rng.random(n) < 0.4).astype(int); u = rng.integers(1, 81, n)
    z = 1.1 - 0.05 * t + 0.025 * (f - 60) + 0.45 * c - 1.6 * a - 0.015 * u + 1.1 * ((c > 5) & (a == 0))
    y = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
    return np.c_[t, f, c, a, u].astype(float), y
X, y = churn(60); X_test, y_test = churn(300)
```

```python
X_tr, y_tr = churn(2000); X_te, y_te = churn(2000)
def hide(X, p, seed): r = np.random.default_rng(seed); X = X.copy(); X[r.random(X.shape) < p] = np.nan; return X
```

### Step 2 · Four strategies

```python
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier
p_train = 0.0
Xtr = hide(X_tr, p_train, 1)
aug = np.vstack([hide(X_tr, p, 10 + i) for i, p in enumerate(np.linspace(0, .7, 8))]); y_aug = np.tile(y_tr, 8)
models = {
  "mean imputation": make_pipeline(SimpleImputer(), StandardScaler(), LogisticRegression(max_iter=1000)).fit(Xtr, y_tr),
  "mean + indicators": make_pipeline(SimpleImputer(add_indicator=True), StandardScaler(), LogisticRegression(max_iter=1000)).fit(Xtr, y_tr),
  "mask augmentation": make_pipeline(SimpleImputer(add_indicator=True), StandardScaler(), LogisticRegression(max_iter=1000)).fit(aug, y_aug),
  "boosting, native NaN": HistGradientBoostingClassifier().fit(aug, y_aug),
}
```

### Step 3 · Sweep the test-time rate

```python
for p in (0, .2, .4, .6):
    Xt = hide(X_te, p, 99)
    print(f"{int(p*100):2d}% missing", {k: round(m.score(Xt, y_te), 3) for k, m in models.items()})
```

Add a tabular foundation model (TabPFN accepts NaN) to the dictionary to compare; with `p_train = 0` the mean-imputation model degrades fastest, which is the failure MIRRAMS-style evaluation is designed to expose.

<div class="cl-end"></div>

## More simulations

[← MIDAS](/ai-simulations/midas-imputation) · [All AI Simulations](/ai-simulations) · [Large tables →](/ai-simulations/tfm-large-tables)
