# MIDAS: multiple imputation with a denoising autoencoder

MIDAS learns to rebuild a table from a deliberately corrupted copy, then fills the gaps several times. The spread between the imputations is the uncertainty that a single filled table would hide.

<div data-sim="tfm" data-v="midas"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

Train the autoencoder and watch the reconstruction loss fall. The table shows the mean of the m imputations; the strip shows all m values for the chosen hidden cell with the true value. The panel compares the error with mean imputation and reports the between-imputation variance.

## Things to try

- Train, then click different hidden cells: some are certain, some are not.
- Set m to 30 for a smoother picture of the spread.
- Reset and stop training early: more spread, larger error.

## How it works in real systems

MIDAS (Lall and Robinson, 2022) is available as MIDASpy; its imputations are analysed separately and pooled with Rubin's rules, so standard errors include the imputation uncertainty.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Data with gaps

```python
import numpy as np
rng = np.random.default_rng(11)
def houses(n):
    s = rng.integers(35, 181, n); ro = np.clip(np.round(s / 32 + rng.normal(0, .6, n)), 1, 6); ag = rng.integers(0, 81, n); d = np.round(rng.uniform(.5, 25, n), 1)
    p = 2.4 * s * (1 + .45 * np.exp(-d / 4)) - .9 * ag + 14 * ro + 40
    return np.c_[s, ro, ag, d].astype(float), np.round(p * (1 + rng.normal(0, .07, n)))
X, y = houses(60); X_test, y_test = houses(300)
```

```python
import pandas as pd
df = pd.DataFrame(np.c_[X, y], columns=["size", "rooms", "age", "distance", "price"])
full = df.copy(); mask = rng.random(df.shape) < .2; df = df.mask(mask)
```

### Step 2 · MIDAS with MIDASpy

```python
# pip install MIDASpy  (TensorFlow under the hood)
import MIDASpy as md
imputer = md.Midas(layer_structure=[128, 128], vae_layer=False, seed=89, input_drop=0.75)
imputer.build_model(df)                       # scales and corrupts the inputs internally
imputer.train_model(training_epochs=50)
imputations = imputer.generate_samples(m=10).output_list   # ten complete tables
```

### Step 3 · Use all m tables

Fit the analysis on each imputed table and pool with Rubin's rules: the average estimate, and a variance that adds the between-imputation spread.

```python
import statsmodels.api as sm
est, var = [], []
for t in imputations:
    r = sm.OLS(t["price"], sm.add_constant(t[["size", "age", "distance"]])).fit()
    est.append(r.params); var.append(r.bse ** 2)
est, var = pd.DataFrame(est), pd.DataFrame(var)
m = len(imputations); q = est.mean(); W = var.mean(); B = est.var()
print(pd.DataFrame({"estimate": q, "std error": np.sqrt(W + (1 + 1 / m) * B)}))
```

<div class="cl-end"></div>

## More simulations

[← Masked value](/ai-simulations/tfm-masked-value) · [All AI Simulations](/ai-simulations) · [Missingness shift →](/ai-simulations/missingness-shift-mirrams)
