MIDAS: multiple imputation with a denoising autoencoder
MIDAS learns to rebuild a table from a deliberately corrupted copy, then fills the gaps several times. The spread between the imputations is the uncertainty that a single filled table would hide.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
Train the autoencoder and watch the reconstruction loss fall. The table shows the mean of the m imputations; the strip shows all m values for the chosen hidden cell with the true value. The panel compares the error with mean imputation and reports the between-imputation variance.
Things to try
- Train, then click different hidden cells: some are certain, some are not.
- Set m to 30 for a smoother picture of the spread.
- Reset and stop training early: more spread, larger error.
How it works in real systems
MIDAS (Lall and Robinson, 2022) is available as MIDASpy; its imputations are analysed separately and pooled with Rubin's rules, so standard errors include the imputation uncertainty.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Data with gaps
import numpy as np
rng = np.random.default_rng(11)
def houses(n):
s = rng.integers(35, 181, n); ro = np.clip(np.round(s / 32 + rng.normal(0, .6, n)), 1, 6); ag = rng.integers(0, 81, n); d = np.round(rng.uniform(.5, 25, n), 1)
p = 2.4 * s * (1 + .45 * np.exp(-d / 4)) - .9 * ag + 14 * ro + 40
return np.c_[s, ro, ag, d].astype(float), np.round(p * (1 + rng.normal(0, .07, n)))
X, y = houses(60); X_test, y_test = houses(300)
import pandas as pd
df = pd.DataFrame(np.c_[X, y], columns=["size", "rooms", "age", "distance", "price"])
full = df.copy(); mask = rng.random(df.shape) < .2; df = df.mask(mask)
Step 2 · MIDAS with MIDASpy
# pip install MIDASpy (TensorFlow under the hood)
import MIDASpy as md
imputer = md.Midas(layer_structure=[128, 128], vae_layer=False, seed=89, input_drop=0.75)
imputer.build_model(df) # scales and corrupts the inputs internally
imputer.train_model(training_epochs=50)
imputations = imputer.generate_samples(m=10).output_list # ten complete tables
Step 3 · Use all m tables
Fit the analysis on each imputed table and pool with Rubin's rules: the average estimate, and a variance that adds the between-imputation spread.
import statsmodels.api as sm
est, var = [], []
for t in imputations:
r = sm.OLS(t["price"], sm.add_constant(t[["size", "age", "distance"]])).fit()
est.append(r.params); var.append(r.bse ** 2)
est, var = pd.DataFrame(est), pd.DataFrame(var)
m = len(imputations); q = est.mean(); W = var.mean(); B = est.var()
print(pd.DataFrame({"estimate": q, "std error": np.sqrt(W + (1 + 1 / m) * B)}))