Özer Cem KELAHMETKELAHMET
AI Simulations · Large tabular foundation model

MIDAS: multiple imputation with a denoising autoencoder

MIDAS learns to rebuild a table from a deliberately corrupted copy, then fills the gaps several times. The spread between the imputations is the uncertainty that a single filled table would hide.

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

Train the autoencoder and watch the reconstruction loss fall. The table shows the mean of the m imputations; the strip shows all m values for the chosen hidden cell with the true value. The panel compares the error with mean imputation and reports the between-imputation variance.

Things to try

  • Train, then click different hidden cells: some are certain, some are not.
  • Set m to 30 for a smoother picture of the spread.
  • Reset and stop training early: more spread, larger error.

How it works in real systems

MIDAS (Lall and Robinson, 2022) is available as MIDASpy; its imputations are analysed separately and pooled with Rubin's rules, so standard errors include the imputation uncertainty.

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · Data with gaps

import numpy as np
rng = np.random.default_rng(11)
def houses(n):
    s = rng.integers(35, 181, n); ro = np.clip(np.round(s / 32 + rng.normal(0, .6, n)), 1, 6); ag = rng.integers(0, 81, n); d = np.round(rng.uniform(.5, 25, n), 1)
    p = 2.4 * s * (1 + .45 * np.exp(-d / 4)) - .9 * ag + 14 * ro + 40
    return np.c_[s, ro, ag, d].astype(float), np.round(p * (1 + rng.normal(0, .07, n)))
X, y = houses(60); X_test, y_test = houses(300)
import pandas as pd
df = pd.DataFrame(np.c_[X, y], columns=["size", "rooms", "age", "distance", "price"])
full = df.copy(); mask = rng.random(df.shape) < .2; df = df.mask(mask)

Step 2 · MIDAS with MIDASpy

# pip install MIDASpy  (TensorFlow under the hood)
import MIDASpy as md
imputer = md.Midas(layer_structure=[128, 128], vae_layer=False, seed=89, input_drop=0.75)
imputer.build_model(df)                       # scales and corrupts the inputs internally
imputer.train_model(training_epochs=50)
imputations = imputer.generate_samples(m=10).output_list   # ten complete tables

Step 3 · Use all m tables

Fit the analysis on each imputed table and pool with Rubin's rules: the average estimate, and a variance that adds the between-imputation spread.

import statsmodels.api as sm
est, var = [], []
for t in imputations:
    r = sm.OLS(t["price"], sm.add_constant(t[["size", "age", "distance"]])).fit()
    est.append(r.params); var.append(r.bse ** 2)
est, var = pd.DataFrame(est), pd.DataFrame(var)
m = len(imputations); q = est.mean(); W = var.mean(); B = est.var()
print(pd.DataFrame({"estimate": q, "std error": np.sqrt(W + (1 + 1 / m) * B)}))

More simulations

← Masked value · All AI Simulations · Missingness shift →

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open