MIDAS
MIDAS explained: how an overcomplete denoising autoencoder produces multiple imputations for mixed data, with a worked example, pooling rules, pitfalls and FAQs.
Overview
MIDAS is a method for filling in missing values in tables. The name stands for multiple imputation with denoising autoencoders. It trains an overcomplete denoising autoencoder on the observed cells of a dataset, then uses that network to draw several plausible completions of the missing cells. Each completion is a full, filled-in copy of the table. Analysts run their normal analysis on each copy and combine the results, which keeps the extra uncertainty caused by missing data visible instead of hiding it.
The method matters because it joins two good ideas. Multiple imputation, a long-standing statistical practice, gives honest uncertainty. Denoising autoencoders, a deep learning technique, learn rich relations between columns, including nonlinear ones, and scale to large tables with mixed numeric and categorical columns. Together they give an imputer that is fast on big data and flexible on messy data. MIDAS is also a useful reference point for how tabular foundation models treat missing cells. See denoising autoencoders for the base idea and /model/research for the wider context.
What it is
MIDAS is an imputation algorithm built around a single neural network design.
- The network is an autoencoder: an encoder and a decoder trained to reproduce the data.
- It is denoising: during training the input is deliberately damaged, and the target is the undamaged data.
- It is overcomplete: the hidden layers are wider than the input, which is only safe because of the denoising setup.
- It is used for multiple imputation: after training, the network is run several times, with randomness, to produce several different completed datasets.
An everyday analogy
Imagine a jigsaw puzzle with missing pieces. A single guess for each hole gives one finished picture, and you would not know how sure you can be. Now imagine asking ten skilled puzzlers to each fill the holes using the visible pieces. Where the surrounding picture is clear, all ten agree. Where it is vague, their guesses differ. The spread among the ten tells you how uncertain the filled areas are. MIDAS acts like a panel of puzzlers produced from one trained model.
Single versus multiple imputation
| Aspect | Single imputation | Multiple imputation |
|---|---|---|
| Output | One filled table | Several filled tables |
| Uncertainty | Hidden. Filled values look as certain as observed ones | Visible. The spread between tables shows it |
| Standard errors | Too small | Corrected by pooling rules |
| Typical use | Quick cleaning, features for prediction | Statistical inference, reporting |
With single imputation, the filled values are treated as if they were real, so confidence intervals come out too narrow. Multiple imputation fixes that by repeating the fill-in and pooling the results.
Why a neural network
Classic multiple imputation methods fit a model for each variable given the others, or assume the data follow a joint distribution such as a multivariate normal. These can struggle with many columns, with nonlinear relations and with interactions. A denoising autoencoder learns relations across all columns at once, with flexible nonlinear layers, and can be trained on large data using standard GPU tools. That is the motivation for the method.
Where it came from
The statistical base is multiple imputation, developed in the 1970s and 1980s as a way to handle missing data in surveys. It rests on a simple recipe: create several completed datasets, analyse each, and combine the results with pooling rules usually called Rubin's rules. Popular software tools for chained equations and for joint normal models made the practice common in the social and medical sciences.
The deep learning base is the denoising autoencoder, introduced for representation learning in 2008. See denoising autoencoders and autoencoders.
The meeting of the two ideas came in the late 2010s. A 2018 preprint described using denoising autoencoders for multiple imputation of missing data, with an emphasis on a design suited to survey and political science data, where datasets can be large and mixed. A longer version appeared in a political science journal in 2022 under the title "The MIDAS Touch: Accurate and Scalable Missing-Data Imputation with Deep Learning". The authors released an open source Python package, commonly called MIDASpy, built on a deep learning framework (see TensorFlow). Later versions of the package added further options, such as a variational variant, so check the current documentation for the exact list of features.
The method sits in a wider line of deep imputation work, including generative adversarial approaches (see GAN), variational approaches (see VAE) and transformer based models. See also imputation networks.
How it works
Step 1: prepare the data
MIDAS needs numeric input for the network, so the table is converted.
- Numeric columns are rescaled, commonly to a fixed range or to zero mean and unit variance.
- Categorical columns are one-hot encoded. A column with k levels becomes k binary columns that sum to one.
- Binary columns may be treated as numeric or as two-level categories.
- Missing cells are marked in a mask, and a placeholder (such as zero) is put in them so the network can read the row.
Keeping the mask is essential. The network must know which cells are real and which are placeholders.
Step 2: build an overcomplete network
The encoder has hidden layers with more units than the input has columns. For instance, if the input has 20 columns, a hidden layer may have 60 units. A common pattern is three layers of a similar width, often with a saturating activation. The decoder mirrors the encoder and ends in an output layer with one unit per input column.
An overcomplete network would normally learn to copy its input and be useless. The denoising setup, next, is what makes it work.
Step 3: corrupt the input during training
Before each training pass, the input layer applies dropout: a random share of input values is set to zero. This share is high, often around half. So on every pass, the network sees a different subset of the row and must restore the whole row from it.
- Dropout is applied to the input, which is the corruption.
- Some designs also use dropout in later layers as an ordinary regularizer.
- The corruption is redrawn at every pass, so the network cannot memorize which cells will be hidden.
Step 4: compute the loss on observed cells only
The output is compared with the clean, observed data. The loss is computed only on cells that were actually observed, because for missing cells there is nothing to compare with.
- For numeric columns, the loss is squared error.
- For categorical columns, the output units for that column pass through a softmax and the loss is cross-entropy.
This mixed loss is what lets MIDAS handle numeric and categorical columns in one network.
Step 5: train
Use a standard optimizer such as Adam, mini-batches and a fixed number of epochs, or stop early using a held-out set of observed cells.
Step 6: draw multiple imputations
After training, the network is used to fill the gaps. To get several different completions, the method introduces randomness each time.
- Run the trained network with input dropout still active, so that each pass sees a random subset of the observed cells.
- Alternatively, retrain from different random seeds and take one fill from each.
- Each pass produces predictions for the missing cells. For numeric columns the prediction is a real value. For categorical columns the output is a probability vector, and a category can be sampled from it.
Because the randomness changes which inputs the network sees, the predicted values differ between draws, and the differences reflect how much the missing cells depend on which other cells are visible.
Make m such draws, with m often between 5 and 20, and you have m completed tables.
# Sketch of the MIDAS workflow using the style of the MIDASpy package.
# Check the package documentation for exact argument names in your version.
import pandas as pd
import MIDASpy as md
df = pd.read_csv("survey.csv")
cat_cols = ["region", "party", "employment"]
# one-hot encode categorical columns, keep track of the new column names
cat_dummies, cat_names, _ = md.cat_conv(df[cat_cols])
data = df.drop(columns=cat_cols).join(cat_dummies)
# scale numeric columns and record the mask of missing cells
data = data.copy()
imputer = md.Midas(
layer_structure=[256, 256, 256], # wide hidden layers: overcomplete
vae_layer=False, # plain denoising autoencoder
seed=89,
input_drop=0.75, # strong input dropout (the corruption)
)
imputer.build_model(data, softmax_columns=cat_names)
imputer.train_model(training_epochs=20)
completed = imputer.generate_samples(m=10).output_list # list of 10 tables
Step 7: analyse each table and pool
Run the intended analysis on each of the m completed tables, for example a regression. Then combine the m sets of results with Rubin's rules.
For a quantity of interest Q (such as a regression coefficient) estimated as Q_i in table i, with within-imputation variance U_i:
- The pooled estimate is the average of the Q_i. Call it Q_bar.
- The within-imputation variance is the average of the U_i. Call it U_bar.
- The between-imputation variance B is the sample variance of the Q_i.
- The total variance is T = U_bar + (1 + 1/m) * B.
The standard error is the square root of T. The term with B is the added uncertainty due to missing data. If the imputations all agree, B is near zero and the missing data cost little. If they disagree, B is large and the confidence interval widens, as it should.
Why input dropout creates useful variation
A subtle point: the variation between draws comes from the model being asked to predict from different random views of the same row, and, if you retrain, from different random starting weights. This is an approximation to sampling from the true predictive distribution of the missing cells. It is not a full Bayesian posterior, and MIDAS does not claim to be one. In practice, the spread tends to be a reasonable guide, but it is not guaranteed to be calibrated in every setting, so it is wise to check.
Why overcomplete
A wide hidden layer gives the network room to represent many combinations of columns, including interactions, which is hard to get with a narrow bottleneck. The strong input dropout then prevents the cheap solution of copying. The pair works like a student with a large notebook who is told that half of every page will be torn out before the exam: the only safe way to study is to learn how the pieces fit together.
What a MIDAS run looks like end to end
- Load the table and list which columns are numeric and which are categorical.
- Count missing cells per column and per row, and plot the pattern.
- Encode and scale the data, and build the mask of observed cells.
- Hold out a small share of observed cells for validation.
- Train the network with strong input dropout and early stopping.
- Check the error on the held-out cells against simple baselines.
- Draw m completed tables with the randomness switched on.
- Run the planned analysis on each table.
- Pool the results with Rubin's rules and report the standard errors.
- Keep a flag for every imputed cell in the released data.
Design choices at a glance
| Choice | Typical setting | Effect |
|---|---|---|
| Hidden layers | Three, each two to three times the input width | More capacity for interactions |
| Activation | Saturating function such as tanh | Bounded, stable outputs |
| Input dropout | Between 0.5 and 0.8 | Stronger corruption, less copying |
| Optimizer | Adam with a small learning rate | Fast and stable training |
| Epochs | Chosen by early stopping | Avoids overfitting |
| Number of draws | Ten to twenty | More stable pooled errors |
| Categorical output | Softmax per column group | Valid probabilities over levels |
Reading the spread between draws
The spread between draws is a signal in itself. A cell whose draws agree is well determined by its row. A cell whose draws differ widely is poorly determined. Sorting cells by spread shows where the imputed data are least reliable. This helps decide which cells to collect again, which to flag in a report, and which to exclude from sensitive analyses.
Key concepts and terms
| Term | Meaning |
|---|---|
| Missing data | Cells with no recorded value |
| Imputation | Filling in missing cells with estimated values |
| Multiple imputation | Making several completed datasets to reflect uncertainty |
| Rubin's rules | Formulas for pooling estimates and variances from several imputed datasets |
| Within-imputation variance | The usual sampling variance of an estimate in one completed table |
| Between-imputation variance | The spread of estimates across completed tables |
| Denoising autoencoder | A network trained to restore clean input from a corrupted copy |
| Overcomplete | Having hidden layers wider than the input |
| Input dropout | Randomly zeroing input values, used here as the corruption |
| Softmax column group | The set of output units for one categorical column, normalized to probabilities |
| One-hot encoding | Representing a category by a vector with a single one |
| Missing completely at random | Missingness unrelated to any data |
| Missing at random | Missingness that depends only on observed data |
| Missing not at random | Missingness that depends on the missing value itself |
| Observed mask | A record of which cells have real values |
The three kinds of missingness
This list matters because any imputation method makes assumptions about it.
- Missing completely at random. A cell is missing by chance, for example a sensor that drops readings at random. Imputation is easy and unbiased in principle.
- Missing at random. Missingness depends on other observed columns, for example older respondents skipping an income question. Methods that condition on other columns, such as MIDAS, can handle this.
- Missing not at random. Missingness depends on the unobserved value itself, for example high earners refusing to state income. No method can fix this from the data alone, and extra assumptions or sensitivity analysis are needed.
A worked example
A numeric walk-through of pooling
Suppose you imputed a table m = 5 times and ran a regression on each. The coefficient of interest and its squared standard error (the within-imputation variance) are:
| Imputation | Estimate Q_i | Variance U_i |
|---|---|---|
| 1 | 2.10 | 0.040 |
| 2 | 1.90 | 0.042 |
| 3 | 2.30 | 0.038 |
| 4 | 2.00 | 0.041 |
| 5 | 1.70 | 0.039 |
Compute the pieces.
- Pooled estimate Q_bar = (2.10 + 1.90 + 2.30 + 2.00 + 1.70) / 5 = 10.0 / 5 = 2.00.
- Within variance U_bar = (0.040 + 0.042 + 0.038 + 0.041 + 0.039) / 5 = 0.200 / 5 = 0.040.
- Between variance B is the sample variance of the estimates. The deviations from 2.00 are 0.10, -0.10, 0.30, 0.00 and -0.30. Their squares are 0.01, 0.01, 0.09, 0.00 and 0.09, summing to 0.20. Dividing by m - 1 = 4 gives B = 0.05.
- Total variance T = U_bar + (1 + 1/5) * B = 0.040 + 1.2 * 0.05 = 0.040 + 0.060 = 0.100.
- The standard error is the square root of 0.100, which is about 0.316.
Compare with single imputation. A single filled table would report a standard error near the square root of 0.040, which is 0.2. The pooled value of 0.316 is about 58 percent larger. The difference is the honest cost of the missing data. A single imputation would have made the result look more certain than it is.
A related quantity is the fraction of missing information, which grows with B relative to T. Here the share of the total variance due to between-imputation spread is 0.06 out of 0.10, which is 60 percent, a sign that the missing data matter a lot for this coefficient.
An end-to-end sketch in plain PyTorch
If you want to see the mechanism without a package, here is a compact version of the same idea in PyTorch. It handles numeric columns only, to keep it short.
import torch
import torch.nn as nn
def fit_midas_like(X, obs, hidden=3, epochs=300, p_drop=0.5, seed=0):
"""X: (N, D) float tensor with placeholders (0) in missing cells.
obs: (N, D) bool tensor, True where the value is real."""
torch.manual_seed(seed)
D = X.shape[1]
H = hidden * D # overcomplete width
net = nn.Sequential(
nn.Dropout(p_drop), # corruption on the input
nn.Linear(D, H), nn.Tanh(),
nn.Linear(H, H), nn.Tanh(),
nn.Linear(H, D),
)
opt = torch.optim.Adam(net.parameters(), lr=1e-3)
for _ in range(epochs):
net.train()
out = net(X)
loss = (((out - X) ** 2) * obs).sum() / obs.sum() # observed cells only
opt.zero_grad()
loss.backward()
opt.step()
return net
def draw_imputations(net, X, obs, m=10):
net.train() # keep dropout ON to get varied draws
tables = []
with torch.no_grad():
for _ in range(m):
pred = net(X)
tables.append(torch.where(obs, X, pred))
return tables
A real implementation also scales the dropout output at inference, handles categorical softmax groups, and may add noise to numeric draws. The sketch shows the core: corrupt the input, learn from observed cells, and draw many fills by keeping the randomness on.
Checking quality
Never trust an imputer without a test. A simple check:
- Take the observed cells and hide an extra 10 to 20 percent at random.
- Impute them.
- Compare imputed values with the truth: mean absolute error for numbers, accuracy for categories.
- Compare against baselines: column mean or mode, nearest neighbours, and a tree-based imputer.
- Check coverage: when you compute a 90 percent interval from the draws, does the truth fall inside about 90 percent of the time?
The last step tests whether the spread among draws is a trustworthy measure of uncertainty.
Strengths and limitations
Strengths
- Uncertainty is kept. Multiple draws feed standard pooling rules.
- Handles mixed types. Numeric and categorical columns are treated together with matching losses.
- Learns nonlinear relations and interactions, with no need to specify them.
- Scales well. Training uses mini-batches and GPUs, so large tables are feasible.
- Needs little tuning in the basic form. Defaults for width, depth and dropout often work.
- Works with any number of columns without writing one model per column.
- Good empirical accuracy compared with several classic multiple imputation approaches in reported studies, especially on larger, more complex datasets.
- Open source tooling exists, so it is easy to try.
Limitations
- Assumes missing at random. If missingness depends on the missing value itself, results can be biased.
- The spread of draws is an approximation. It is not a formal posterior, and coverage can fall short of the nominal level.
- Needs enough data. Neural networks can overfit small tables. On small samples, classic methods may do as well or better.
- Needs preprocessing. Scaling, one-hot encoding and consistent handling of rare categories are all required.
- Sensitive to hyperparameters. Dropout rate, width and number of epochs matter, and a poor choice gives poor fills.
- Wide one-hot encodings. Columns with many levels increase the width of the network.
- Less interpretable than a regression-based imputation model. It is hard to say why a particular value was chosen.
- Fixed schema. A trained network is tied to the columns it was trained on.
- Not designed for time series structure, text or images, unless features are extracted first.
- The imputation model and the analysis model should be compatible. If the analysis uses an interaction or nonlinearity that the imputation model could not represent, results may be biased.
How it compares with alternatives
| Method | Idea | Strength | Weakness |
|---|---|---|---|
| MIDAS | Overcomplete denoising autoencoder, multiple draws | Scalable, flexible, mixed types, uncertainty | Approximate uncertainty, needs enough data |
| Mean or mode filling | Fill with the column average | Trivial | Ignores relations, shrinks variance |
| k-nearest neighbour imputation | Copy values from similar rows | Simple, local | Slow on large tables, sensitive to scaling |
| Chained equations | Cycle through columns, model each given the others | Flexible, well understood, a standard in statistics | Slow with many columns, model per column |
| Joint normal model | Assume a multivariate normal distribution | Fast, principled | Poor for categories and nonlinearity |
| Tree-based imputers (see Random forest) | Predict each column using trees | Handles nonlinearity and mixed types | One model per column, limited uncertainty |
| Gradient boosting imputers (see XGBoost, LightGBM) | Predict each column with boosted trees | Strong accuracy | Cost grows with columns |
| GAN based imputers | Adversarial generator fills cells | Can capture complex joint laws | Training can be unstable |
| VAE based imputers | Latent variable model, sample completions | Natural sampling, uncertainty | Harder to tune, blurry means |
| Tabular foundation models | Pre-trained model fills cells in context | No per-dataset training, many tables seen | Dependent on pre-training coverage |
If your problem is a medium-sized table with a modest number of columns, chained equations remain a strong baseline. If your problem is a large table with many columns, mixed types and complex relations, a denoising autoencoder approach like MIDAS is attractive. If you want one model that imputes across many different tables without per-dataset training, look at foundation models, covered next.
Role in tabular foundation models
A tabular foundation model is pre-trained on a large number of tables, often synthetic ones drawn from a prior over data-generating processes, sometimes real ones, and later applied to new tables with in-context learning, with no weight updates. Such a model must deal with exactly the issues MIDAS addresses: rows and columns, mixed types, missing cells and, often, no column names. See TabPFN, TabICL, TabSTAR and XTab, and see /model/architecture for how such models are built.
The same objective at a larger scale
MIDAS trains one network on one table to restore hidden cells. A tabular foundation model trains one network on many tables to do the same, among other tasks. The shared principle is masked or corrupted cell prediction (see masked cell modeling). The difference is generalization. MIDAS learns the relations of a single dataset and is then specific to it. A foundation model learns general habits of tables, such as the way correlated columns behave, how skewed distributions look and how categories relate to numbers, and so it can fill cells in a table it has never seen, using the observed rows as context.
Imputation as a built-in capability
For a foundation model, imputation is a natural extra task. A missing cell is a query: given the visible cells of this row and the rows in the context, predict this cell. This is the same format as classification and regression, where the label is the hidden cell. So imputation needs no separate pipeline. MIDAS is a clear example of the single-table version, and a good baseline to compare a foundation model against.
Uncertainty and multiple draws
MIDAS shows why uncertainty matters. Foundation models can provide it through predictive distributions, for example by predicting over bins for numeric targets or by sampling several outputs. A multiple-imputation workflow with a foundation model would draw several fills, analyse each and pool. The pooling rules are the same as for MIDAS.
Mixed types and no column names
MIDAS handles types by using one-hot encodings and matching losses, and it uses column names only to keep track of groups. A foundation model must go further when names are absent or uninformative: it has to infer the type and role of a column from its values and from its relations to others. Pre-training on tables with scrambled or removed names teaches that skill. Tables that carry text descriptions let the model use names when they exist, as in TabSTAR.
Missingness patterns
MIDAS is built for the missing at random case. A foundation model can be pre-trained with a variety of missingness mechanisms, including random, column dependent and value dependent patterns, to be more robust. It cannot remove the basic limit that missing not at random data are not identifiable from the data alone, but pre-training on varied patterns reduces surprises.
Anomaly detection and cleaning
An imputer is also a detector. If you hide an observed cell and the model predicts it poorly, the cell may be an error. Cells with large gaps between observed and predicted values can be flagged for review. See anomaly detection. This links imputation, cleaning and outlier detection in a single mechanism.
Sector and semantic classification
Imputation quality depends on knowing what a column means. A model that has learned semantic types, such as a date, a postal code or a currency amount, fills cells better. Semantic and sector heads (see semantic sector heads) can provide that context to the imputer, and imputed tables give cleaner inputs to semantic classifiers.
Continual learning
As a foundation model is extended with new datasets, its imputation skill can erode. Keeping masked-cell examples, or recipes to regenerate synthetic ones, in the training mix through experience replay protects that skill. See also EWC and LwF.
For measured results see /model/benchmarks, and for the research background see /model/research.
Practical guidance and best practices
Diagnose missingness first
Before imputing, measure how much is missing per column and per row, and look for patterns. Plot a map of missing cells. Ask whether missingness plausibly depends on the missing value itself. If so, plan a sensitivity analysis.
Include the variables of your analysis
Include in the imputation all variables that will be used later in the analysis, including the outcome, and variables that help predict missingness. Leaving out the outcome from the imputation can bias the analysis toward no effect.
Prepare the data carefully
- Rescale numeric columns, and consider transforms for heavy tails.
- One-hot encode categories and group rare levels.
- Keep identifiers and free text out, or turn them into features first.
- Remove columns that are almost entirely missing, or that carry no information.
Choose the network size sensibly
Start with hidden layers about two or three times the input width and three layers. Increase width for complex data and reduce it for small tables. Watch validation error on hidden observed cells.
Set the corruption level
Use strong input dropout, often between 0.5 and 0.8. Too little allows copying, too much leaves too little information. Tune on a held-out set of observed cells.
Use early stopping
Hold out a share of observed cells and stop when their error stops improving. This prevents overfitting and saves time.
Pick the number of imputations
Five was once the common default, but with modern computing, using 10 to 20 or more gives more stable standard errors, particularly when much data is missing. A rule of thumb is to use at least as many imputations as the percentage of incomplete cases.
Make draws that truly differ
If all draws come out nearly identical, the between-imputation variance is under-estimated. Check that the draws differ and consider retraining with different seeds as well as keeping dropout on.
Check calibration
Hide known cells, impute them, and test whether intervals from the draws cover the truth at the stated rate. If coverage is low, increase variability or use a more conservative interval.
Pool correctly
Use Rubin's rules for estimates and standard errors. Do not average the filled tables into one and analyse that, because this brings back the problem of overconfidence. Many statistical packages have routines for pooling.
Match imputation and analysis
If the analysis model contains interactions or nonlinear terms, make sure the imputation can represent them. Flexible neural imputation helps, but check.
Keep a record
Store the seeds, the network settings, the data version and the mask of originally missing cells. Flag imputed cells in released data.
Use reliable tooling
The MIDASpy package supports this workflow. For general data work see scikit-learn for baselines and metrics, and PyTorch or TensorFlow for custom variants.
Common mistakes
- Using a single imputed table and reporting standard errors as if the data were complete.
- Averaging the imputed tables into one table before analysis.
- Computing the loss on placeholder values in missing cells.
- Leaving dropout off when drawing imputations, so every draw is identical.
- Leaving dropout on when evaluating point predictions without understanding the added variance.
- Forgetting to scale numeric columns, so one large-valued column dominates.
- Ignoring the outcome variable in the imputation model.
- Imputing the training and test sets together before a split, which leaks information.
- Using too few imputations and getting unstable standard errors.
- Treating imputed values as observed data in later reporting.
- Assuming missing at random without thinking about it.
- Using a narrow bottleneck network and expecting it to learn complex interactions.
- Using too many epochs on a small table and overfitting.
- Skipping a validation test where known cells are hidden and recovered.
- Ignoring rare categories, which then get poor or arbitrary fills.
Frequently asked questions
What is MIDAS?
MIDAS is a missing data imputation method that stands for multiple imputation with denoising autoencoders. It trains an overcomplete denoising autoencoder on the observed cells of a table and uses it to draw several plausible completions of the missing cells, so that analysts can pool results and keep the uncertainty from missing data.
What is multiple imputation and why is it better than filling once?
Multiple imputation creates several completed copies of a dataset, analyses each one and combines the results with pooling rules. Filling once treats guessed values as real, which makes confidence intervals too narrow. Multiple imputation adds the between-copy variation to the variance, giving more honest uncertainty.
What does overcomplete mean in this context?
An overcomplete autoencoder has hidden layers wider than its input. Normally this would let the network copy its input without learning anything. In MIDAS, strong dropout on the input prevents copying, so the extra width is used to represent many combinations of columns and their interactions.
How does MIDAS handle categorical variables?
Categorical columns are one-hot encoded for input. At the output, the units for each categorical column pass through a softmax so that they form a probability distribution over levels, and the loss for that column is cross-entropy. When drawing imputations, a category can be sampled from the predicted probabilities.
How are multiple imputations produced?
After training, the network is run several times with its input dropout still active, so each pass sees a random subset of the observed cells and gives somewhat different predictions for the missing ones. Retraining with different random seeds is another way. Each run gives one completed table, and several runs give the set of imputations.
How do I combine results from the imputed datasets?
Use Rubin's rules. Average the estimates across the datasets, average the within-dataset variances, compute the variance of the estimates across datasets, and add the within variance to the between variance scaled by one plus one over the number of imputations. The square root of the total is the pooled standard error.
Does MIDAS work if data are missing not at random?
Not on its own. Like most imputation methods it relies on missing at random, where missingness depends only on observed data. If missingness depends on the unseen value, the data alone cannot reveal the bias, and you need extra assumptions or a sensitivity analysis to see how results change.
How many imputations do I need?
A common older default was five, but ten to twenty or more is safer with modern computing, especially when a large share of the data is missing. More imputations make the standard errors more stable. A common rule of thumb is to use at least as many imputations as the percentage of incomplete cases.
Is MIDAS better than chained equations?
It depends on the data. Reported studies suggest MIDAS can be accurate and much faster on large, complex tables with many columns, while chained equations remain a strong and well understood choice for small or medium tables with modest numbers of variables. The best practice is to compare both on your data by hiding known cells.
How does MIDAS relate to tabular foundation models?
Both rely on predicting hidden cells from visible ones. MIDAS trains on one table, while a tabular foundation model is pre-trained on many tables and fills cells in new tables using context, with no per-table training. MIDAS is a clear single-table baseline for imputation and shows why uncertainty and mixed types matter.
Glossary
- MIDAS: multiple imputation with denoising autoencoders.
- Imputation: filling in missing values in a dataset.
- Multiple imputation: producing several completed datasets to represent uncertainty.
- Rubin's rules: formulas for pooling results across multiply imputed datasets.
- Denoising autoencoder: a network trained to restore clean data from a corrupted copy.
- Overcomplete: having hidden layers wider than the input.
- Input dropout: randomly zeroing input values during training.
- Softmax: a function that turns scores into probabilities that sum to one.
- One-hot encoding: representing a category as a vector with a single one.
- Within-imputation variance: the variance of an estimate inside one completed dataset.
- Between-imputation variance: the variance of estimates across completed datasets.
- Missing completely at random: missingness unrelated to any data.
- Missing at random: missingness that depends only on observed data.
- Missing not at random: missingness that depends on the missing value itself.
- Coverage: the share of times an interval contains the true value.
- Fraction of missing information: the part of the total variance caused by missing data.
Summary
MIDAS combines the honesty of multiple imputation with the flexibility of a denoising autoencoder. An overcomplete network is trained with strong input dropout and a loss on observed cells only, using squared error for numeric columns and cross-entropy over softmax groups for categorical ones. Several randomized passes then give several completed tables, which analysts pool with Rubin's rules so that uncertainty from missing data stays visible.
It works best on larger tables with many columns, mixed types and nonlinear relations, and it assumes data are missing at random. Always diagnose missingness, include the outcome in the imputation, validate by hiding known cells, check calibration and pool correctly. In tabular foundation models, the same idea of predicting hidden cells from visible ones is scaled across many tables and used in context, with imputation, anomaly detection and semantic classification as natural neighbours. For related reading, see denoising autoencoders, imputation networks, masked cell modeling, /model/research, /model/architecture and /model/benchmarks.
Related
It belongs to Self-supervised for tables in my stack, under Foundation models and applied ML. Others in the same group: