Denoising autoencoders

Overview

A denoising autoencoder is a neural network that is trained to rebuild a clean input from a corrupted copy of it. The corruption can be random noise, hidden values, shuffled entries or dropped features. Because the network cannot simply copy its input, it must learn the structure of the data: which values go together, which are redundant and which are predictable from the rest. The result is a set of robust internal features, and a model that can fill in what was damaged.

This idea is one of the roots of modern self-supervised learning. Masked language models, masked image models and the denoising steps of diffusion models are all relatives of it. For tables, the denoising autoencoder is a natural fit. Real tables have missing cells, noisy measurements and mixed column types, so a model trained to restore hidden cells learns exactly the skill that imputation and cleaning need. Methods such as MIDAS are built directly on it. For the wider picture, see /model/research and /model/architecture.

What it is

A plain autoencoder has two parts: an encoder that squeezes the input into a code, and a decoder that rebuilds the input from that code. It is trained to make the output equal to the input. A denoising autoencoder changes only one thing. The encoder sees a corrupted input, but the loss compares the output with the original clean input.

In symbols, let x be a clean example. A corruption process produces a damaged version x~ from x. The encoder f maps x~ to a code h. The decoder g maps h to a reconstruction r. The loss measures the distance between r and the clean x, not between r and x~.

An everyday analogy

Imagine a friend who reads a newspaper with a coffee stain over some words. A person who knows the language can still read the sentence, because the surrounding words limit what the hidden ones can be. Each time you practice with a new stain, you get better at using context. A denoising autoencoder practices the same thing on data. The stain is the corruption. The skill that grows is the knowledge of which values fit together.

Why plain copying is not enough

A plain autoencoder with a wide code can learn the identity function. It copies the input to the output and learns nothing useful. Standard fixes limit the code size (an undercomplete design) or add a penalty (sparsity or contraction). A denoising autoencoder gives a different fix: corrupt the input, so that identity is no longer the right answer. The network must go beyond copying, even when the code is as wide as the input or wider. This is why an overcomplete denoising design is possible, and why it is useful.

The main corruption types

Each type teaches the model a different invariance. Masking teaches it to fill gaps. Swap noise teaches it which values are plausible within a column, so replaced entries look natural and the model must use the other columns to spot them.

Where it came from

Autoencoders themselves were studied for many years before the denoising version appeared, under names such as auto-association and bottleneck networks. The denoising idea was introduced for deep learning in 2008, in a paper titled "Extracting and Composing Robust Features with Denoising Autoencoders". A longer 2010 journal paper, "Stacked Denoising Autoencoders", showed that stacking layers trained this way gave strong unsupervised pre-training for deep classifiers. At that time, layer-wise pre-training was the main way to get deep networks to train well, and denoising layers were a competitive alternative to restricted Boltzmann machines.

Related work followed.

• A 2011 paper connected denoising autoencoders to score matching, showing that the reconstruction direction points toward regions of higher data density. This gave the method a firm probabilistic reading.
• Work in 2013 on generalized denoising autoencoders showed that they can be used as generative models, by sampling through a Markov chain that alternates corruption and reconstruction.
• Contractive autoencoders and variational autoencoders (VAE) offered other ways to regularize the code.
• Masked language modelling, as used in BERT-style pre-training in 2018 and later, can be seen as a denoising autoencoder on token sequences, where the corruption is masking.
• Masked autoencoders for images (MAE, 2021) apply the same idea with high masking ratios on image patches.
• Diffusion models (DDPM, 2020, and the wider family in diffusion models) train a network to remove noise at many levels, which is a multi-scale denoising objective.
• For tables, methods such as VIME (2020) used corruption and reconstruction of masks and values as a self-supervised task, and imputation methods such as MIDAS used overcomplete denoising autoencoders for multiple imputation.

The lesson of this history is that "corrupt and restore" keeps coming back as a pre-training signal, because it needs no labels and scales to large data.

How it works

Step 1: take a clean example

Start with a row x from the training set. In a table this is a vector of column values, with numeric values scaled and categorical values encoded.

Step 2: corrupt it

Apply a random corruption to get x~. Draw a new corruption every time the example is used. Fresh corruptions each epoch act like data augmentation and keep the network from memorizing.

For a table, a typical corruption is to pick a random share of cells and either mask them or replace them with values from other rows. Keep a record of which cells were corrupted. This record is useful for the loss and for some architectures.

Step 3: encode

The encoder maps x~ through one or more layers to a hidden code h. In the simplest case this is one linear layer followed by a nonlinearity.

Step 4: decode

The decoder maps h back to the input space. The output r has the same shape as x. For numeric columns, r is a real number. For categorical columns, r is a vector of scores, turned into probabilities by a softmax.

Step 5: compare with the clean input

The loss compares r with the original clean x. Mean squared error fits numeric columns. Cross-entropy fits categorical columns. In a table, the total loss is a sum over columns, possibly with weights.

A key design choice is where the loss is applied.

• On all cells. This is the original form. The model learns to reproduce clean values everywhere.
• On corrupted cells only. This focuses learning on the part the model could not see, and is the form used in masked modelling.
• On all cells with a larger weight for corrupted ones. A blend of the two.

Step 6: update and repeat

Backpropagation updates the encoder and decoder weights. Over many passes with fresh corruptions, the network learns a map from damaged inputs to clean outputs.

import torch
import torch.nn as nn

class DAE(nn.Module):
    def __init__(self, d_in, d_hidden):
        super().__init__()
        self.enc = nn.Sequential(nn.Linear(d_in, d_hidden), nn.ReLU())
        self.dec = nn.Linear(d_hidden, d_in)

    def forward(self, x_noisy):
        return self.dec(self.enc(x_noisy))

def corrupt(x, p=0.3):
    mask = (torch.rand_like(x) < p)
    x_noisy = x.clone()
    x_noisy[mask] = 0.0
    return x_noisy, mask

model = DAE(d_in=20, d_hidden=64)
opt = torch.optim.Adam(model.parameters(), lr=1e-3)

for epoch in range(50):
    for x in loader:                 # x: clean batch
        x_noisy, mask = corrupt(x)
        recon = model(x_noisy)
        loss = ((recon - x) ** 2).mean()   # compare with the CLEAN x
        opt.zero_grad()
        loss.backward()
        opt.step()

What the network learns

Because the corruption is random, the network cannot memorize which cells are damaged. It must learn a general rule: given the visible cells, what are the most likely values of the hidden ones? To do that well, it must model the joint distribution of the columns. The hidden units become detectors for combinations of values that tend to occur together.

A useful geometric picture: real data lie close to a curved surface (a manifold) inside the full input space. Corruption pushes a point off the surface. The denoising autoencoder learns to push it back. The vector from the corrupted point to the reconstruction points toward the surface, so the model learns the shape of the surface.

Why robust features appear

Features learned this way are stable under small damage, because the network was trained to give the same clean answer from many damaged versions. That stability makes them good inputs for a downstream classifier. It also makes them useful for finding errors, since points that the model cannot restore well are far from the learned surface.

Link to masked modelling

Masked modelling is the special case where the corruption is masking and the loss is applied to the masked cells. The difference between a classic denoising autoencoder and a masked model is mostly one of architecture and scale. Classic versions use fully connected layers. Modern versions use transformers with attention across tokens or columns. The principle is the same.

Link to probability

If the noise is small and Gaussian, the optimal denoiser has a clear meaning: its output minus its input is proportional to the gradient of the log density of the data (the score) at the noisy point. This is why a denoiser can drive sampling and why diffusion models work. The reading is general: learning to remove noise means learning where the data live and how dense they are there. See probability and statistics for the background.

Architecture variants

The denoising idea is independent of the network type. Common choices are listed below.

• Fully connected. The classic form, good for small tables and vectors.
• Convolutional. Used for images and for signals with local structure.
• Recurrent or sequence based. Used for time series and text.
• Transformer based. Used when columns or tokens interact through attention, as in modern masked models.
• Sparse or tied weights. The decoder shares weights with the encoder to cut parameters.
• Stacked. Several layers, trained one at a time or all together.

Where the loss can sit

A quick test of your understanding

Ask three questions about any denoising setup. What is damaged? What is the target? Where is the loss computed? If all three are clear, the setup is sound. Most bugs come from an unclear answer to one of them.

Training dynamics to expect

• The loss falls fast at first, because the network learns the column averages.
• It then falls slowly, as the network learns the relations between columns.
• A gap between training loss and validation loss on simulated gaps signals overfitting.
• Raising the masking rate raises the loss, and this is normal.
• If the loss on corrupted cells does not fall below the loss of a simple column-mean guess, the model has not learned the relations yet.

A simple baseline to beat

Before trusting any denoising model, compute the error of the column mean (or mode) on the same hidden cells. A good model should beat it clearly. Then compare with a nearest neighbour imputer. This keeps expectations honest.

Key concepts and terms

A worked example

A tiny numeric walk-through

Take a table with three columns: height in centimetres, weight in kilograms and shoe size. After scaling, one row is x = (0.8, 0.7, 0.9), meaning this person is tall, heavy and has a large shoe size. These columns are strongly related in real data.

Corrupt the row by masking the second column, so x~ = (0.8, 0.0, 0.9).

A trained denoising autoencoder looks at the visible values 0.8 and 0.9. It has learned that tall people with large shoes are heavy, so its reconstruction might be r = (0.79, 0.68, 0.88). The loss compares r with the clean x:

• Column 1: (0.79 - 0.8)^2 = 0.0001
• Column 2: (0.68 - 0.7)^2 = 0.0004
• Column 3: (0.88 - 0.9)^2 = 0.0004

The mean is (0.0001 + 0.0004 + 0.0004) / 3 = 0.0003. A network that had learned nothing useful would put something near the column average in the masked cell, and the error would be much larger.

The important point is that the second column was restored from the other two. This is the imputation skill, learned without any labels.

An imputation workflow on a table

The following code shows a compact imputation workflow with mixed types. It scales numeric columns, one-hot encodes a categorical column, trains with masking noise, and fills missing cells at the end.

import numpy as np
import pandas as pd
import torch
import torch.nn as nn

df = pd.read_csv("table.csv")
num_cols = ["age", "income", "tenure"]
cat_cols = ["segment"]

# 1. prepare: scale numerics, one-hot categoricals, keep a missing mask
X_num = df[num_cols]
obs_num = ~X_num.isna()
X_num = (X_num - X_num.mean()) / X_num.std()
X_num = X_num.fillna(0.0)                      # placeholder for missing cells
X_cat = pd.get_dummies(df[cat_cols].fillna("missing"))

X = torch.tensor(np.hstack([X_num.values, X_cat.values]), dtype=torch.float32)
obs = torch.tensor(np.hstack([obs_num.values,
                              np.ones_like(X_cat.values, dtype=bool)]))

# 2. model: overcomplete denoising autoencoder with input dropout
d = X.shape[1]
model = nn.Sequential(
    nn.Dropout(0.5),                 # the corruption, applied in training mode
    nn.Linear(d, 3 * d), nn.Tanh(),
    nn.Linear(3 * d, 3 * d), nn.Tanh(),
    nn.Linear(3 * d, d),
)
opt = torch.optim.Adam(model.parameters(), lr=1e-3)

# 3. train: the loss counts only cells that were truly observed
for epoch in range(200):
    model.train()
    recon = model(X)
    loss = (((recon - X) ** 2) * obs).sum() / obs.sum()
    opt.zero_grad()
    loss.backward()
    opt.step()

# 4. impute: run in eval mode and copy predictions into the missing cells
model.eval()
with torch.no_grad():
    filled = torch.where(obs, X, model(X))

Three details are worth noticing. The loss uses only cells that were really observed, because the model has no target for the missing ones. The dropout layer is the corruption, and it is active only in training mode. And at inference the model reads the real observed cells and writes predictions for the missing ones.

Using the code as features

After training, the hidden layer can serve as an embedding. Feed the clean row through the encoder, then use the hidden vector as input to a classifier, a clustering step or a nearest neighbour search. Embeddings from a denoising autoencoder are often more stable than those from a plain one.

Strengths and limitations

Strengths

• Needs no labels. Any table or dataset can be used.
• Learns robust features. The features tolerate noise and missing values.
• Simple to build. A few layers and a corruption function are enough.
• Handles overcomplete codes. The corruption prevents the identity solution.
• Works well for imputation. The training task is the imputation task.
• Flexible with data types. Different losses can be used for different columns.
• Scales. Training is cheap per step and parallel on GPUs (see CUDA).
• Connects to a theory. The link to score matching explains why it works.
• Good pre-training signal. It can be added to a supervised loss as an auxiliary objective.

Limitations

• The corruption must match reality. If the noise in training does not resemble the damage in practice, the model may not help. Missing values in real tables are often not random.
• Point estimates only. A basic denoising autoencoder gives one best guess, not a distribution. Multiple imputation methods add a way to draw several guesses.
• Blurry output. With squared error, the reconstruction tends toward the average of plausible values.
• Choice of noise level. Too little noise gives a near copy. Too much noise destroys the signal.
• Fixed columns. A classic network has a fixed input size and does not transfer between tables with different columns.
• Order and scaling sensitivity. Poor scaling of numeric columns hurts training.
• Rare categories. Categorical columns with many levels need care, such as embeddings.
• Hard to read. Hidden units do not have clear meanings.
• Not a classifier. For prediction it is a pre-training or feature step, and a separate head is needed.
• Can be beaten on small tables. On small datasets, simple methods such as k-nearest neighbours or tree-based imputers can be competitive.

How it compares with alternatives

For a general introduction to the neighbouring ideas, see self-supervised learning and unsupervised learning.

Role in tabular foundation models

A tabular foundation model learns from raw tables. Its inputs have rows and columns, mixed types, missing cells and often no useful column names. It is pre-trained on a very large number of tables, either synthetic ones sampled from a prior over data-generating processes, or real ones, or both. Examples include TabPFN, TabICL, TabSTAR and XTab. Self-supervised tabular transformers such as SAINT and FT-Transformer also use denoising style objectives. See /model/architecture for how these parts fit together.

The denoising autoencoder is relevant at several levels.

As the pre-training objective

Masked cell modelling is the denoising autoencoder objective on a table. Hide a share of cells, then train the model to predict them from the rest. See masked cell modeling. Because every cell is a possible target, a single table supplies many training signals. The model must learn the relations between columns, the marginal distribution of each column and the way types behave. Nothing about this requires labels or column names.

As the engine of imputation

Real tables have missing cells. A model trained to restore masked cells already knows how to fill real ones. This is why imputation can be a built-in capability of a tabular foundation model, not an extra step. Methods in the family of imputation networks and MIDAS show the pattern. The foundation model version goes further, because it is trained on many tables and can adapt to a new table at inference time through context, without retraining.

Handling different missingness patterns

Missing data may be missing completely at random, at random given other columns, or not at random. A model trained only with random masking learns the first case best. A richer corruption process should include:

• Random cell masks.
• Whole-column masks within a row group.
• Row-level masks that hide many cells together.
• Masks that depend on the values (for example, high incomes are more often missing).

Pre-training with a mix of these makes the model more robust when it meets real patterns.

Mixed types

A table may hold numbers, categories, dates, flags and text. A denoising model for tables uses a different head and a different loss for each type: squared error or a distribution over bins for numbers, cross-entropy for categories. Swap noise is especially natural for mixed types, because the replaced value is a real value from the same column and so respects the type.

No column names

If column names are missing or meaningless, the model has to identify what a column is from its values and from how it relates to its neighbours. Denoising pre-training pushes in this direction: to restore a masked cell, the model must use the other cells, not the names. A good pre-training mix also scrambles or removes names so that the model does not lean on them.

In-context learning

Foundation models for tables often predict by reading a set of labelled rows in the context and answering for query rows, without updating weights. A denoising objective supports this because the model learns to condition on visible cells to predict hidden ones. A query row with a hidden target is just a row with one masked cell. Seen this way, classification and regression are special cases of masked cell prediction where the hidden cell is the label.

Synthetic pre-training

When tables are generated from a prior, the clean version is known exactly, so a corruption can be applied and the exact clean answer used as a target. There is no label noise in the reconstruction target. Synthetic data also allows an unlimited supply of tables with chosen missingness patterns and chosen types, which is hard to get from real collections.

Anomaly detection

A model trained to restore normal data will restore anomalies poorly, since they lie far from the learned surface. The reconstruction error per row, or per cell, is an anomaly score. This gives a direct bridge from denoising to anomaly detection, alongside classical methods such as isolation forest and one-class SVM. Per-cell errors also point to which column looks wrong, which helps with data cleaning.

Continual learning

When a tabular model is extended with new datasets, the denoising objective can be kept in the training mix so that the imputation skill does not fade. Experience replay of masked-cell batches, or of generator recipes for synthetic tables, is a simple way to do this. See also EWC and LwF.

Semantic and sector classification

Learned row and column representations can feed a head that predicts the meaning of a table or column, such as its sector or its semantic type. A denoising pre-trained encoder gives stable representations for that head, because the features were trained to survive missing and noisy cells. See semantic sector heads.

For measured results of such models, see /model/benchmarks, and for background reading see /model/research.

Practical guidance and best practices

Choose the corruption to match the problem

• If the goal is imputation, mask cells the way real gaps appear.
• If the goal is robust features, mix several corruption types.
• If the data are continuous and smooth, add Gaussian noise at a level small enough to keep the structure.
• If the data are mixed, prefer masking and swap noise.

Tune the corruption rate

Start with a masking rate between 10 and 50 percent and test a few values. Higher rates force the network to learn more global structure. Lower rates teach finer detail. Some methods vary the rate from batch to batch, which trains the model to handle many levels of damage.

Scale the data

Standardize numeric columns, or use a robust scaling for heavy tails. Consider a log or rank transform for skewed columns. For categories, use one-hot codes for small sets and embeddings for large ones.

Compute the loss on the right cells

Never train on cells that are missing in the original data and have only a placeholder. Use an observed-mask. This is the most common source of silent bugs in tabular denoising.

Use an overcomplete design when you want imputation

A wider hidden layer than the input, together with strong input dropout, often works well for imputation, as the corruption removes the risk of copying.

Regularize

Use weight decay, early stopping on a validation set with simulated missing cells, and a modest depth. Large networks can overfit small tables.

Validate by hiding known cells

To test imputation quality, hide a share of known cells, impute them, and compare with the truth. Use the metric that matches the type: mean absolute error or root mean squared error for numbers, accuracy or log loss for categories. Evaluate on the missingness pattern you expect in practice.

Quantify uncertainty

A single imputation hides uncertainty. Draw several completions, by using different corruption masks or different random seeds, then analyse the spread. Combine analysis results across completions with standard rules for multiple imputation. See MIDAS for a worked design.

Use it as an auxiliary loss

For supervised tasks, a denoising loss added to the main loss can improve generalization, especially with few labels. Weight it so it does not dominate.

Watch for leakage

When using the denoising model on a train and test split, fit scalers and encoders on the training set only.

Use good tooling

Frameworks such as PyTorch, TensorFlow and Keras all support this model class. For evaluation, scikit-learn offers metrics and baseline imputers.

Common mistakes

• Comparing the output with the corrupted input instead of the clean input. This turns the model back into a plain autoencoder.
• Using the same corruption mask every epoch. The network memorizes the pattern.
• Applying corruption at inference time. Dropout must be off when imputing.
• Computing the loss on placeholder values for truly missing cells.
• Forgetting to scale columns, so that one large-valued column dominates the loss.
• Using a bottleneck that is too narrow when the aim is imputation of subtle relations.
• Setting the noise too high, so that nothing is left to learn from.
• Testing imputation with a missingness pattern that is easier than the real one.
• Treating the imputed values as real data in later statistics without accounting for uncertainty.
• Ignoring categorical structure, for example treating category codes as ordered numbers.
• Fitting scalers on the full data before splitting.
• Expecting a model trained on one table to work on a table with different columns, without a design that supports varying schemas.
• Assuming a good reconstruction error means a good downstream result. Check the task you care about.

Frequently asked questions

What is a denoising autoencoder?

A denoising autoencoder is a neural network trained to rebuild a clean input from a corrupted version of it. The encoder compresses the damaged input into a code and the decoder produces a reconstruction, which is compared with the original clean data. This forces the network to learn the structure of the data instead of copying.

How is a denoising autoencoder different from a regular autoencoder?

A regular autoencoder is given the clean input and asked to reproduce it, so it can learn the identity function unless the code is narrow or penalized. A denoising autoencoder is given a corrupted input and asked for the clean one. That small change makes copying useless and leads to more robust features, even when the code is larger than the input.

What kinds of noise can be used?

Common choices are Gaussian noise, masking noise that zeroes a random share of values, salt and pepper noise, dropout on inputs, and swap noise that replaces values with others from the same column. For tables, masking and swap noise are popular because they respect value ranges and types, and they mimic missing or wrong cells.

Why does a denoising autoencoder learn useful features?

To restore damaged input, the network must know which values tend to occur together, and so it must model the joint structure of the data. The hidden units become detectors for these patterns. Since the same clean target is reached from many different damaged versions, the features are stable under small changes.

Can a denoising autoencoder fill in missing values?

Yes. Training with masked cells is the same task as imputation, so a trained model can fill real gaps by reading the observed cells and predicting the rest. For reliable statistics it is best to produce several plausible completions, not one, so that uncertainty is not hidden. Methods such as MIDAS follow this approach.

How does it relate to masked language models and masked image models?

Masked modelling is a denoising autoencoder where the corruption is masking and the loss falls on the masked parts. BERT-style language models mask tokens and masked autoencoders for images mask patches. The core idea of learning by restoring hidden parts of the input is the same, only the architecture and the scale differ.

Is a denoising autoencoder a generative model?

Not in the basic form, but it can be used generatively. The relation to score matching shows that the denoiser learns the direction toward higher data density, and sampling procedures can alternate between corrupting and denoising. Diffusion models extend this idea by denoising at many noise levels and are strong generative models.

How much noise should I add?

It depends on the data. A masking rate between 10 and 50 percent is a common starting range for tables. The right level is the one at which the task is hard but still solvable. Test a few values on a validation set that simulates the real damage, and consider varying the rate between batches.

Can it detect anomalies?

Yes. A model trained on normal data restores normal rows well and abnormal rows poorly, so the reconstruction error acts as an anomaly score. Per-cell errors show which columns look wrong. The method works best when the training data are mostly clean, and the threshold should be chosen on labelled or reviewed examples when possible.

How do denoising autoencoders relate to tabular foundation models?

Tabular foundation models are pre-trained on many tables with objectives such as masked cell prediction, which is the denoising autoencoder idea applied to tables. This gives them imputation, robust representations and the ability to condition on visible cells to predict hidden ones, including labels, all without needing column names.

Glossary

• Autoencoder: a network that learns to reproduce its input through an internal code.
• Denoising autoencoder: an autoencoder trained to restore clean input from a corrupted copy.
• Encoder: the part of the network that maps input to a code.
• Decoder: the part of the network that maps a code back to the input space.
• Corruption: the random damage applied to the input during training.
• Masking noise: corruption that sets a random share of values to zero or a mask token.
• Swap noise: corruption that replaces values with values from other rows in the same column.
• Reconstruction loss: the measure of difference between the output and the clean input.
• Overcomplete: having a code larger than the input.
• Manifold: a low-dimensional surface near which real data lie.
• Score: the gradient of the log density of the data.
• Imputation: filling in missing values.
• Multiple imputation: creating several plausible completions to reflect uncertainty.
• Self-supervised learning: learning from labels made out of the data itself.
• Representation: the internal feature vector for an input.
• Missing at random: missingness that depends only on observed values.

Summary

A denoising autoencoder learns by restoring clean data from a damaged copy. That one change to the plain autoencoder removes the trap of copying, allows wide codes, and yields robust features and a strong imputation skill. The main choices are the corruption type, the corruption rate, the loss on observed cells and the architecture.

For tables, the method is a natural fit because real data have missing cells, noise and mixed types. It underlies masked cell modelling, imputation networks such as MIDAS, anomaly scoring by reconstruction error, and the pre-training of tabular foundation models that learn from raw rows and columns without needing names. Scale the data, compute the loss on observed cells only, match the corruption to real damage, and validate by hiding known cells. For related ideas, see autoencoders, masked cell modeling, imputation networks, /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Self-supervised for tables in my stack, under Foundation models and applied ML. Others in the same group: