Autoencoders

Overview

An autoencoder is a neural network trained to copy its input to its output through a narrow middle layer. The first half, the encoder, squeezes the input into a short code. The second half, the decoder, rebuilds the input from that code. Because the code is smaller than the input, the network cannot simply memorise everything. It must learn which patterns matter.

This simple recipe is useful in many ways. The code can serve as a compact feature vector for other models. The reconstruction can be a cleaned-up version of a noisy input. The reconstruction error can act as an anomaly score, because inputs unlike the training data are rebuilt poorly. Autoencoders need no labels, so they are a basic tool of self-supervised learning. They also lead directly to Denoising autoencoders, the VAE and the VQ-VAE.

What it is

The basic structure

An autoencoder has three parts:

• An encoder that maps the input x to a code z.
• A bottleneck, the code z itself, with fewer dimensions than x or with some other constraint.
• A decoder that maps z back to a reconstruction x-hat.

Training adjusts the weights so that x-hat is as close to x as possible. The loss is the difference between them, such as the mean squared error for numbers or cross entropy for categories.

An everyday picture

Imagine describing a face over the phone so someone else can sketch it. You can only use twenty words. You would not list every pore. You would say "round face, wide eyes, short dark hair, small scar on the left cheek". Those twenty words are the code. The sketch artist is the decoder. If the sketch looks like the person, your twenty words captured what matters.

A good autoencoder learns to pick the right twenty words on its own.

Why the bottleneck matters

Without a constraint, a network could learn the identity function and pass the input straight through. That gives zero error and no useful code. The bottleneck forces compression. There are several ways to apply it:

• Make the code smaller than the input (undercomplete).
• Add a penalty so most code units stay near zero (sparse).
• Corrupt the input and ask for the clean version (denoising).
• Penalise the sensitivity of the code to small input changes (contractive).
• Force the code to follow a known distribution (variational).

What it is not

• It is not a classifier. It has no labels by default.
• It is not guaranteed to find meaningful features. The code only has to be useful for reconstruction.
• A linear autoencoder with squared error learns the same subspace as principal component analysis. Nonlinear ones can go further.

Where it came from

Early ideas

In the 1980s, researchers trained networks to reproduce their input through a narrow layer, often called auto-association. The goal was to learn internal representations without a teacher. Their link to principal component analysis was noted early.

Deep autoencoders

In 2006, a well-known paper showed that deep autoencoders could be trained by pre-training layer by layer, reaching lower reconstruction error than principal component analysis on images and documents. This helped restart interest in deep learning.

Denoising and sparse variants

In 2008, denoising autoencoders were introduced: corrupt the input and train the network to recover the clean version. This forces the model to learn structure in the data instead of copying. Sparse and contractive variants followed around the same time.

Probabilistic and discrete variants

In 2013, the variational autoencoder combined the autoencoder idea with probabilistic inference. It learns a distribution over codes, so you can sample new data. Later, the VQ-VAE used a discrete codebook for the latent space and became a building block for image and audio generation. See VAE and VQ-VAE.

Masked autoencoders

More recently, masked autoencoders hide parts of the input and reconstruct them. In vision this is the MAE. In language it is the idea behind masked language models. In tables it appears as Masked cell modeling. The same idea, reconstruct what was hidden, powers many pre-training methods.

How it works

Step 1: encode

The encoder is a neural network. For a table row, it is often a stack of dense layers that shrink the width. For images it is convolutional. For sequences it may be recurrent or attention-based. The last encoder layer outputs the code z.

Step 2: choose the bottleneck

Step 3: decode

The decoder is usually a mirror image of the encoder. It expands the code back to the input size. The final layer matches the data type: a linear layer for real numbers, a sigmoid for values in a range, a softmax for categories.

Step 4: compute the loss

The reconstruction loss compares x-hat with x.

• Squared error for continuous values.
• Cross entropy for binary or categorical values.
• A mix, summed over feature groups, for mixed data.

Extra terms can be added, such as a sparsity penalty or a KL divergence.

Step 5: train

Training uses stochastic gradient descent with backpropagation. Because the target is the input itself, no labels are needed. A typical run uses Adam, mini-batches, and early stopping on a validation set.

Step 6: use the pieces

After training, you can use each part on its own.

• The encoder alone yields features for other models.
• The decoder alone, in a variational model, generates new samples.
• Both together reconstruct and denoise.
• The error between input and reconstruction flags unusual inputs.

A simple analogy for the whole loop

A student reads a chapter and writes a one-page summary. A second student, who has not read the chapter, tries to rebuild the key facts from that summary alone. If the second student succeeds, the summary was good. Over many chapters, the first student learns what to put in a summary, and the second learns how to unpack it.

Autoencoders and principal component analysis

Key concepts and terms

The manifold idea

Most real data do not fill the whole space of possible inputs. A table of body measurements, for example, has a few underlying factors, such as overall size, build and age, behind dozens of columns. The data lie near a low-dimensional surface in a high-dimensional space. An autoencoder learns coordinates on that surface. Points off the surface get large errors, which is why reconstruction error finds outliers.

Architectures by data type

Training diagnostics

• Plot training and validation loss. A growing gap means overfitting.
• Plot error per feature. A column with a very high error may need a better encoding.
• Plot the distribution of per-row error on normal data. It should be tight, with a thin tail.
• Look at the variance of each code unit. Units with almost no variance are unused, which suggests the code is too wide.
• Reconstruct a few rows by hand and compare them with the originals.

Beyond the basics

Stacked and tied designs

Stacked autoencoders place several layers in the encoder and a mirrored set in the decoder. With tied weights, the decoder uses the transpose of the encoder matrices, which halves the parameter count and acts as a mild regulariser.

Sparse codes

A sparsity penalty on the code, such as an L1 term or a limit on average activation, makes only a few units active for each row. This gives features that are easier to inspect.

Convolutional and sequence variants

For images, convolutional layers keep spatial structure. For logs and sensor data, a sequence autoencoder reads a window and rebuilds it, and the code summarises the window.

Multi-task use

An autoencoder can share its encoder with a supervised head. The reconstruction term then acts as a regulariser that keeps the code informative about the whole input, not only about the label.

A worked example

A numeric walk-through

Take a tiny linear autoencoder with two inputs and one code unit. Suppose the data lie on a line: whenever the first feature is x, the second is about 2x. Examples: (1, 2), (2, 4), (3, 6).

Let the encoder be z = 0.447 * x1 + 0.894 * x2 (a unit direction along the line) and the decoder be x-hat = z * (0.447, 0.894).

The first three points lie on the line and are rebuilt perfectly from one number. The last point is off the line. The code keeps only the part along the line, so the reconstruction lands at (1, 2) and the error is large. A threshold on that error would flag the last point as an anomaly.

A dense autoencoder in PyTorch for tabular rows

import torch
import torch.nn as nn

class TabularAE(nn.Module):
    def __init__(self, n_features=30, code=6):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(n_features, 32), nn.ReLU(),
            nn.Linear(32, 16), nn.ReLU(),
            nn.Linear(16, code),
        )
        self.decoder = nn.Sequential(
            nn.Linear(code, 16), nn.ReLU(),
            nn.Linear(16, 32), nn.ReLU(),
            nn.Linear(32, n_features),
        )

    def forward(self, x):
        z = self.encoder(x)
        return self.decoder(z), z

model = TabularAE()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
X = torch.randn(2048, 30)                       # placeholder for standardised numeric data

for epoch in range(50):
    perm = torch.randperm(len(X))
    for i in range(0, len(X), 128):
        batch = X[perm[i:i + 128]]
        recon, _ = model(batch)
        loss = ((recon - batch) ** 2).mean()
        opt.zero_grad(); loss.backward(); opt.step()

Scoring anomalies

with torch.no_grad():
    recon, _ = model(X_new)
    err = ((recon - X_new) ** 2).mean(dim=1)       # one score per row

threshold = torch.quantile(err_val, 0.99)          # chosen on normal validation data
flags = err > threshold

Denoising version

noise = 0.1 * torch.randn_like(batch)
recon, _ = model(batch + noise)
loss = ((recon - batch) ** 2).mean()               # compare with the CLEAN input

Handling mixed types and missing values

For a table with numeric and categorical columns, a common recipe is:

• Standardise numeric columns.
• Embed or one-hot encode categorical columns.
• Add a mask channel that marks missing cells.
• Use a mean squared error on numeric outputs and cross entropy on categorical outputs.
• Multiply each loss term by the observed mask so that missing cells are ignored.

def masked_loss(recon_num, x_num, mask_num, logits_cat, y_cat, mask_cat):
    mse = (((recon_num - x_num) ** 2) * mask_num).sum() / mask_num.sum().clamp(min=1)
    ce = torch.nn.functional.cross_entropy(logits_cat, y_cat, reduction="none")
    ce = (ce * mask_cat).sum() / mask_cat.sum().clamp(min=1)
    return mse + ce

Strengths and limitations

Strengths

• Needs no labels.
• Flexible: works for images, text, audio, tables and graphs.
• Learns nonlinear compression, which can beat linear methods.
• Gives reusable features for other models.
• Reconstruction error is a simple anomaly score.
• Denoising variants improve robustness.
• Can be extended to generation with variational and discrete variants.
• Easy to implement and train.

Limitations

• The code is only guaranteed to be good for reconstruction, not for any downstream task.
• It can learn to reconstruct anomalies too, if the bottleneck is large or the model is trained on contaminated data.
• Reconstruction loss weights all features equally by default, so scale and type choices matter.
• Plain autoencoders do not produce a smooth latent space for sampling.
• It can blur fine details, especially with squared error.
• Needs tuning of bottleneck size, depth and regularisation.
• Mixed types and missing values need careful handling.
• On small tabular data, simpler methods such as PCA or tree ensembles may do as well.

The identity trap

If the bottleneck is wide, or the network is powerful and trained long, it can learn a near-identity mapping. Symptoms are tiny reconstruction error on everything, including anomalies. Remedies are a narrower code, noise, dropout, weight decay or a sparsity penalty.

How it compares with alternatives

Autoencoder versus contrastive learning

An autoencoder learns by reconstruction. Contrastive learning learns by pulling similar views together and pushing different ones apart. Reconstruction keeps fine detail, including noise. Contrastive methods tend to give features that are more useful for classification but need good augmentations.

Autoencoder versus masked modelling

A masked autoencoder is an autoencoder in which the corruption is the removal of parts of the input. It trains the model to predict the missing parts from the visible ones. This is closer to what is needed for imputation and for pre-training large models.

Role in tabular foundation models

Why autoencoders matter for tables

Tabular foundation models learn from raw tables: row and column structure, mixed numeric and categorical types, missing values and often no column names. Reconstruction is one of the oldest and most natural ways to learn from such tables without labels. Many ideas in modern table models, such as masked cell prediction, are autoencoders under another name.

Row and column structure

A table row can be fed to an autoencoder as a vector, where each column gets its own slot. The code then summarises the whole row. A column-wise view is also possible: transpose the table and encode each column from its values. Modern table models often do both, with attention across rows and across columns, but the autoencoding objective, rebuild the hidden part from the visible part, stays the same. See FT-Transformer and TabPFN for attention-based designs.

Mixed types

Each type needs a suitable encoder and decoder head:

• Numeric: a small network that maps a number to a vector, and a regression head to read it back.
• Categorical: an embedding table, and a softmax head over categories.
• Binary: a single logit.
• Dates and text: converted into numeric or token features first.

The loss sums the terms across types, and care is needed so that no type dominates.

Missing values

Missingness is the best match for autoencoders. A denoising autoencoder trained with random masking learns to fill in cells from the rest of the row. At test time, real missing cells are treated like masked ones. This is the basis of many imputation networks. See Imputation networks and Masked cell modeling. It is important to include missingness patterns in training, since in real data cells are rarely missing at random.

No column names

If column names are missing, the model has to learn the meaning of a column from its values and its relation to other columns. A column-wise autoencoder, which encodes the distribution of a column into a vector, can supply a name-free column embedding. Two columns with similar distributions get similar codes. This helps transfer across tables with different schemas.

Pre-training on synthetic and real tables

Pre-training for tabular foundation models uses a very large number of tables, often synthetic ones drawn from random generators, plus real tables. The self-supervised objective can be reconstruction of masked cells. Because the task needs no labels, any table can be used. Synthetic generators can produce unlimited tables with known structure. See Synthetic data generation.

In-context learning

In-context learning means predicting on new rows from labelled example rows shown in the input, without weight updates. The pretraining task is often a supervised one built from synthetic tables. Autoencoders contribute an adjacent skill: a good row representation. A model that has learned to reconstruct rows tends to have a row code that clusters similar rows, which helps when examples are used as context. In some designs a reconstruction loss is added as an auxiliary task.

Continual learning

A learned code can be updated as new data arrive. A frozen encoder gives stable features, and only the decoder or the head is updated. Alternatively, a replay buffer of past rows is reconstructed during updates to limit forgetting. A drift in reconstruction error is also a useful signal that the data have changed and the model should be refreshed.

Anomaly detection

This is the most direct use. Train on normal rows, then score new rows by reconstruction error. Variants use the distance to the nearest code cluster or the likelihood in the latent space. See Anomaly detection. Reconstruction can be combined with simple tabular detectors such as Isolation forest for a stronger ensemble. Per-feature error also points to which column looks unusual, which helps explain an alert.

Sector and semantic classification

Deciding what industry a table or company belongs to, or what a column means, starts with a good representation. The code of an autoencoder over a column's values or over a table's summary statistics is a compact input for a classifier. See Semantic sector heads. For how such parts fit into complete models, see /model/architecture, /model/research and /model/benchmarks.

An honest summary of the role

Autoencoders are less often the final model in a tabular foundation system. Their value is as a training principle and a building block: reconstruct hidden cells, learn compact row and column codes, flag unusual inputs. On small tables, tree ensembles often beat autoencoder features for supervised prediction, so use autoencoders where labels are scarce, missing values are common or anomaly scores are the goal.

Practical guidance and best practices

Choosing a variant

• Use a plain undercomplete autoencoder for quick compression and anomaly scoring.
• Use a denoising autoencoder for imputation and robustness.
• Use a variational autoencoder if you need sampling or a smooth latent space.
• Use masked modelling with attention for large-scale pre-training.

Sizing the model

• Start with a code size of roughly 10 to 25 percent of the input width for tabular data.
• Use two to four hidden layers. Deeper is rarely needed for tables.
• Increase width before depth.
• Use dropout or weight decay if the model overfits.

Preparing tabular data

• Standardise numeric columns, or use a quantile transform for skewed ones.
• Encode categories with embeddings when they have many levels.
• Keep a missing mask and use it in the loss.
• Remove or cap extreme outliers if the goal is a model of normal behaviour.
• Split by time or group if rows are related, to avoid leakage.

Training habits

• Train on normal data only for anomaly detection.
• Use early stopping on validation reconstruction error.
• Use a learning rate schedule and Adam.
• Log per-feature error to see which columns the model finds hard.

Setting an anomaly threshold

• Use a high percentile of validation error, such as the 99th.
• Check precision and recall against any known incidents.
• Re-calibrate regularly if the data drift.
• Combine with a rule-based filter for known cases.

Evaluating the code

• Train a simple classifier on the code and compare with raw features.
• Visualise the code with UMAP to check for structure.
• Check that nearby codes correspond to similar rows.

A short decision table

Common mistakes

• Making the bottleneck too wide and learning the identity function.
• Training on data that include many anomalies and then expecting them to stand out.
• Using the raw mean squared error on unscaled features, so large-scale columns dominate.
• Treating missing values as zeros without a mask.
• Mixing one-hot categories with numeric columns under the same loss without weighting.
• Choosing a threshold on training data, not on held-out normal data.
• Interpreting the latent code as if each unit had a single meaning.
• Reporting reconstruction error as accuracy.
• Ignoring row leakage between training and validation.
• Using a deep autoencoder on a small table where PCA already works.

Frequently asked questions

What is an autoencoder in simple terms?

An autoencoder is a neural network that learns to compress its input into a short code and then rebuild the input from that code. It is trained to make the rebuilt version match the original. Because the code is small, the network must learn the most important patterns in the data.

What are autoencoders used for?

They are used for dimensionality reduction, feature learning, denoising, anomaly detection, imputation of missing values and, in variational form, generating new data. Since they need no labels, they are common when labelled data are scarce.

How does an autoencoder detect anomalies?

It is trained on normal data, so it learns to reconstruct normal patterns well. An input that differs from the training data is rebuilt poorly, and the reconstruction error is large. A threshold on that error flags the input as unusual.

What is the difference between an autoencoder and a variational autoencoder?

A plain autoencoder maps each input to a single code, while a variational autoencoder maps it to a distribution over codes and adds a penalty that keeps those distributions close to a simple prior. The result is a smooth latent space from which you can sample to generate new examples.

What is a denoising autoencoder?

It is an autoencoder trained to recover the clean input from a corrupted copy, for example one with added noise or randomly hidden values. This stops it from copying the input and makes it learn structure in the data. It is a natural tool for cleaning and for filling in missing values.

Can autoencoders work on tabular data?

Yes. A dense autoencoder can take a row of standardised numbers and encoded categories as input. With a mask for missing values and a loss that fits each column type, it supports compression, anomaly scoring and imputation. For supervised prediction on small tables, tree ensembles are often stronger.

How do I choose the size of the bottleneck?

Start with a code that is a fraction of the input width, and tune it on validation reconstruction error and on the quality of any downstream task. If the code is too wide, the model may copy the input. If it is too narrow, the reconstruction loses useful detail.

Is an autoencoder the same as PCA?

A linear autoencoder with squared error learns the same subspace as principal component analysis. With nonlinear activations and several layers, an autoencoder can capture curved structure that PCA cannot. PCA has a closed-form solution, whereas an autoencoder is trained by gradient descent.

How do autoencoders relate to masked modelling?

A masked model is an autoencoder in which the corruption is hiding parts of the input and the target is to predict those parts. Masked image, language and table models all follow this pattern. The core idea of reconstructing what is hidden from what is visible is the same.

Do autoencoders need a lot of data?

They need enough data to learn the main patterns, but they are not as data hungry as large supervised models. For small tables, a small autoencoder with strong regularisation works. For very high-dimensional data such as images, more data and larger models help.

Glossary

• Anomaly score: a number that says how unusual an input is, often the reconstruction error.
• Bottleneck: the narrow middle layer that forces compression.
• Code: the compressed vector produced by the encoder.
• Contractive penalty: a term that makes the code less sensitive to small input changes.
• Decoder: the half of the network that rebuilds the input from the code.
• Denoising: training to recover clean input from a corrupted version.
• Encoder: the half of the network that compresses the input.
• KL divergence: a measure of the gap between two distributions, used in variational models.
• Latent space: the space of codes.
• Manifold: a low-dimensional surface near which data lie.
• Mask: a marker that shows which cells are observed or hidden.
• Reconstruction loss: the training loss that compares output with input.
• Sparse autoencoder: an autoencoder that keeps most code units close to zero.
• Tied weights: decoder weights that mirror the encoder weights.
• Undercomplete: having a code smaller than the input.
• VQ-VAE: a variational-style autoencoder with a discrete codebook.

Summary

Autoencoders learn by compressing an input into a short code and rebuilding it. The bottleneck forces them to keep only the most important structure. Variants add noise, sparsity, probability or discrete codes, and each serves different goals such as denoising, generation or tokenising data. The reconstruction error gives a simple and widely used anomaly score.

For tabular foundation models, the autoencoding principle is central: reconstruct masked cells, learn row and column codes without labels, handle missing values naturally and flag unusual rows. The final model is often attention based, but the learning signal is autoencoder-like. Related reading: Denoising autoencoders, VAE, Anomaly detection, /model/architecture, /model/research and /model/benchmarks.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: