Diffusion models

Overview

Diffusion models are generative models that learn to turn random noise into realistic data by removing noise one small step at a time. During training, real data is gradually corrupted with noise, and a neural network learns to undo each small corruption. During generation, the model starts from pure noise and applies the learned denoising steps until a clean sample appears.

This simple idea powers modern image, audio and video generators, and it is increasingly used for structured data such as tables. For tabular work, diffusion models matter in two ways. They can create synthetic rows that follow the patterns of a real table, and their denoising objective is close to the masked prediction used to learn imputation. See DDPM and DDIM for the two best-known formulations.

What it is

A diffusion model has two processes. The forward process is fixed and has no learning: it adds a little Gaussian noise to a data point at each step until nothing but noise is left. The reverse process is learned: a network predicts how to remove noise at each step, so that sampling can run the chain backwards from noise to data.

The core promise

Direct generation from noise in one step is hard, because the model must jump from a shapeless cloud to a detailed sample. Diffusion breaks the jump into many tiny steps. Each step only needs to make a small improvement, which is a much easier learning problem.

What the network predicts

Different formulations train the network to predict different targets, which are mathematically related:

• The noise that was added.

• The clean data itself.

• A velocity that mixes the two.

• The score, which is the gradient of the log data density.

Conditioning

A diffusion model can be conditioned on extra information, such as a text prompt, a class label, an image or a table row's known values. Conditioning steers sampling toward outputs that match the condition.

Where it came from

Early work in the 2010s framed generation as reversing a gradual noising process, taking inspiration from non-equilibrium physics. These ideas were interesting but did not match the quality of GANs at the time.

A parallel line of work on score matching learned the gradient of the data density at several noise levels and sampled with Langevin dynamics. The two lines merged in the early 2020s. The paper "Denoising Diffusion Probabilistic Models" appeared in 2020 and showed that a simple noise-prediction objective gives high-quality images. A related paper on faster deterministic sampling led to DDIM, and a score-based view with stochastic differential equations unified the pictures.

Latent diffusion moved the process from pixels into the compact latent space of an autoencoder, which cut cost and made large-scale text-to-image systems such as Stable Diffusion practical. Adapters such as ControlNet added spatial control. Later work introduced flow-matching and consistency ideas to reduce the number of sampling steps. For tabular data, methods that adapt diffusion to numeric and categorical columns appeared in the following years. See /model/research for how generative ideas relate to tabular research.

How it works

Picture a photograph slowly dissolving in a bowl of water. At first you can still see the picture, then only a blur, then a uniform grey. Now imagine a skilled restorer who has watched this happen thousands of times. Given a slightly blurred picture, the restorer can guess how it looked a moment earlier. Chain many such guesses together and you can start from grey and arrive at a picture.

Step 1: The forward (noising) process

Start with a data point x0. At each step t, add a bit of Gaussian noise:

q( x_t | x_{t-1} ) = Normal( sqrt(1 - beta_t) * x_{t-1}, beta_t * I )

Here beta_t is a small number from the noise schedule. After T steps, x_T is nearly pure noise.

Step 2: A closed form for any step

Because sums of Gaussians are Gaussian, you can jump straight to any step t:

x_t = sqrt(alpha_bar_t) * x0 + sqrt(1 - alpha_bar_t) * eps

where eps is standard Gaussian noise and alpha_bar_t is the running product of (1 - beta_s). This formula makes training cheap: pick a random t, add the right amount of noise in one shot and ask the network to undo it.

Step 3: The noise schedule

The schedule decides how fast signal is destroyed. A linear schedule raises beta steadily. A cosine schedule keeps more signal early and avoids wasting steps on nearly pure noise. The schedule affects both quality and the number of useful steps.

Step 4: Train the denoiser

For each training example:

1. Sample a time step t at random.

2. Sample noise eps.

3. Build x_t using the closed form.

4. Ask the network to predict eps from x_t and t.

5. Minimise the mean squared error between predicted and true noise.

loss = || eps - eps_theta( x_t, t ) ||^2

The network gets the time step as input, often through a sinusoidal embedding, so it knows how noisy the input is.

Step 5: Sample by reversing the chain

To generate, start with x_T drawn from a standard Gaussian. For t from T down to 1, use the network's noise prediction to compute a slightly cleaner x_{t-1}. In the stochastic version, add a small amount of fresh noise at each step. After the last step, x0 is the sample.

Analogy: sculpting from a rough block. Early steps carve large shapes. Late steps polish fine detail.

Step 6: Choose a sampler

• Ancestral (DDPM) sampling is stochastic and needs many steps.

• Deterministic samplers such as DDIM can skip steps and run in tens of steps.

• Higher-order solvers treat the reverse process as an ordinary differential equation and take larger, more accurate steps.

• Distilled or consistency-based models can generate in a few steps, at some cost in diversity.

Step 7: Guidance

Classifier-free guidance trains the model both with and without the condition, by randomly dropping the condition during training. At sampling time, the model combines the two predictions:

eps_guided = eps_uncond + w * ( eps_cond - eps_uncond )

A larger w makes outputs follow the condition more strongly, at the cost of diversity.

Step 8: Network architecture

For images, the denoiser is often a U-Net with attention, or a transformer over patches. For audio, one-dimensional versions are used. For tables, the denoiser is usually a multilayer network or a transformer over columns, with embeddings for the time step.

Latent diffusion in one paragraph

Instead of diffusing raw pixels, an autoencoder first compresses the image to a small latent grid. The diffusion model works in that latent space, which is far cheaper. A decoder turns the final latent into an image. This is how Stable Diffusion reaches high resolution at reasonable cost. The autoencoder is a relative of the VAE and the VQ-VAE.

Key concepts and terms

A worked example

A numeric walk-through of the forward process

Suppose a single data value x0 = 2.0. Choose three steps with alpha_bar values of 0.9, 0.5 and 0.1.

1. At step 1: x_1 = sqrt(0.9) * 2.0 + sqrt(0.1) * eps, which is about 1.90 + 0.32 * eps.

2. At step 2: x_2 = sqrt(0.5) * 2.0 + sqrt(0.5) * eps, about 1.41 + 0.71 * eps.

3. At step 3: x_3 = sqrt(0.1) * 2.0 + sqrt(0.9) * eps, about 0.63 + 0.95 * eps.

The signal shrinks and the noise grows. At step 3, the original value contributes little. The denoiser's job at each step is to guess eps, so that it can estimate how much of x_t is signal.

A minimal training loop for one-dimensional data

import torch
import torch.nn as nn

T = 200
betas = torch.linspace(1e-4, 0.02, T)
alphas = 1.0 - betas
alpha_bar = torch.cumprod(alphas, dim=0)

class Denoiser(nn.Module):
    def __init__(self, dim, hid=128):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(dim + 1, hid), nn.SiLU(),
            nn.Linear(hid, hid), nn.SiLU(),
            nn.Linear(hid, dim),
        )

    def forward(self, x, t):
        t_feat = (t.float() / T).unsqueeze(1)
        return self.net(torch.cat([x, t_feat], dim=1))

model = Denoiser(dim=2)
opt = torch.optim.Adam(model.parameters(), lr=1e-3)

def train_step(x0):
    t = torch.randint(0, T, (x0.size(0),))
    eps = torch.randn_like(x0)
    ab = alpha_bar[t].unsqueeze(1)
    xt = ab.sqrt() * x0 + (1 - ab).sqrt() * eps
    pred = model(xt, t)
    loss = ((pred - eps) ** 2).mean()
    opt.zero_grad()
    loss.backward()
    opt.step()
    return loss.item()

This works for any two-dimensional data, for example two numeric columns of a table.

Sampling from the trained model

@torch.no_grad()
def sample(n, dim=2):
    x = torch.randn(n, dim)
    for t in reversed(range(T)):
        tt = torch.full((n,), t)
        eps = model(x, tt)
        a, ab, b = alphas[t], alpha_bar[t], betas[t]
        mean = (x - b / (1 - ab).sqrt() * eps) / a.sqrt()
        noise = torch.randn_like(x) if t > 0 else 0.0
        x = mean + b.sqrt() * noise
    return x

Each loop iteration removes a little noise. After T steps, the samples should resemble the training distribution.

A tabular sketch

For a table with numeric and categorical columns, a common recipe handles each type differently. Numeric columns are scaled and diffused with Gaussian noise. Categorical columns are either diffused with a discrete corruption process, such as random replacement of categories, or embedded and treated as continuous.

import pandas as pd
from sklearn.preprocessing import QuantileTransformer

df = pd.read_parquet("customers.parquet")         # placeholder file
num_cols = ["age", "income", "tenure_days"]
cat_cols = ["segment", "region"]

qt = QuantileTransformer(output_distribution="normal")
X_num = qt.fit_transform(df[num_cols])

# One-hot for categories, to be diffused with a discrete process,
# or embedded and diffused as continuous vectors.
X_cat = pd.get_dummies(df[cat_cols]).to_numpy(dtype="float32")

Quantile scaling to a normal shape suits Gaussian noise. Categories need either a discrete process or a careful embedding. After training, sample rows, invert the scaling and take the most likely category per column.

Checking synthetic table quality

# Compare real and synthetic columns.
for c in num_cols:
    print(c, df[c].mean(), synth[c].mean(), df[c].std(), synth[c].std())

# Compare pairwise relationships.
print(df[num_cols].corr() - synth[num_cols].corr())

# Train on synthetic, test on real.

A good synthetic table matches single-column shapes, pairwise relationships and downstream model performance. It should also not copy real rows.

Which target to predict

Inpainting loop for a table row

# known_mask: 1 for observed cells, 0 for missing cells
x = torch.randn_like(row)
for t in reversed(range(T)):
    eps = model(x, torch.full((1,), t))
    x = denoise_step(x, eps, t)                  # placeholder reverse step
    # Re-impose the known cells at the current noise level.
    noisy_known = add_noise(row, t)              # placeholder forward step
    x = known_mask * noisy_known + (1 - known_mask) * x

At each step the observed cells are reset to a properly noised copy of the truth. Only the missing cells evolve freely, so the final row agrees with the observed values and fills the gaps in a consistent way. Running the loop several times gives several plausible imputations.

Strengths and limitations

Strengths

• High sample quality and good coverage of the data distribution.

• Stable training with a simple regression loss, unlike adversarial training in GANs.

• Flexible conditioning, including text, labels, images and partial rows.

• Natural support for editing and filling in missing parts, since the denoiser can be asked to repair part of a sample.

• Scales well with data and model size.

• Works across modalities, including images, audio, video, molecules and tables.

Limitations

• Slow sampling, because many network evaluations are needed. Fast samplers and distillation reduce this.

• Training can be expensive at high resolution.

• Likelihood estimates are harder to get than in normalizing flows.

• Discrete data, such as categories and text, needs special treatment.

• Risk of memorising training samples if the model is too large for the data.

• Evaluation is hard. Metrics capture only part of what makes samples good.

• Control can be indirect, and guidance trades diversity for fidelity.

How it compares with alternatives

Diffusion and flow-based methods are closely related. Both define a path between noise and data. Diffusion uses a stochastic path with a score or noise target, while flow matching learns a deterministic velocity field. Related ideas such as denoising autoencoders and masked cell modeling share the "repair a corrupted input" objective.

Role in tabular foundation models

Tabular foundation models learn from raw tables: rows and columns with mixed numeric, categorical and date types, missing cells and often no useful column names. They are pre-trained on large collections of synthetic or real tables and then predict through in-context learning or light adaptation. Diffusion models connect to this world in several concrete ways.

Synthetic tables for pre-training

Pre-training needs many diverse tables. Real tables are limited by privacy and by cost. Synthetic tables can come from random structural models, and in principle from generative models such as diffusion models trained on real tables. A model like TabPFN is pre-trained on synthetic datasets, and the quality and variety of those datasets shape what the model learns. See synthetic data generation for the wider picture, and /model/architecture for how pre-training fits the design of such models.

Privacy-preserving data sharing

A diffusion model trained on a sensitive table can produce new rows that follow the same distribution without exposing real records. This helps teams share data for development and testing. It does not remove all privacy risk. Memorisation tests and distance checks to the nearest real record are needed, and formal privacy methods can be added during training.

Imputation as inpainting

In images, inpainting fills a masked region. The same idea applies to tables: keep the known cells fixed, noise the unknown cells, and denoise them conditioned on the known ones. This is a principled way to impute missing values because it samples from the conditional distribution, so it can also express uncertainty by drawing several fills. It is related to masked cell modeling and imputation networks, and compares with simple approaches such as MIDAS. For related tooling, see denoising autoencoders.

Mixed types

A table has numeric columns that suit Gaussian noise and categorical columns that do not. Tabular diffusion models handle this in a few ways: multinomial or masking-based diffusion for categories, continuous embeddings for categories, or separate processes for each type that share a denoiser. Missing cells can be treated as a special category or as a mask that the model learns to fill. None of these approaches needs column names, since columns are identified by position and by learned embeddings.

Denoising as a learning signal

A tabular foundation model can use a diffusion-style loss as a self-supervised objective: corrupt random cells, ask the model to restore them and learn rich features along the way. This mirrors self-supervised learning in vision and language. The learned representation can then support classification, regression, imputation and anomaly detection.

Anomaly detection

If a model has learned the distribution of normal rows, rows that are hard to denoise or reconstruct are unusual. Reconstruction error after adding and removing noise is a simple anomaly score. It plays the same role as in autoencoders, with a richer model of the data. See anomaly detection for the broader framing.

Sector and semantic classification

Pooled representations from a denoiser can feed a head that predicts the domain of a table, which relates to semantic sector heads. Conditioning a generator on a sector label is also possible, so that synthetic tables look like finance, retail or logistics data.

Continual learning

When new data arrives, a generative model can replay samples of old tasks instead of storing real rows. This is called generative replay. It connects diffusion models to experience replay and to methods that limit forgetting such as EWC. How such ideas are measured is discussed at /model/benchmarks.

In-context learning versus generation

A tabular foundation model that predicts through in-context learning gives answers directly from labelled context rows. A diffusion model generates samples by iterative denoising. They answer different questions. The first predicts a target given features. The second models the joint distribution of all columns. In practice, they can be combined: generate extra rows to balance classes, then use a foundation model for prediction.

Practical guidance and best practices

Data preparation

• Scale numeric columns, often with a quantile transform to a normal shape.

• Handle heavy tails and outliers before training.

• Decide how to encode categories and missing cells.

• Split into train and test before any fitting of scalers.

Model and training choices

• Start with a small denoiser and a modest number of steps.

• Use a cosine schedule or a schedule tuned for your data scale.

• Use an exponential moving average of the weights for sampling.

• Condition on the time step through an embedding.

• Train long enough for the loss to plateau, and check samples regularly.

Sampling

• Use a deterministic fast sampler for most uses.

• Increase the number of steps if samples look noisy.

• Use classifier-free guidance sparingly, since it reduces diversity.

• Draw many samples and filter by simple validity rules, such as value ranges.

Evaluation for tables

• Compare column distributions.

• Compare pairwise correlations and dependence between categories.

• Train a downstream model on synthetic data and test on real data.

• Check nearest-neighbour distances between synthetic and real rows to detect copying.

• Review rare categories and rare combinations.

Governance

• Document what the synthetic data is for and where it must not be used.

• Keep the training data and the model under the same access controls as the original data.

• Re-evaluate when the source data changes.

A quick decision guide

Budgeting compute

• Training cost grows with data size, model size and number of epochs.

• Sampling cost grows linearly with the number of steps.

• Halving the steps halves the sampling time, but check the effect on quality.

• For tables, models are small and training is usually cheap compared with images.

Common mistakes

• Judging samples by eye only, without distribution checks.

• Ignoring categorical structure and treating categories as ordered numbers.

• Training on unscaled data with extreme outliers, which dominates the noise scale.

• Assuming synthetic data is automatically private.

• Using too few sampling steps and blaming the model for noisy outputs.

• Over-using guidance, which collapses diversity.

• Fitting scalers on all data, including the test set.

• Forgetting to invert preprocessing before comparing with real data.

• Expecting a generator to replace a predictive model, when the tasks differ.

Frequently asked questions

What is a diffusion model in simple terms?

A diffusion model is a generator that learns to remove noise. During training, real examples are covered in increasing amounts of noise, and a network learns to undo each small step. To create something new, the model starts from random noise and repeatedly cleans it until a realistic sample appears.

Why do diffusion models need many steps?

Each step only removes a small amount of noise, because small steps are easy to learn and keep the reverse process accurate. Many small steps add up to a large change. Faster samplers, distillation and flow-based training can reduce the number of steps, usually with some trade-off in quality or diversity.

How is a diffusion model different from a GAN?

A GAN trains a generator against a discriminator and can be unstable, while a diffusion model trains with a simple regression loss and is generally more stable. GANs generate in one pass, so they are fast, but they can miss parts of the data distribution. Diffusion models cover the distribution better but sample more slowly.

What is the noise schedule?

The noise schedule is the list of how much noise is added at each step. It controls how quickly the signal is destroyed in the forward process and how the model spends its effort. Linear and cosine schedules are common, and the choice can affect sample quality.

What is classifier-free guidance?

It is a technique for steering samples toward a condition such as a text prompt. The model is trained both with and without the condition, and at sampling time the two predictions are blended. A higher guidance weight makes outputs follow the condition more strictly, but it reduces variety.

What is latent diffusion?

Latent diffusion runs the diffusion process in the compressed latent space of an autoencoder instead of in raw data space. This makes training and sampling much cheaper, which is why large text-to-image systems use it. A decoder converts the final latent back into an image.

Can diffusion models generate tabular data?

Yes. Tabular variants scale numeric columns and use Gaussian noise for them, and use discrete or embedded processes for categories. Quality is judged by column distributions, pairwise relationships and how well models trained on the synthetic data perform on real data.

Are diffusion models good for imputation?

They can be. By keeping known cells fixed and denoising only the missing ones, a diffusion model samples from the conditional distribution of the missing values. This also gives several plausible fills, which expresses uncertainty. It is slower than simple imputers, so it suits cases where quality and uncertainty matter.

Is synthetic data from a diffusion model private?

Not automatically. The model may memorise rare or unique records and reproduce them. Teams check distances to the nearest real record, test for membership leakage and, where needed, add formal privacy methods during training. Synthetic data reduces risk but should be governed like the source data.

How do diffusion models relate to tabular foundation models?

They are complementary. Diffusion models can supply synthetic tables for pre-training or balancing, provide a denoising objective for self-supervised learning, and support imputation and anomaly detection. A tabular foundation model, in contrast, usually predicts targets directly through in-context learning from labelled rows.

Glossary

• Classifier-free guidance: A way to strengthen conditioning by blending conditional and unconditional predictions.

• Denoiser: A network that predicts noise or clean data from a noisy input.

• Forward process: The fixed process that adds noise to data.

• Latent space: A compressed representation learned by an autoencoder.

• Noise schedule: The plan for how much noise is added at each step.

• Reverse process: The learned process that removes noise.

• Sampler: The algorithm that runs the reverse process to produce data.

• Score: The gradient of the log density of the data.

• Time step: The index of the current noise level.

• U-Net: A convolutional network with skip connections between matching resolutions.

• Velocity prediction: A training target that mixes data and noise.

Summary

Diffusion models generate data by learning to reverse a gradual noising process. A network is trained with a simple regression loss to predict the noise added at a random step, and sampling runs the chain backwards from pure noise. They offer stable training, high quality and flexible conditioning, at the cost of slow sampling, which fast samplers and distillation reduce. Compared with GANs they are more stable, and compared with VAEs they give sharper samples. For tabular foundation models, they help create synthetic tables, support privacy-aware sharing, enable imputation by inpainting, provide a denoising objective for self-supervised learning and offer a route to anomaly detection and generative replay. See /model/architecture, /model/research and /model/benchmarks for further reading.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: