DDPM

Overview

DDPM, short for denoising diffusion probabilistic model, is a generative model that learns to create data by reversing a slow process of adding noise. During training, real samples are corrupted with small amounts of Gaussian noise over many steps until nothing but noise remains. A neural network then learns to undo one step of that corruption at a time. To generate new data, you start from pure noise and apply the learned denoising step again and again until a clean sample appears.

The idea is simple, stable to train and very flexible. It replaced many older approaches for images and audio, and it is now a standard baseline in research on diffusion models. The same recipe also works for rows of a table, where it can create synthetic records, fill in missing cells and support anomaly scoring. This article explains the method step by step, shows a small worked example, compares it with other generators such as the VAE, the GAN and normalizing flows, and describes its role around models that learn from raw tables.

What it is

A DDPM is a latent-variable generative model with a fixed forward process and a learned reverse process.

• The forward process is a Markov chain. At each step it adds a little Gaussian noise to the data. It has no learned parameters.
• The reverse process is another Markov chain. At each step a neural network predicts how to remove a little noise. This part is learned.
• The training objective is a simple regression. The network is shown a noisy sample and asked to predict the noise that was added.
• The sampler runs the reverse chain from random noise to a clean sample. The original version uses hundreds or thousands of steps.

An everyday analogy

Imagine a drop of ink in a glass of water. Over time the ink spreads until the water is evenly cloudy. The forward process is that spreading. Now imagine a film of the same event played backward. You would see a cloud of ink gather into one drop. A DDPM learns to play the film backward, one tiny frame at a time. Each frame is easy to predict because only a small change happens between two frames.

What it is not

• It is not a single-step generator. Quality comes from many small steps.
• It is not an autoencoder. There is no small bottleneck code that stores the sample. The noise levels act as the chain of latent variables, and each has the same size as the data.
• It is not a classifier. It models the distribution of the data itself.

Where it came from

The roots of diffusion modelling go back to work on non-equilibrium thermodynamics as an inspiration for generative models, published around 2015. That early paper showed that you can define a slow destruction of structure and learn its reversal. It received limited attention at first, because results lagged behind other generators.

A second line of work came from score-based generative modelling. Researchers learned the gradient of the log density of the data, called the score, and sampled by following it with Langevin dynamics. Training the score at several noise levels solved the problem that the score is badly defined far from the data.

In 2020 the paper "Denoising Diffusion Probabilistic Models" by Ho, Jain and Abbeel showed that a simple noise-prediction loss, a U-Net and a fixed noise schedule could produce high quality images. It linked the two lines of work: predicting the noise is equivalent to estimating the score up to a scale. After that, progress was fast:

• Improved DDPM work added learned variances and better noise schedules.
• DDIM showed how to sample in far fewer steps with a deterministic path.
• Score-based models were unified with diffusion through stochastic differential equations.
• Classifier guidance and classifier-free guidance made conditional generation controllable.
• Latent diffusion moved the process into the compressed space of an autoencoder, which led to systems such as Stable Diffusion.

For tables, later research adapted the idea to mixed numeric and categorical columns. Examples include models that combine Gaussian diffusion for numbers with a multinomial diffusion for categories. See /model/research for how this literature fits with other table models.

How it works

Step 1: define the forward process

Take a clean sample x0. For each step t from 1 to T, create a slightly noisier version:

• x_t = sqrt(1 - beta_t) * x_{t-1} + sqrt(beta_t) * noise

Here beta_t is a small number that is set by a noise schedule. Early steps use tiny values and later steps use larger ones. After T steps, x_T is close to a standard Gaussian, so the starting point of generation is known and easy to sample.

Step 2: use the closed-form shortcut

Adding noise repeatedly looks slow, but because each step is Gaussian you can jump straight to any step t. Define alpha_t = 1 - beta_t and alpha_bar_t as the running product of alpha values. Then:

• x_t = sqrt(alpha_bar_t) * x0 + sqrt(1 - alpha_bar_t) * eps

where eps is standard Gaussian noise. This is the key to cheap training. You never run the chain forward during training. You pick a random t, draw eps, and build x_t in one line.

An analogy: to know how a photocopy of a photocopy looks after 50 copies, you do not need to make 50 copies. A formula tells you how much of the original remains and how much grain was added.

Step 3: train the denoiser

The network eps_theta takes the noisy sample x_t and the step index t. It predicts eps. The loss is the mean squared error between the true noise and the predicted noise:

• L = E[ || eps - eps_theta(x_t, t) ||^2 ]

Training steps in plain words:

1. Draw a clean sample from the data.
2. Draw a random step t.
3. Draw random noise eps and build x_t.
4. Ask the network to guess eps.
5. Update the weights to reduce the error.

Why predict noise and not the clean sample? The two are tied by the closed-form formula, but noise prediction gives targets with the same scale at every step, which makes optimisation well behaved.

Step 4: sample by reversing the chain

To generate:

1. Draw x_T from a standard Gaussian.
2. For t from T down to 1, predict the noise, compute the mean of the previous step, and add a small amount of fresh noise (except at the last step).
3. Return x_0.

Each reverse step is a small correction. The fresh noise keeps the process stochastic, so the same start can lead to different results.

Step 5: understand the variance choices

The original paper fixes the reverse variance to either beta_t or a related smaller value, and both work well. Later work learned the variance, which helps likelihood and allows fewer steps.

The link to the score

The score is the direction in which the data density increases fastest. Predicting the noise is the same as predicting the negative score, scaled by the noise level. So a DDPM can be viewed as learning a field of arrows that points from noisy points back towards the data. Sampling follows the arrows with some randomness.

Architecture choices

• Images typically use a U-Net with attention and a time embedding.
• Tables typically use an MLP with residual blocks and a time embedding, since there is no spatial grid.
• The step index is turned into a sinusoidal or learned embedding and added to hidden layers.
• Conditioning information (labels, text, other columns) can be concatenated or injected through cross-attention. See transformers for the attention mechanism.

Key concepts and terms

A worked example

A tiny numeric walk-through

Suppose a single number x0 = 2.0 and a short schedule with T = 3 and beta = 0.1, 0.2, 0.3.

• alpha = 0.9, 0.8, 0.7
• alpha_bar = 0.9, 0.72, 0.504

If we pick t = 2 and draw eps = 0.5:

• x_2 = sqrt(0.72) * 2.0 + sqrt(0.28) * 0.5
• sqrt(0.72) is about 0.849, so the first term is about 1.697
• sqrt(0.28) is about 0.529, so the second term is about 0.265
• x_2 is about 1.962

The network is shown x_2 = 1.962 and t = 2 and must output 0.5. Over many samples and noise draws it learns the mean pattern: for a given noisy point and level, which noise most likely caused it.

A compact PyTorch sketch for tabular rows

The code below trains a small DDPM on numeric columns. It is a teaching sketch, not a production tool. See PyTorch for the framework.

import torch
import torch.nn as nn
import math

T = 200
betas = torch.linspace(1e-4, 0.05, T)
alphas = 1.0 - betas
alpha_bar = torch.cumprod(alphas, dim=0)

class TimeEmbed(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.dim = dim
        self.mlp = nn.Sequential(nn.Linear(dim, dim), nn.SiLU(), nn.Linear(dim, dim))
    def forward(self, t):
        half = self.dim // 2
        freqs = torch.exp(-math.log(10000) * torch.arange(half) / half)
        args = t[:, None].float() * freqs[None]
        emb = torch.cat([args.sin(), args.cos()], dim=-1)
        return self.mlp(emb)

class Denoiser(nn.Module):
    def __init__(self, d_in, hidden=256):
        super().__init__()
        self.temb = TimeEmbed(hidden)
        self.inp = nn.Linear(d_in, hidden)
        self.blocks = nn.ModuleList([
            nn.Sequential(nn.SiLU(), nn.Linear(hidden, hidden)) for _ in range(3)
        ])
        self.out = nn.Linear(hidden, d_in)
    def forward(self, x, t):
        h = self.inp(x) + self.temb(t)
        for blk in self.blocks:
            h = h + blk(h)
        return self.out(h)

def train_step(model, opt, x0):
    t = torch.randint(0, T, (x0.shape[0],))
    eps = torch.randn_like(x0)
    ab = alpha_bar[t][:, None]
    xt = ab.sqrt() * x0 + (1 - ab).sqrt() * eps
    loss = ((eps - model(xt, t)) ** 2).mean()
    opt.zero_grad()
    loss.backward()
    opt.step()
    return loss.item()

@torch.no_grad()
def sample(model, n, d_in):
    x = torch.randn(n, d_in)
    for t in reversed(range(T)):
        tt = torch.full((n,), t)
        eps_hat = model(x, tt)
        a, ab, b = alphas[t], alpha_bar[t], betas[t]
        mean = (x - b / (1 - ab).sqrt() * eps_hat) / a.sqrt()
        noise = torch.randn_like(x) if t > 0 else 0
        x = mean + b.sqrt() * noise
    return x

How to read the result

1. Standardise each numeric column before training (zero mean, unit variance).
2. Train until the loss flattens. Use EMA weights for sampling.
3. Sample many rows and undo the standardisation.
4. Compare marginal histograms and pairwise correlations between real and synthetic rows.
5. Run a downstream test: train a model on synthetic rows and test on real rows.

Adding categorical columns

Gaussian noise does not make sense for categories. Common choices:

• One-hot encode the category and treat it as numeric (simple, often weaker).
• Use a multinomial diffusion, where noise means randomly resampling the category.
• Embed categories and diffuse in the embedding space, then decode by nearest neighbour.

Strengths and limitations

Strengths

• Stable training. The loss is a plain regression. There is no adversarial game, so there is no mode collapse of the kind seen in a GAN.
• High sample quality and diversity. Samples cover the data distribution well.
• Flexible conditioning. Labels, text, masks or known columns can steer generation.
• Principled objective. It optimises a bound on the likelihood.
• Scales with data and compute. Bigger networks and longer training keep improving results.
• Natural fit for inpainting. A partly known sample can be completed by fixing the known parts at each step.

Limitations

• Slow sampling. The basic sampler needs hundreds of network calls per sample. Faster samplers such as DDIM reduce this.
• Weak exact likelihoods. The bound is loose compared with normalizing flows, which give exact densities.
• Mixed types are awkward. Tables with categories, integers and skewed numbers need extra design.
• Heavy tails and rare values. Gaussian noise can blur extreme values unless columns are transformed first.
• Privacy is not automatic. A model can memorise training rows, especially when the data set is small.
• Hyperparameter sensitivity. The schedule, step count and loss weighting all influence results.

How it compares with alternatives

A short guide to choosing:

• Need exact densities for scoring? Use flows or autoregressive models.
• Need fast sampling with moderate quality? Use a VAE or a GAN.
• Need the best coverage of a complex distribution and can pay for sampling time? Use DDPM or DDIM.

Role in tabular foundation models

Tabular foundation models are trained to work on raw tables: rows as examples, columns as features of mixed type, missing cells, and often no meaningful column names. A diffusion model is not itself such a foundation model, but it touches several parts of that story. See /model/architecture for how table-native models handle structure, and /model/benchmarks for how they are measured.

Synthetic pre-training data

Many in-context table learners, such as TabPFN, are pre-trained on large numbers of synthetic tables drawn from priors. Generators decide what the model sees. A diffusion model fitted to real tables can act as one source of realistic rows and column dependencies, adding variety beyond hand-built priors. The quality of such rows matters, because a foundation model learns whatever structure the generator contains. See synthetic data generation for the wider topic.

Row and column structure

A diffusion denoiser for tables sees one row as a vector. The network must learn column interactions without a grid. This is why MLP and transformer denoisers are common. Treating each column as a token and using attention gives permutation-friendly behaviour, in line with the design of models like FT-Transformer and SAINT.

Mixed types and missing values

A table has numbers, categories, dates and gaps. Diffusion adapts in these ways:

• Numbers: transform with quantile or power transforms, then add Gaussian noise.
• Categories: use multinomial diffusion or embeddings.
• Missing cells: add a mask channel, or condition on the observed cells only.

Imputation

Imputation is where DDPM is most natural. You keep the observed cells fixed and let the reverse process fill the missing ones. At each step, the observed part is replaced by a noised copy of the known values, and only the unknown part follows the learned denoiser. Because sampling is stochastic, you can draw many completions and measure uncertainty, which supports multiple imputation. This links to imputation networks and to masked cell modeling, where a model also learns to predict hidden cells.

Anomaly detection

A denoiser trained on normal rows reconstructs normal rows well and unusual rows poorly. Add moderate noise to a test row, denoise it, and measure the gap. A large gap is an anomaly signal. This parallels reconstruction-based scoring with a VAE and complements isolation forest and other detectors listed under anomaly detection.

In-context learning and continual learning

In-context learners predict from a few example rows given at inference time. A diffusion model can be conditioned on such context rows, but this is an active research area, and most table foundation models use other designs. For continual learning, a generative model can replay synthetic rows of old tasks to reduce forgetting, a method called generative replay. Quality and drift of the replayed rows decide how well it works.

Sector and semantic classification

Tables from different sectors have different column meanings. A conditional diffusion model can take a sector label or a semantic type as a condition and produce rows that match. In the other direction, features learned by the denoiser can feed a classifier that predicts the sector or semantic type of a column. This is a research idea rather than a settled method, so treat claims with care.

What a diffusion model does not give you

• It does not give a ready-made predictor. Prediction needs a separate head or a conditional formulation.
• It does not remove the need for evaluation on real data.
• It is slower than a forward pass of a pre-trained table model at inference time.

Practical guidance and best practices

Data preparation

• Split into train and test before fitting any transform.
• Apply a quantile transform to skewed numeric columns.
• Handle missing values explicitly with a mask, not by filling with zero.
• Keep rare categories or group them, and document the choice.
• Remove identifier columns that carry no pattern.

Training

• Start with T between 100 and 1000 for tables, and test a smaller T first.
• Use a cosine or linear schedule and compare both.
• Keep an EMA copy of the weights for sampling.
• Use a time embedding and residual MLP blocks.
• Monitor loss across noise levels, not only the mean.

Sampling

• Use a faster sampler such as DDIM once the model is trained.
• Draw more samples than you need and filter by validity rules (for example, age must not be negative).
• Fix the random seed when you need repeatable results.

Evaluation

Privacy

• Measure memorisation, for example by the share of synthetic rows that are near copies of training rows.
• Consider differential privacy training if the data is sensitive.
• Do not assume that synthetic means anonymous.

Variants and extensions worth knowing

Noise schedules

• The linear schedule raises beta steadily. It is simple and was used in the original work.
• The cosine schedule keeps more signal in early steps. It often helps on small or low-dimensional data.
• For tables with few columns, a gentler schedule can avoid destroying all information too soon.
• Always check that the final alpha_bar is near zero and the first one is near one.

Loss weighting

• The simple unweighted loss gives good samples.
• Weighting by signal-to-noise ratio can favour either low or high noise levels.
• Low noise levels teach fine detail. High noise levels teach coarse structure.
• For tables, fine detail matters for exact values such as small counts.

Conditional generation

• Class-conditional: feed a label embedding along with the time embedding.
• Column-conditional: feed the known columns as extra input and generate the rest.
• Classifier-free guidance: drop the condition at random during training, then mix both predictions at sampling time.
• Guidance improves fidelity to the condition but lowers diversity.

Faster sampling

• Deterministic samplers such as DDIM skip steps.
• Higher-order solvers treat sampling as solving a differential equation.
• Distillation trains a student to match several teacher steps in one.
• For tables, even 20 to 50 steps often give acceptable quality.

Latent diffusion

• Instead of diffusing raw data, first compress it with an autoencoder such as a VAE or VQ-VAE.
• The diffusion model then works in the smaller latent space.
• This saves compute and lets the autoencoder handle details like mixed types.
• It is the basis of Stable Diffusion for images.

A short decision checklist

Before choosing DDPM for a table task, ask these questions in order.

1. Do I need new rows, or do I need predictions? If predictions, a pre-trained table model or gradient boosting is usually the better first try.
2. Is the table large enough to learn from? Very small tables overfit and memorise.
3. Are columns mostly numeric? If so, the Gaussian setup fits well.
4. Do I have many categorical columns with many levels? If so, plan a multinomial or embedding approach.
5. Can I afford many network calls per sample? If not, plan for a fast sampler.
6. How will I measure success? Decide on fidelity, utility and privacy checks first.

Cost and compute notes

A minimal configuration file for a tabular run might look like this:

data:
  path: example.internal/tables/train.parquet
  numeric_transform: quantile
  categorical: multinomial
model:
  hidden: 256
  blocks: 3
  time_embedding: sinusoidal
diffusion:
  steps: 200
  schedule: cosine
  prediction: epsilon
train:
  batch_size: 512
  lr: 0.0002
  ema_decay: 0.999
sample:
  sampler: ddim
  steps: 50
  rows: 10000

Common mistakes

• Skipping the transform. Raw skewed columns lead to poor tails and odd peaks.
• Using one-hot columns with plain Gaussian noise and assuming it models categories well.
• Too few steps. A short chain with a poor schedule leaves residual signal at the end, so the sampling start does not match the training end.
• Too much noise at the end or too little. Check that alpha_bar_T is close to zero.
• Forgetting the EMA. Samples from the raw weights can look worse.
• Judging by loss alone. Low loss does not guarantee useful rows.
• Leaking test data. Fitting transforms or choosing hyperparameters on test rows inflates scores.
• Ignoring memorisation on small data sets.
• Comparing against weak baselines. Compare with a tuned gradient boosting model or a pre-trained table model.
• Expecting a speed match with a single forward pass. Sampling needs many network calls unless you use a faster sampler.

Frequently asked questions

What does DDPM stand for?

DDPM stands for denoising diffusion probabilistic model. It is a generative model that creates data by learning to reverse a gradual noising process, and the name comes from the 2020 paper that made this recipe popular.

How is DDPM different from DDIM?

DDPM defines the training objective and a stochastic sampler with many steps. DDIM uses the same trained network but follows a deterministic path that can skip steps, so it generates samples much faster with little loss in quality.

Why does a DDPM need so many steps?

Each step only removes a small amount of noise, and the Gaussian assumption in the reverse step holds only when the step is small. Large jumps would break that assumption, so the basic sampler takes many small ones.

Can DDPM be used for tabular data?

Yes. Rows can be treated as vectors, numeric columns can be diffused with Gaussian noise, and categorical columns can use multinomial diffusion or embeddings. It is used for synthetic data, imputation and anomaly scoring, though it needs careful preprocessing.

How does DDPM relate to a VAE?

Both are latent-variable models trained with a variational bound. A VAE learns a compact code in one step, while a DDPM uses a long chain of noisy versions of the data that all have the same size as the data.

Does DDPM give exact likelihoods?

No. It optimises a bound on the log likelihood and the exact value is hard to obtain. If exact density values matter, normalizing flows are a better fit.

How do I fill missing values with a diffusion model?

Keep the observed cells fixed and let the reverse process change only the missing ones. At every step, replace the observed part with a correctly noised copy of the real values. Repeating with different random seeds gives several plausible fills.

Is synthetic data from a DDPM private?

Not by default. A model may reproduce training rows, especially with small data sets. Test for near copies, limit training length, and consider formal privacy methods when the data is sensitive.

Do I need a GPU?

For small tables a CPU can be enough, but training and sampling many steps is much faster on a GPU. Large tables and big denoisers benefit from CUDA capable hardware.

Is DDPM a tabular foundation model?

No. It is a general generative method. It can supply synthetic training data, imputation and anomaly signals around such models, but a tabular foundation model is a pre-trained network that predicts directly from raw tables, often through in-context learning.

Glossary

• Alpha bar: the running product of one minus beta, giving the share of the original signal left at a step.
• Beta schedule: the sequence of noise amounts used by the forward process.
• Classifier-free guidance: a way to strengthen a condition by mixing conditional and unconditional predictions.
• Denoiser: the network that predicts noise or the clean sample from a noisy input.
• Diffusion model: a generative model built on gradual noising and learned denoising.
• ELBO: evidence lower bound, a tractable lower bound on the log likelihood.
• EMA: exponential moving average of model weights.
• Forward process: the fixed noising chain.
• Gaussian noise: random values drawn from a normal distribution.
• Generative replay: reusing generated samples to retain old knowledge.
• Imputation: estimating missing values.
• Inpainting: completing masked parts of a sample.
• Markov chain: a sequence where each step depends only on the previous one.
• Reverse process: the learned denoising chain.
• Score: gradient of the log density.
• Timestep embedding: vector encoding of the noise level.
• U-Net: an encoder-decoder network with skip connections, common in image diffusion.

Summary

A DDPM learns to generate data by reversing a fixed process that slowly turns samples into noise. Training is a simple noise-prediction regression, which makes it stable and scalable. Sampling is slow in its basic form, but samplers like DDIM fix much of that. For tables, the method needs care with mixed types, skewed numbers and missing cells, yet it offers synthetic rows, multiple imputation and reconstruction-based anomaly scores. It complements, rather than replaces, models that learn directly from raw tables. For more on the surrounding research and design, see /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: