Normalizing flows

Overview

Normalizing flows are generative models that turn a simple probability distribution, such as a standard Gaussian, into a complex one by applying a chain of invertible transformations. Because each step can be undone and its effect on volume can be computed, the model gives an exact probability density for any data point. It can also draw samples by running the chain in the forward direction. This combination of exact likelihood, exact sampling and exact inversion is what sets flows apart from most other deep generative models.

The price is a design constraint. Every layer must be invertible and must have a cheap determinant, which limits the kinds of networks you can use. In return, flows are a natural tool for density estimation, anomaly scoring and any task where the actual value of the probability matters. This article explains the idea with everyday analogies, shows the key formulas and a worked example, compares flows with the VAE, the GAN and DDPM, and describes how flows connect to models that learn from raw tables.

What it is

A normalizing flow is a sequence of invertible functions f_1, f_2, ..., f_K that maps data x to a latent variable z with a known simple distribution. The name comes from the way the data distribution is "normalised" into a standard one, and the direction of flow is the chain of changing variables.

Key properties:

• Invertible. Given z you can recover x exactly, and the other way round.
• Same dimension. The latent z has the same number of dimensions as the data x.
• Tractable Jacobian. The determinant of each layer's derivative is cheap to compute.
• Exact log likelihood. The density of x is the density of z plus a sum of log determinant terms.
• Exact sampling. Draw z from the simple distribution and apply the inverse chain.

An everyday analogy

Picture a ball of dough shaped like a sphere. You can knead it into a complex shape, stretching some parts and squashing others, without cutting or gluing. Because you never cut, you can always reverse the kneading. If you know how much each part was stretched, you know how the thickness of the dough changed. In a flow, probability is the dough. Stretching a region spreads probability thinner. Squashing it makes it denser. The log determinant term keeps track of that thickness change.

What it is not

• It is not a compressing model. The latent has the same size as the data.
• It is not a VAE. There is no separate encoder and decoder with a gap in between, and the likelihood is exact, not a bound.
• It is not a GAN. There is no discriminator.

Where it came from

Ideas about transforming densities by changing variables are very old in statistics. Their use in deep learning started in the mid 2010s.

• The paper "NICE: Non-linear Independent Components Estimation" by Dinh, Krueger and Bengio (2014) introduced additive coupling layers with trivial determinants.
• "Variational Inference with Normalizing Flows" by Rezende and Mohamed (2015) popularised the term and used flows to build richer approximate posteriors. See variational inference.
• "Density estimation using Real NVP" by Dinh, Sohl-Dickstein and Bengio (2016) added scale terms in coupling layers and multi-scale design.
• Masked autoregressive flows and inverse autoregressive flows used autoregressive networks to build flexible but still invertible maps.
• Glow (2018) added invertible one-by-one convolutions and showed high quality image synthesis.
• Neural spline flows used monotonic piecewise functions for more flexible, stable transforms.
• Continuous-time flows, linked to neural ODEs, defined the transformation as the solution of a differential equation.
• Flow matching and rectified flows, from the early 2020s, reuse the continuous view but train differently, and they bridge flows with diffusion models.

For a wider view of related research, see /model/research.

How it works

Step 1: the change of variables formula

Let z = f(x) be an invertible, differentiable map, and let p_z be a known density. The density of x is:

• p_x(x) = p_z(f(x)) * | det( df/dx ) |

In logs:

• log p_x(x) = log p_z(f(x)) + log | det( df/dx ) |

The determinant measures how much f stretches or squashes a small region near x. If f stretches a region, probability per unit volume falls, and the determinant term accounts for that.

Step 2: compose many simple layers

A single simple map cannot represent a complex density, so we stack many. For a chain of K layers, the log determinants add up:

• log p_x(x) = log p_z(z_K) + sum over k of log | det( df_k / dz_{k-1} ) |

Each layer is simple enough that its determinant is cheap. Depth makes up for simplicity, like many small kneading moves adding up to a complex shape.

Step 3: coupling layers

The most common building block is the affine coupling layer. Split the input into two parts, a and b. Leave a unchanged. Transform b with a scale and shift that depend on a:

• y_a = a
• y_b = b * exp(s(a)) + t(a)

Here s and t are arbitrary neural networks. They need not be invertible, because a stays fixed and can be used to undo the change:

• b = (y_b - t(y_a)) * exp(-s(y_a))

The Jacobian is triangular, so its determinant is just the product of the diagonal terms, which is exp of the sum of s(a). The log determinant is therefore the sum of s(a).

An analogy: you may adjust the volume of the second half of a song by an amount that depends on the first half, and the first half stays as is. Anyone who hears the first half can undo your change exactly.

To avoid leaving part of the input untouched across all layers, successive layers swap the roles of a and b or use a fixed permutation.

Step 4: autoregressive layers

An autoregressive flow transforms each dimension using a function of the earlier dimensions:

• z_i = (x_i - mu_i(x_1..x_{i-1})) * exp(-alpha_i(x_1..x_{i-1}))

The Jacobian is triangular again. Two versions trade speed:

• Masked autoregressive flow: fast density evaluation, slow sampling.
• Inverse autoregressive flow: fast sampling, slow density evaluation.

Step 5: train by maximum likelihood

Training maximises the exact log likelihood of the data:

1. Take a batch of rows x.
2. Pass them through the chain to get z and the sum of log determinants.
3. Compute log p_z(z), usually a standard Gaussian log density.
4. Add the log determinant sum.
5. Minimise the negative average. Update weights by gradient descent.

This is a plain supervised-looking objective with no sampling noise, which makes training stable.

Step 6: sample and score

• Sample: draw z from the base distribution, apply the inverse of each layer in reverse order.
• Score: compute log p_x(x) for any row. Low values mean the row is unusual.
• Condition: feed a condition into the networks s and t, or into the base distribution, to get a conditional density.

Step 7: handle discrete data with dequantization

Flows model continuous variables. For integers and categories, add small uniform noise to turn discrete values into a continuous bump before training. This is called dequantization. The likelihood of the continuous version then bounds the likelihood of the discrete data.

Key concepts and terms

A worked example

A one-dimensional numeric example

Suppose z is standard Gaussian and the map is x = 2z + 1. Then the inverse is z = (x - 1) / 2 and the derivative dz/dx is 0.5.

For x = 3:

• z = (3 - 1) / 2 = 1
• log p_z(1) = -0.5 * (1 + log(2 pi)), about -1.419
• log | dz/dx | = log 0.5, about -0.693
• log p_x(3) = -1.419 + (-0.693) = -2.112

Check by the direct route: x is Gaussian with mean 1 and standard deviation 2. Its log density at 3 is -log(2) - 0.5 * log(2 pi) - 0.5 * ((3 - 1) / 2)^2, which is about -0.693 - 0.919 - 0.5 = -2.112. The two results agree, as they must.

A compact PyTorch flow with coupling layers

See PyTorch for the framework. The sketch handles numeric table columns.

import torch
import torch.nn as nn
import math

class AffineCoupling(nn.Module):
    def __init__(self, dim, hidden=128, flip=False):
        super().__init__()
        self.dim, self.flip = dim, flip
        self.k = dim // 2
        self.net = nn.Sequential(
            nn.Linear(self.k, hidden), nn.ReLU(),
            nn.Linear(hidden, hidden), nn.ReLU(),
            nn.Linear(hidden, 2 * (dim - self.k)),
        )

    def _split(self, x):
        if self.flip:
            x = x.flip(-1)
        return x[:, :self.k], x[:, self.k:]

    def _merge(self, a, b):
        out = torch.cat([a, b], dim=-1)
        return out.flip(-1) if self.flip else out

    def forward(self, x):
        a, b = self._split(x)
        s, t = self.net(a).chunk(2, dim=-1)
        s = torch.tanh(s)                    # keep scales bounded
        y_b = b * torch.exp(s) + t
        return self._merge(a, y_b), s.sum(dim=-1)

    def inverse(self, y):
        a, y_b = self._split(y)
        s, t = self.net(a).chunk(2, dim=-1)
        s = torch.tanh(s)
        b = (y_b - t) * torch.exp(-s)
        return self._merge(a, b)

class Flow(nn.Module):
    def __init__(self, dim, n_layers=6):
        super().__init__()
        self.layers = nn.ModuleList(
            [AffineCoupling(dim, flip=(i % 2 == 1)) for i in range(n_layers)]
        )
        self.dim = dim

    def log_prob(self, x):
        logdet = torch.zeros(x.shape[0])
        for layer in self.layers:
            x, ld = layer(x)
            logdet = logdet + ld
        base = -0.5 * (x ** 2).sum(-1) - 0.5 * self.dim * math.log(2 * math.pi)
        return base + logdet

    @torch.no_grad()
    def sample(self, n):
        z = torch.randn(n, self.dim)
        for layer in reversed(self.layers):
            z = layer.inverse(z)
        return z

Training and use

flow = Flow(dim=10)
opt = torch.optim.Adam(flow.parameters(), lr=1e-3)

for epoch in range(200):
    for (x,) in loader:
        loss = -flow.log_prob(x).mean()       # negative log likelihood
        opt.zero_grad(); loss.backward(); opt.step()

# score rows: low log probability means unusual
with torch.no_grad():
    scores = -flow.log_prob(test_x)

# generate rows
rows = flow.sample(1000)

Reading the result

1. A falling negative log likelihood on validation data shows learning. Rising validation loss with falling training loss shows overfitting.
2. Compare sampled marginals and correlations with real data.
3. Check that inversion works: pass a row forward then back and confirm that the error is tiny.
4. For anomaly work, compare the score distribution of normal and unusual rows on labelled data.

Adding a condition

class CondCoupling(AffineCoupling):
    def __init__(self, dim, cond_dim, hidden=128, flip=False):
        super().__init__(dim, hidden, flip)
        self.net = nn.Sequential(
            nn.Linear(self.k + cond_dim, hidden), nn.ReLU(),
            nn.Linear(hidden, hidden), nn.ReLU(),
            nn.Linear(hidden, 2 * (dim - self.k)),
        )
    # forward and inverse concatenate the condition with a before calling net

With such layers you can learn the density of numeric columns given a class label, a segment or another set of columns.

Strengths and limitations

Strengths

• Exact likelihood. You get real probability values, useful for scoring and model comparison.
• Exact inversion. Any data point maps to a unique latent point and back.
• Fast sampling and scoring for coupling-based flows, in one pass each.
• Stable training. Maximum likelihood with no adversary.
• Flexible conditioning. Conditions can enter any subnetwork.
• Useful for variational inference. Flows make posteriors richer.
• Good for low and medium dimensional data, which covers many tables.

Limitations

• Architecture constraints. Layers must be invertible with cheap determinants, which limits network design.
• Same-dimension latent. No built-in compression.
• Hard on discrete and mixed data. Dequantization or special layers are required.
• Sample quality can lag behind diffusion models on complex data such as images.
• Likelihood is not always a good guide. Flows can assign high density to out-of-distribution inputs, which weakens naive anomaly detection.
• Numerical issues. Large scales can cause overflow, and inverses can be unstable.
• Depth needed. Many layers are needed for rich shapes, which costs memory.
• Heavy tails and multimodality need careful design, such as spline layers or mixture bases.

How it compares with alternatives

How to decide:

• If the numeric value of the density matters, use a flow or an autoregressive model.
• If you need sharp complex samples, use diffusion.
• If you need a small code for each row, use a VAE.
• If your data are mostly categorical, consider an autoregressive or tree-based generator before a flow.

Role in tabular foundation models

Tabular foundation models are pre-trained on many tables, often synthetic, so that they can predict on a new table through in-context learning. Their inputs are raw tables with rows as examples, columns of mixed type, missing cells and sometimes no column names. A normalizing flow is not such a model, but it is a strong tool for the density side of table work, and its ideas show up in several parts of the pipeline. See /model/architecture for how table-native models treat structure, and /model/benchmarks for how they are compared.

Density estimation for numeric columns

Tables are mostly low or medium dimensional compared with images, which is where flows do well. A flow can learn the joint density of numeric columns, including correlations and non-Gaussian shapes. The result can be used to score rows, to generate synthetic ones, and to compute conditional densities. This is the cleanest role for a flow in a table pipeline.

Mixed types and missing values

Numeric columns fit directly. Categories need dequantization, embeddings or a hybrid design in which an autoregressive model handles categories and a flow handles the continuous part given them. For missing cells, a flow with a mask can marginalise by treating unobserved dimensions as free, or an autoregressive flow can condition on the observed prefix. In practice, a common route is to fill a missing cell by sampling from the conditional density of that column given the observed ones. This links to imputation networks and to masked cell modeling, which use other mechanisms to predict hidden cells.

Anomaly detection

Low log density is the textbook anomaly score, and flows give it exactly. For tables it works well when anomalies are far from the bulk in the modelled columns. Two cautions apply. First, density in raw space can be dominated by dimensions with large variance, so standardise columns. Second, deep density models can give high density to some out-of-distribution points, so validate on labelled cases and compare with isolation forest and other detectors listed under anomaly detection.

Synthetic pre-training data

Pre-training a table model on synthetic tables needs realistic joint distributions. A flow fitted to real numeric tables can generate rows that keep correlation and shape. It can complement hand-made priors by adding distributions seen in real data. See synthetic data generation.

In-context learning and continual learning

In-context learners such as TabPFN predict from context rows without per-table training. A flow is trained per data set, so it does not offer in-context prediction. A related idea is to use a flow as a calibrated density head inside a larger model, giving a full predictive distribution for a numeric target rather than a single value. For continual learning, a flow can serve as a generative memory: sample from the old density and mix with new data. Exact densities also help detect drift, since a drop in average log likelihood on fresh data signals that the distribution has changed.

Sector and semantic classification

Class-conditional flows model p(x given class). Combining them with class priors gives a classifier through Bayes rule. In sector or semantic classification of tables or columns, a flow over summary statistics of a column can describe each type's distribution, and a new column can be assigned to the type with the highest density. This is a transparent approach for small feature sets, though modern pre-trained models usually do better with raw cells.

Probabilistic targets and uncertainty

Many business targets are skewed, bounded or multimodal. A conditional flow can model the full distribution of the target given features, which supports quantiles, intervals and risk measures. This is the same goal that probabilistic table models pursue, reached by a different route.

Limits of the connection

• Flows do not remove the need for careful encoding of categories.
• They do not provide in-context learning.
• Density comparisons across tables with different columns are not meaningful without common scaling.

Practical guidance and best practices

Data preparation

• Standardise every numeric column, or use a quantile transform to a Gaussian shape.
• Dequantize integers by adding uniform noise in the range of one unit.
• Encode categories with embeddings or handle them with a separate model.
• Use a mask input for missing cells.
• Remove constant columns, which cause degenerate densities.
• Add small noise to columns with many repeated values to avoid spikes in the density.

Architecture choices

Training tips

1. Bound the scale outputs with a tanh or a similar function.
2. Use gradient clipping and a modest learning rate.
3. Watch validation likelihood and stop early.
4. Test inversion error regularly.
5. Use weight decay or dropout in the inner networks when data is small.

Evaluation checklist

• Held-out log likelihood per row.
• Marginal and pairwise fit of samples.
• Inversion error on real rows.
• Anomaly detection metrics on labelled cases.
• Calibration of conditional quantiles when used for targets.
• Nearest-neighbour distance between samples and training rows.

Example configuration

model:
  type: normalizing_flow
  layers: 8
  coupling: affine
  hidden: 128
  actnorm: true
  permutation: alternate
train:
  epochs: 200
  batch_size: 512
  lr: 0.001
  grad_clip: 5.0
  early_stopping_patience: 15
data:
  source: example.internal/tables/sensors.parquet
  numeric_transform: quantile_normal
  dequantize_integers: true
  mask_missing: true

Flows beside other tools in a pipeline

1. Use a flow to score density of numeric features and flag low-density rows.
2. Use a tree model or a pre-trained table model for the actual prediction.
3. Use the flow again after deployment to watch for drift in the incoming data.
4. Retrain the flow when the average log likelihood on new data drops beyond a set margin.

Debugging guide

Common mistakes

• Forgetting the log determinant or using the wrong sign.
• Not standardising columns, so a few high-variance columns dominate the likelihood.
• Using raw integers or repeated values without dequantization, which creates infinite density spikes.
• Treating categorical data as continuous numbers and trusting the density.
• Using unbounded scale outputs, causing overflow and NaN losses.
• Trusting likelihood for out-of-distribution detection without validating.
• Comparing densities across different preprocessing or different dimensionalities.
• Skipping the inversion test, which would reveal numerical problems.
• Overfitting small tables with large flows and then reading low training loss as success.
• Choosing a flow when a simple Gaussian mixture or a tree-based model would do. Always include a simple baseline.

Frequently asked questions

What is a normalizing flow?

A normalizing flow is a generative model built from a chain of invertible transformations that turns a simple distribution, such as a Gaussian, into a complex data distribution. Because the chain is invertible and its volume change is tracked, the model gives exact probabilities.

Why must the transformations be invertible?

Invertibility lets the model map data to the latent space for scoring and map latent points back to data for sampling, and it makes the change of variables formula valid. Without it, the density could not be computed exactly.

What is the log determinant for?

It measures how much a transformation stretches or squashes space around a point. Since probability must still add up to one, stretching space lowers density and squashing raises it, and the log determinant is the correction term in the likelihood.

What is a coupling layer?

A coupling layer leaves part of its input unchanged and transforms the other part using a scale and shift computed from the unchanged part. It is invertible by construction and has a simple determinant, so the networks that compute the scale and shift can be arbitrary.

How do flows differ from VAEs?

A VAE has a smaller latent code and trains on a bound of the likelihood. A flow keeps the same dimension as the data and trains on the exact likelihood. A flow does not compress, but it gives exact density values.

Can normalizing flows model tabular data?

Yes, especially numeric columns in low to medium dimensions. Categories and integers need dequantization or a hybrid design, and columns should be standardised. Flows are used for density estimation, synthetic rows and anomaly scores on tables.

Are flows good for anomaly detection?

They can be, since they give the log density of each row. However, deep density models sometimes give high density to unusual points, so you should validate scores on labelled cases and compare with simple detectors.

How do flows compare with diffusion models?

Flows give exact likelihoods and sample in one pass but face architectural limits. Diffusion models such as DDPM give sharper, more diverse samples on complex data but need many steps and offer only likelihood bounds.

What is flow matching?

Flow matching is a training method for continuous flows. It learns a velocity field that moves a simple distribution to the data distribution by regressing on paths between samples, without simulating the differential equation during training. It is closely related to diffusion.

Are normalizing flows tabular foundation models?

No. A flow is trained on a single data set to model its density. It can supply densities, synthetic rows and drift or anomaly signals around table models, but a tabular foundation model is pre-trained across many tables and predicts through in-context learning.

Glossary

• Affine coupling layer: a layer that scales and shifts one part of the input using the other part.
• ActNorm: a learned invertible per-feature scale and shift.
• Autoregressive flow: a flow where each variable depends on earlier ones.
• Base distribution: the simple distribution at the latent end of the flow.
• Bijection: a one-to-one and onto mapping.
• Change of variables: the rule relating densities before and after an invertible map.
• Continuous normalizing flow: a flow defined by an ordinary differential equation.
• Dequantization: adding noise to discrete data to make it continuous.
• Flow matching: a regression-based way to train continuous flows.
• Jacobian: the matrix of first derivatives of a map.
• Log determinant: the log of the absolute determinant of the Jacobian.
• Negative log likelihood: the loss that flows minimise.
• Spline flow: a flow using monotonic piecewise curves as its transforms.
• Volume change: how a map stretches or compresses space locally.

Summary

Normalizing flows model data by pushing a simple distribution through a chain of invertible, differentiable maps and tracking the change in volume with log determinants. This gives exact likelihoods, exact inversion and fast sampling, and trains stably by maximum likelihood. The cost is a restricted architecture, a latent as large as the data, and extra work for discrete columns. For tables, flows suit density estimation over numeric columns, conditional distributions of targets, anomaly scores and drift monitoring, and they complement tools such as the VAE, DDPM and VQ-VAE. They support, and do not replace, models that learn directly from raw tables. See also /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: