Variational inference

Overview

Variational inference is a way to approximate a hard probability distribution with a simpler one by turning inference into optimization. In Bayesian modeling, the quantity you want is the posterior: what you believe about hidden quantities after seeing data. The exact posterior is usually impossible to compute. Variational inference picks a family of simple distributions, then searches for the member of that family that is closest to the true posterior.

This matters for tables because many useful questions about tabular data are questions about hidden structure and uncertainty: what is the most plausible value of a missing cell, how confident is a prediction, which latent group does a row belong to. Variational methods answer them quickly and at scale, and they sit inside VAEs, Bayesian deep learning and many generators of synthetic tables. Related background: Probability, Statistics and Optimization.

What it is

Variational inference, often shortened to VI, is a family of methods for approximating the posterior distribution of a probabilistic model.

The problem it solves

A probabilistic model describes how data could be generated from hidden variables. Bayesian inference reverses this: given observed data x, find the distribution of hidden variables z. The update rule is

posterior(z | x) = likelihood(x | z) * prior(z) / evidence(x)

The numerator is easy to evaluate. The denominator, the evidence, requires summing or integrating over every possible z. For anything beyond toy models, that sum is intractable.

The variational idea

Instead of computing the exact posterior, choose a family of distributions q(z) with adjustable parameters, called variational parameters. Then adjust them until q is as close as possible to the true posterior. Closeness is measured by a divergence, most often the KL divergence from q to the posterior.

Analogy: you cannot draw an exact map of a winding coastline, but you can pick a simple shape, such as an ellipse, and stretch and shift it until it hugs the coast as well as an ellipse can. The ellipse family is your variational family. The best-fitting ellipse is your approximation.

Why it counts as optimization

Once a family is chosen, finding the best q is a normal optimization problem with an objective and gradients. This lets VI use the same tools as deep learning: automatic differentiation, mini-batches and stochastic gradient methods. Compare with sampling methods such as MCMC, which are asymptotically exact but slow. VI trades some accuracy for speed and scalability.

A family tree of related ideas

Seeing these as special cases helps: VI is a spectrum that runs from a single best guess to a rich distribution.

Three everyday pictures

• Weather forecast. A single temperature is a point estimate. A range with probabilities is a distribution. VI gives the range cheaply.
• Map with a fog layer. The true map is complicated. You draw a coarse outline that is good enough to navigate by.
• Jury summary. A jury hears all the evidence (too much to replay), and a foreperson gives a short summary of the verdict and how sure they are.

Where it came from

The mathematical ingredients are old. The calculus of variations, which optimizes over functions, dates back centuries, and physicists used mean-field ideas to approximate systems of many interacting parts. In statistics and machine learning, variational methods for graphical models were developed in the 1990s, together with the idea of bounding the log evidence from below.

Through the 2000s, VI was applied to topic models and other latent variable models, with hand-derived update equations for each model. Stochastic variational inference, in the early 2010s, made it scale to huge datasets with mini-batches. The big step for deep learning came in 2013 and 2014 with the variational autoencoder, which paired VI with neural networks and the reparameterization trick, allowing low-variance gradient estimates through random sampling. See VAEs and Autoencoders.

Later work expanded the idea: flexible variational families built with normalizing flows, importance-weighted bounds that tighten the approximation, black-box methods that need no model-specific derivations, and variational treatments of neural network weights. Diffusion models, though trained differently, also have a variational derivation. See Diffusion models.

How it works

Step 1: Write the model

Specify a prior over hidden variables p(z) and a likelihood p(x | z). For example, each row of a table comes from one of several hidden groups, and within each group the columns follow simple distributions.

Step 2: Pick a variational family

Choose a shape for q(z). Options:

• Mean-field: each hidden variable gets its own independent distribution. Simple and fast, but it ignores correlations between variables.
• Structured families: allow some dependence between variables.
• Flow-based families: start from a simple distribution and bend it with invertible transformations to get a flexible shape.
• Amortized families: a neural network takes the data x and outputs the parameters of q, so one network serves all data points.

Step 3: Build the objective

Directly minimizing the KL divergence to the posterior is impossible, because it involves the unknown evidence. But a little algebra shows that the log evidence equals the ELBO (evidence lower bound) plus a non-negative gap, the KL divergence. So maximizing the ELBO is the same as minimizing that KL divergence.

ELBO = E_q[ log p(x | z) ] - KL( q(z) || p(z) )

Read it as two forces:

1. Reconstruction term. Draw z from q and see how well it explains the data. This pulls q toward values of z that fit.
2. Regularization term. The KL divergence between q and the prior pulls q toward the prior, so the model does not become overconfident.

Analogy: a detective weighing a theory. One part of the score asks, how well does the theory explain the evidence? The other part asks, how far-fetched is the theory compared with what I believed before? A good theory fits well without being absurd.

Step 4: Estimate the expectation

The expectation over q is usually estimated with a few random samples. The difficulty is that sampling is random, and gradients do not pass through randomness easily.

Step 5: Use the reparameterization trick

For a normal q with mean mu and standard deviation sigma, write a sample as z = mu + sigma * eps, where eps is drawn from a standard normal distribution. Now the randomness lives in eps, which does not depend on the parameters, and gradients flow through mu and sigma normally. This trick is what makes VAEs trainable by plain backpropagation.

Step 6: Optimize

Compute the ELBO on a mini-batch, take gradients with automatic differentiation and update with an optimizer such as Adam. Repeat. When the ELBO stops improving, q is as close to the posterior as the family allows.

Step 7: Use the result

Use q as a stand-in for the posterior: get means and intervals for hidden variables, draw samples to make predictions, or compute the predictive distribution for new data by averaging over draws.

Key concepts and terms

The ELBO identity, spelled out

The log evidence can be written as the ELBO plus the KL divergence from q to the true posterior. Because the KL divergence is never negative, the ELBO never exceeds the log evidence. That gives two useful facts.

• Raising the ELBO can only help: it either raises the evidence estimate or shrinks the gap.
• The gap itself cannot be observed, since it needs the true posterior. You can only watch the ELBO and use other checks.

Different ways to compute the gradient

KL divergence in plain words

KL divergence in plain words

The KL divergence from q to p measures the extra surprise you suffer by using q to describe events that really follow p. It is zero when the two match and positive otherwise. It is not symmetric, and the direction matters. The direction used in standard VI, from q to the posterior, punishes q for putting mass where the posterior has none. As a result, q tends to cover one mode and stay narrow, rather than spread over all modes. This is why VI is often said to underestimate uncertainty.

Forward and reverse choices

A worked example

We will approximate a simple posterior with a normal distribution, then look at a tiny model with a latent variable.

Numeric walk-through

Suppose a hidden quantity z has a standard normal prior (mean 0, variance 1). We observe one data point x = 2 with noise: x equals z plus normal noise with variance 1. The exact posterior is known here: mean 1 and variance 0.5. Pretend we do not know that, and fit q = Normal(mu, sigma^2) by maximizing the ELBO.

The ELBO has the reconstruction term, which rewards mu near 2, and the KL term, which pulls mu toward 0. The best compromise is mu = 1, and the best sigma^2 is 0.5. Because the posterior here is itself normal, VI recovers it exactly. In harder models the true posterior is not in the family, and the result is only the best approximation within it.

Code: fitting q by gradient steps

import torch

torch.manual_seed(0)
x = torch.tensor(2.0)

mu = torch.tensor(0.0, requires_grad=True)
log_sigma = torch.tensor(0.0, requires_grad=True)
opt = torch.optim.Adam([mu, log_sigma], lr=0.05)

def log_normal(v, mean, var):
    return -0.5 * (torch.log(2 * torch.pi * var) + (v - mean) ** 2 / var)

for step in range(800):
    sigma = log_sigma.exp()
    eps = torch.randn(64)
    z = mu + sigma * eps                       # reparameterization
    log_lik = log_normal(x, z, torch.tensor(1.0))
    log_prior = log_normal(z, torch.tensor(0.0), torch.tensor(1.0))
    log_q = log_normal(z, mu, sigma ** 2)
    elbo = (log_lik + log_prior - log_q).mean()
    opt.zero_grad()
    (-elbo).backward()
    opt.step()

print(round(mu.item(), 3), round(log_sigma.exp().item() ** 2, 3))

The printed values are close to 1.0 and 0.5, matching the exact answer. The estimate of the ELBO here uses sampling, which is the Monte Carlo part, and the reparameterization lets gradients flow into mu and log_sigma.

A small VAE for a numeric table

import torch
from torch import nn

class TabVAE(nn.Module):
    def __init__(self, d_in, d_lat=4):
        super().__init__()
        self.enc = nn.Sequential(nn.Linear(d_in, 32), nn.ReLU())
        self.mu = nn.Linear(32, d_lat)
        self.logvar = nn.Linear(32, d_lat)
        self.dec = nn.Sequential(nn.Linear(d_lat, 32), nn.ReLU(),
                                 nn.Linear(32, d_in))

    def forward(self, x):
        h = self.enc(x)
        mu, logvar = self.mu(h), self.logvar(h)
        z = mu + torch.randn_like(mu) * (0.5 * logvar).exp()
        return self.dec(z), mu, logvar

def loss_fn(x, recon, mu, logvar):
    recon_err = ((x - recon) ** 2).sum(dim=1)
    kl = -0.5 * (1 + logvar - mu ** 2 - logvar.exp()).sum(dim=1)
    return (recon_err + kl).mean()

The encoder is the amortized q. The decoder is the likelihood. The loss is the negative ELBO under a normal likelihood with fixed variance. Rows that are poorly reconstructed or that need a large KL cost are unusual, which is useful for anomaly detection.

A mean-field example with a hidden group

Imagine a table with one numeric column and two hidden groups. Each row belongs to one group, and within a group the column is normally distributed. Exact inference needs a sum over all assignments of rows to groups. Mean-field VI instead gives each row its own probability of belonging to group one, and updates those probabilities and the group means in turn.

import numpy as np

rng = np.random.default_rng(0)
x = np.concatenate([rng.normal(-2, 1, 150), rng.normal(3, 1, 100)])

mu = np.array([-1.0, 1.0])      # initial group means
pi = np.array([0.5, 0.5])       # group weights
for _ in range(50):
    # responsibilities: how likely each row is in each group
    logp = -0.5 * (x[:, None] - mu[None, :]) ** 2 + np.log(pi)[None, :]
    r = np.exp(logp - logp.max(axis=1, keepdims=True))
    r /= r.sum(axis=1, keepdims=True)
    # update group means and weights
    mu = (r * x[:, None]).sum(axis=0) / r.sum(axis=0)
    pi = r.mean(axis=0)

print(mu.round(2), pi.round(2))

The loop alternates between guessing group membership and refitting the groups, and settles near the true means of -2 and 3 with weights near 0.6 and 0.4. Here the updates are simple enough to write by hand. Black-box VI automates the same pattern for models with no such formulas.

Reading the result

Reading the result

Variational inference in a modeling workflow

1. Write the generative story for your table, even roughly.
2. Choose likelihoods that match column types.
3. Pick a variational family and an optimizer.
4. Fit, and watch the ELBO and its parts.
5. Run posterior predictive checks: simulate data from the fitted model and compare with the real table.
6. Revise the model, not just the optimizer, when checks fail.

Many failures blamed on the inference method are really failures of the model.

Strengths and limitations

Strengths

• Fast and scalable. It uses mini-batches and gradients, so it handles large datasets.
• Amortized inference. After training, one forward pass gives an approximate posterior for any new input.
• Flexible. Any differentiable model can be plugged in, and deep networks can serve as encoder and decoder.
• Gives uncertainty. Output is a distribution, not just a point.
• Deterministic objective. The ELBO is a single number you can monitor and compare.
• Composable. It combines easily with flows, mixtures and hierarchical structure.

Limitations

• Approximation bias. The result is the best member of the chosen family, which may be far from the truth.
• Underestimated uncertainty. Mean-field and reverse KL tend to give overly narrow posteriors.
• Mode seeking. It can miss other modes of a multi-modal posterior.
• Optimization difficulties. Noisy gradients, local optima and posterior collapse can hurt.
• Hard to diagnose. A high ELBO does not prove the posterior is good.
• Sensitive to model misspecification. If the generative model is wrong, a tidy posterior is still wrong.

Tighter bounds and richer posteriors

If a mean-field family is too crude, there are several upgrades.

• Importance weighting. Draw several samples per data point and weight them. The bound gets tighter as the number of samples grows.
• Normalizing flows. Apply a chain of invertible transformations to a simple base distribution to produce a flexible posterior.
• Mixture posteriors. Use a weighted sum of simple distributions to capture several modes.
• Hierarchical posteriors. Add auxiliary variables that induce correlation between latents.
• Learned priors. Let the prior itself be learned, which often improves latent use.

Each upgrade costs more compute, so add them only when diagnostics show the simpler family falls short. See Normalizing flows.

How it compares with alternatives

A good rule of thumb: use sampling when exactness matters and the problem is small, use VI when scale matters, and use ensembles when you need a simple, strong baseline for uncertainty.

Role in tabular foundation models

A tabular foundation model is pre-trained on a large collection of tables so it can predict on a new table without retraining. Variational inference connects to that idea in several concrete ways. See /model/research and /model/architecture.

Amortized inference is the same idea

The central trick of VAEs, amortization, means training one network to output an approximate posterior for any input, instead of running a fresh optimization for each data point. A pre-trained tabular model does the same at the level of whole datasets: it learns, from many tables, a network that maps a labeled context to a predictive distribution for new rows in one pass. In Bayesian terms, this is amortized inference over datasets. The more varied the pre-training tables, the closer the network gets to a true Bayesian predictor over a broad prior. See Meta-learning and Few-shot learning.

The prior comes from synthetic tables

In this view, the pre-training distribution of tables acts as the prior. Synthetic data generators define what kinds of structure the model expects: how columns relate, how noisy they are, how many categories there are. Variational and Bayesian thinking tells us why coverage of that prior matters, since the model can only be well calibrated on table types it has effectively seen. See Synthetic data generation.

Latent-variable generators for tables

VAEs and related models can generate synthetic rows by sampling a latent vector and decoding it. For tables this requires handling mixed types: numeric columns decode to distributions over values, categorical columns decode to class probabilities, and missing cells are modeled with a mask. Such generators help create training or test data and stress-test models. See VAEs and Diffusion models.

Mixed types, missing values and no names

The likelihood in a variational model can be a different distribution per column type: a normal or a flexible binned distribution for numbers, a categorical distribution for labels, and an explicit missingness model. The latent code summarizes the whole row and does not depend on column names, only on values and their pattern, which fits the setting where headers are meaningless or missing.

Imputation

Missing cells can be imputed by inferring a latent code from the observed cells, then decoding the missing ones. Because the latent is a distribution, you can draw several plausible fills, which reflects uncertainty about the gap. This is the logic of multiple imputation done with a learned model. See Imputation networks.

Anomaly detection

A row is anomalous if the model finds it unlikely. The ELBO, or its reconstruction term, gives a score. Rows that the model cannot explain well with a latent code stand out. See Anomaly detection.

Continual learning

Variational methods have a natural story for continual learning: the posterior after one task becomes the prior for the next. Used literally, this keeps a running distribution over parameters and discourages changes to weights the earlier data was confident about. In practice it is approximated with penalties or low-rank updates. See Bayesian deep learning and LoRA.

Sector and semantic classification

When classifying datasets or columns by meaning, a variational latent can capture cluster structure: groups of columns that behave alike share a region of latent space. Uncertainty then tells you when a column does not clearly belong to any known class, which is useful for flagging items for review rather than guessing.

Calibrated confidence

Predictive distributions produced through variational or amortized inference come with probabilities, and these can be checked for calibration on held-out data. A well-calibrated tabular model lets users decide when to trust a prediction and when to defer. See Statistics.

Practical guidance and best practices

Choosing a variational family

• Start with mean-field to get a baseline quickly.
• Move to a structured or flow-based family when correlations between latents matter.
• Use amortization when you will infer many data points.
• Match the likelihood to the column type, rather than forcing everything to normal.

Training tips

• Monitor the two ELBO terms separately.
• Use KL annealing: start the KL weight at zero and raise it slowly, so the latent is used before regularization takes over.
• Use free bits or a floor on the KL per dimension to avoid collapse.
• Use more than one sample per data point when gradients are noisy.
• Standardize numeric columns and handle heavy tails, for example with a log transform.
• Use a modest learning rate and gradient clipping.

Checking the result

For tabular problems

• Treat missing values explicitly. Feed a mask along with the values.
• For categorical columns with many levels, use embeddings in the encoder and decoder.
• For skewed numeric columns, model the log scale or use a mixture likelihood.
• Keep a simple baseline, such as a tree ensemble (XGBoost), to see if the added complexity pays off.

Quick reference: which approximation for which job

Questions to ask before trusting a variational posterior

• Is the model plausible for this kind of table?
• Does the approximate posterior put mass where simple checks say it should?
• Do repeated runs with different seeds agree?
• Do predictive samples reproduce the shape of each column?
• Does the uncertainty grow when the input is far from the training data?
• Does coverage on held-out rows match the stated level?
• Is the result stable when the family is made richer?

If several answers are no, treat the output as a rough guide and look for a better family or a better model.

Common mistakes

1. Equating a high ELBO with a good posterior. The bound can be high while the approximation is poor.
2. Ignoring posterior collapse. If the KL is near zero throughout, the latent is unused.
3. Trusting mean-field uncertainty. Intervals are often too narrow.
4. Forcing one distribution on every column. A normal likelihood on counts or categories is a poor fit.
5. Forgetting to mask missing cells in the loss. The model then learns to predict placeholders.
6. Leaking test information through normalization. Fit scalers on training rows only.
7. Using too small a latent or too weak a decoder. Reconstruction stays poor.
8. Using too large a latent. The model memorizes rows instead of learning structure.
9. Skipping the reparameterization. Naive gradients through sampling are very noisy.
10. Comparing ELBOs across different models without care. Likelihood families and data scaling change the scale of the bound.
11. Judging only by visual samples. Check calibration and downstream task performance as well.
12. Treating the prior as harmless. A badly chosen prior can dominate on small tables.

Frequently asked questions

What is variational inference in simple terms?

It is a method that approximates a complicated probability distribution with a simpler one by optimization. You pick a family of easy distributions and tune its parameters until it matches the true posterior as closely as possible. The result is a fast, scalable way to get beliefs and uncertainty about hidden quantities.

What is the ELBO?

The ELBO, or evidence lower bound, is a quantity that lower-bounds the log probability of the data under a model. Maximizing it pushes the approximate posterior toward the true one. It has a reconstruction part, which rewards explaining the data, and a KL part, which keeps the approximation close to the prior.

What is KL divergence and why does it appear everywhere?

KL divergence measures how different one distribution is from another, in terms of the extra information needed when you use the wrong one. It appears because minimizing the KL from the approximation to the posterior is exactly what variational inference does, and because it turns up in the regularization term of the ELBO.

How is variational inference different from MCMC?

MCMC draws samples that, given enough time, represent the exact posterior, but it can be slow and hard to scale. Variational inference fits a simple distribution by optimization, which is much faster and works with mini-batches, but the answer is only an approximation whose quality depends on the chosen family.

What is the reparameterization trick?

It rewrites a random draw as a deterministic function of the parameters and a fixed noise source, for example mean plus standard deviation times standard normal noise. This lets gradients pass through the sampling step and makes training with backpropagation possible and stable. It is what makes VAEs practical.

What is amortized inference?

Amortized inference uses a neural network to predict the approximate posterior directly from the data, instead of optimizing separate variational parameters for each data point. After training, inference for a new input is a single forward pass. VAE encoders are the standard example, and pre-trained tabular models apply the same idea to whole datasets.

Does variational inference underestimate uncertainty?

Often yes. The usual objective penalizes putting probability where the true posterior has little, so the approximation tends to be narrower than the truth and may focus on a single mode. Richer families, importance weighting and checks against sampling or calibration tests help detect and reduce the problem.

How does variational inference help with missing data in tables?

You infer a latent description of a row from its observed cells and decode the missing ones, and because the latent is a distribution you can draw several possible fills. This gives imputations that reflect uncertainty instead of a single guess, and it handles different column types by choosing a matching likelihood for each.

How does variational inference relate to tabular foundation models?

The shared idea is amortization. A VAE learns one encoder that handles every row, while a tabular foundation model learns one network that handles every table by conditioning on labeled rows. Both trade a big up-front training cost for very cheap inference later, and both depend on how well the training distribution covers the cases seen in use.

When should I avoid variational inference?

Avoid it when you need exact posteriors for a small model, such as in a careful scientific analysis where sampling is affordable, or when the cost of overconfident uncertainty is high and you cannot validate calibration. In those cases, sampling or ensembles with careful checks may be a better choice.

Glossary

• Amortized inference: using a learned network to produce approximate posteriors for any input.
• Approximate posterior: the simple distribution q that stands in for the true posterior.
• Decoder: a network that maps a latent code to the data distribution.
• ELBO: the evidence lower bound that variational inference maximizes.
• Encoder: a network that maps data to the parameters of q.
• Evidence: the probability of the data under the model, with hidden variables summed out.
• KL annealing: gradually increasing the weight on the KL term during training.
• KL divergence: a non-symmetric measure of difference between distributions.
• Latent variable: a hidden variable inferred from data.
• Likelihood: the probability of data given hidden variables.
• Mean-field: a family in which hidden variables are independent under q.
• MCMC: sampling methods that draw from a posterior by running a random walk.
• Posterior collapse: a failure in which the decoder ignores the latent.
• Prior: the distribution over hidden variables before seeing data.
• Reparameterization trick: expressing a sample as a function of parameters and fixed noise.
• Variational family: the set of distributions searched by the method.

Summary

Variational inference approximates an intractable posterior by choosing a family of simple distributions and optimizing their parameters to maximize the ELBO, which balances explaining the data with staying close to the prior. With the reparameterization trick and amortized encoders, it scales to large datasets and deep models. Its main costs are approximation bias, narrow uncertainty and some training pitfalls such as posterior collapse.

For tabular foundation models, variational thinking supplies both a toolbox and a way to understand them. Amortized inference explains how one pre-trained network can answer for many tables, the pre-training distribution acts as the prior, and latent-variable models support synthetic data, imputation, anomaly detection and calibrated confidence on rows with mixed types and missing cells, with or without column names. Related pages: Bayesian deep learning, VAEs, Probability, Optimization and /model/benchmarks.

Related

It belongs to Foundations in my stack, under AI and machine learning. Others in the same group: