Bayesian deep learning

Overview

Bayesian deep learning combines neural networks with Bayesian probability so that a model can say not only what it predicts, but how sure it is. A standard network returns one answer. A Bayesian network returns a distribution of answers, reflecting everything the model does not know about the weights and about the data. That extra information lets you tell a confident prediction from a guess.

For tables, honest uncertainty is often the difference between a useful model and a dangerous one. Tables feed decisions such as lending, maintenance, pricing and triage, where a wrong but confident answer is costly. Tabular foundation models, which predict on new tables without retraining, are a natural place to apply these ideas: their outputs can be full predictive distributions, and their pre-training can be viewed as learning an approximation to Bayesian inference across many tables. Related pages: Variational inference, Probability, Statistics and Optimization.

What it is

Bayesian deep learning (BDL) is the study of neural networks whose parameters, predictions or both are treated as uncertain quantities with probability distributions.

Two kinds of uncertainty

The most useful idea in the field is that uncertainty has two sources.

• Aleatoric uncertainty is noise in the data itself. Even with a perfect model, two identical customers may behave differently. More data does not remove it.
• Epistemic uncertainty is uncertainty about the model, caused by limited data. It shrinks as you collect more data, and it grows for inputs unlike anything the model has seen.

Analogy: forecasting the roll of a die versus forecasting a coin that might be rigged. The die's randomness is aleatoric: you know the rules and still cannot predict the roll. Your doubt about whether the coin is rigged is epistemic: more flips would resolve it.

Distribution over weights

In an ordinary network, training finds one set of weights. In a Bayesian network, you place a prior over the weights, then use data to update it into a posterior. A prediction is the average of the network's outputs over all weight settings, each weighted by how plausible it is. When the posterior is wide, different plausible networks disagree on new inputs, and the disagreement shows up as uncertainty.

Predictive distribution

For a new input, the output is a distribution of possible targets. For classification, it is a set of class probabilities. For regression, it is a mean and a spread, or a full set of quantiles. A good predictive distribution is both sharp (narrow when the model knows) and calibrated (its stated probabilities match observed frequencies).

Exact versus practical

Exact Bayesian inference for deep networks is impossible at scale, because the posterior over millions of weights has no closed form. BDL is therefore a toolbox of approximations: variational methods, sampling, ensembles, dropout tricks and local approximations around a trained solution. The choices differ in cost, accuracy and ease of use.

Where it came from

Bayesian ideas for neural networks go back to the 1990s, when researchers showed how to treat network weights as random and how to regularize training through priors. Early methods used local approximations around a trained solution, and sampling methods that explored the weight space.

Interest revived in the 2010s as networks grew and uncertainty became a practical concern. Variational methods for weights, such as the "Bayes by backprop" approach published in 2015, made weight uncertainty trainable with standard tools. Work on test-time dropout, published around 2016, showed that a cheap trick could be read as approximate Bayesian inference. Deep ensembles, published in 2017, turned out to be a simple and strong baseline for uncertainty. Later work studied scalable sampling, last-layer Bayesian methods, calibration, and the behavior of uncertainty under distribution shift.

A parallel line of research linked neural networks to Gaussian processes, showing that very wide networks behave like them. And in the 2020s, a new thread emerged: pre-trained transformers that approximate the Bayesian posterior predictive distribution directly, by being trained on data drawn from a prior. This is the thread most relevant to tabular foundation models. See TabPFN and /model/research.

How it works

Step 1: Start with a probabilistic model

Write the network as a likelihood p(y | x, w) with weights w, and put a prior p(w) on the weights. A common prior is a normal distribution centered at zero, which behaves like weight decay. The likelihood depends on the task: normal for regression, categorical for classification.

Step 2: Aim at the posterior

Bayesian inference seeks the posterior p(w | data), proportional to the likelihood of the data times the prior. This is intractable for deep networks, so we approximate it.

Step 3: Choose an approximation

Step 4: Train

Training depends on the approach. Variational weights maximize the ELBO. Ensembles minimize the usual loss several times with different seeds. Dropout is trained as normal. In all cases, standard optimizers like Adam are used. See Optimization.

Step 5: Predict with many models

At prediction time, run the input through several sampled networks, or several ensemble members, or several dropout masks. Collect the outputs. The average is the prediction, and the spread tells you about uncertainty.

Analogy: asking a panel of experts instead of one. If they agree, you can be fairly confident. If they scatter, you should be careful. A panel trained on different slices of experience gives a more honest answer than a single expert.

Step 6: Separate the uncertainty types

Given several predictive distributions, compute the total spread, the average within-member spread, and the spread between members. The within-member part estimates aleatoric noise, and the between-member part estimates epistemic uncertainty. This decomposition tells you whether more data would help.

Step 7: Calibrate

Even the best methods are often miscalibrated. Check that a predicted probability of 0.8 is right about 80 percent of the time. Fix with a held-out set: temperature scaling for classifiers, or conformal methods that give intervals with a guaranteed coverage rate under mild assumptions.

Step 8: Use the uncertainty

Use it to defer hard cases to a person, to collect more data where epistemic uncertainty is high, to flag out-of-distribution inputs, or to set decision thresholds that reflect cost.

Key concepts and terms

Metrics for judging uncertainty

A worked example

We will build a small ensemble for a regression table, separate the two uncertainty types, and then show how a Bayesian view treats a coin-flip update.

A numeric warm-up: updating a belief

Suppose you track the success rate of a process. Start with a prior that is uniform over rates (no opinion). After 8 successes in 10 trials, the posterior over the rate is a smooth bump centered near 0.75 and fairly wide. After 80 successes in 100 trials, it is centered near 0.8 and much narrower. The data count shrank the epistemic uncertainty. The inherent randomness of each trial, the aleatoric part, did not change. This is the pattern BDL tries to reproduce for networks.

Code: a deep ensemble for tabular regression

import numpy as np
import torch
from torch import nn

torch.manual_seed(0)
rng = np.random.default_rng(0)

# Synthetic table: two features, noisy target, a gap in the data
x = np.concatenate([rng.uniform(-3, -1, 150), rng.uniform(1, 3, 150)])
x = x.reshape(-1, 1).astype("float32")
y = (np.sin(x) + 0.1 * rng.normal(size=x.shape)).astype("float32")
X, Y = torch.tensor(x), torch.tensor(y)

def make_net():
    return nn.Sequential(nn.Linear(1, 64), nn.Tanh(),
                         nn.Linear(64, 64), nn.Tanh(),
                         nn.Linear(64, 2))      # mean and log-variance

def nll(out, y):
    mean, logvar = out[:, :1], out[:, 1:]
    return (0.5 * logvar + 0.5 * (y - mean) ** 2 / logvar.exp()).mean()

members = []
for seed in range(5):
    torch.manual_seed(seed)
    net = make_net()
    opt = torch.optim.Adam(net.parameters(), lr=1e-2)
    for _ in range(400):
        opt.zero_grad()
        loss = nll(net(X), Y)
        loss.backward()
        opt.step()
    members.append(net)

grid = torch.linspace(-4, 4, 200).reshape(-1, 1)
with torch.no_grad():
    outs = torch.stack([m(grid) for m in members])        # (5, 200, 2)
means, variances = outs[..., 0], outs[..., 1].exp()

aleatoric = variances.mean(dim=0)       # average noise estimate
epistemic = means.var(dim=0)            # disagreement between members
total = aleatoric + epistemic

Each member predicts both a mean and a noise level, so aleatoric uncertainty is modeled directly. The disagreement between members is the epistemic part. In the gap between the two data clusters and outside the training range, the members disagree and the epistemic term grows, exactly the behavior you want.

Code: test-time dropout

import torch
from torch import nn

net = nn.Sequential(nn.Linear(10, 64), nn.ReLU(), nn.Dropout(0.2),
                    nn.Linear(64, 1))

def predict_with_uncertainty(net, x, passes=50):
    net.train()                       # keep dropout active
    with torch.no_grad():
        preds = torch.stack([net(x) for _ in range(passes)])
    return preds.mean(dim=0), preds.std(dim=0)

This is cheap: one trained network and many random forward passes. The spread is a rough epistemic signal, and it is usually less reliable than a real ensemble.

Reading the outputs

Strengths and limitations

Strengths

• Honest confidence. Predictions come with a measure of what the model does not know.
• Better decisions. Costs of errors can be combined with probabilities.
• Out-of-distribution awareness. Epistemic uncertainty rises on unfamiliar inputs.
• Active learning. The model can say where new labels would help most.
• Regularization. Priors and averaging reduce overfitting, especially on small tables.
• Principled combination of evidence. Prior knowledge and data are merged in a clear way.

Limitations

• Cost. Ensembles multiply training. Sampling multiplies prediction cost.
• Approximation quality. Cheap methods may give poor uncertainty without telling you.
• Prior choice. Priors over millions of weights are hard to specify meaningfully.
• Miscalibration. Even Bayesian methods can be over- or under-confident under shift.
• Complexity. More moving parts to tune and debug.
• Evaluation difficulty. Judging uncertainty needs proper scoring rules and many held-out cases.
• Model misspecification. A wrong model with tidy posterior is still wrong.

How it compares with alternatives

For many practical tabular problems, a calibrated tree ensemble with quantile outputs or conformal intervals is a hard baseline to beat. BDL earns its keep when you need epistemic uncertainty, small-data robustness or a coherent treatment of missing information.

Role in tabular foundation models

Tabular foundation models are trained on very many tables and then predict on a new table without retraining. They connect to Bayesian deep learning in a direct way. See /model/research, /model/architecture and /model/benchmarks.

Prior-data fitted prediction

A tabular transformer can be trained on tables sampled from a prior: a distribution over data-generating processes. For each sampled table, a few rows are masked and the network is trained to predict them from the visible rows. The training objective, minimized in expectation, is achieved when the network outputs the true posterior predictive distribution under that prior. So the pre-trained model is an amortized Bayesian predictor: one forward pass produces what exact Bayesian inference would produce, for tables like those in the prior. This is the cleanest link between BDL and foundation models. See TabPFN, Meta-learning and Few-shot learning.

Predictive distributions, not only labels

Because the training target is a distribution, these models naturally output probabilities for classes and full distributions for numeric targets, often as histograms over value ranges. That gives quantiles, intervals and calibrated probabilities with no extra machinery. It also allows decisions that depend on the shape of the distribution, such as skewness or multi-modality.

Epistemic uncertainty from context

With few labeled rows in the context, the model should be uncertain, and with many it should be sharper. This behavior is exactly what the Bayesian posterior does, and it is learned from the pre-training. Ensembling over row orderings, column orderings or feature transformations at inference time can improve calibration and stability. See Statistics.

Priors and synthetic data

The prior is not abstract: it is the synthetic table generator. Its design decides what the model finds plausible, such as the range of feature types, nonlinearity, interactions, noise levels and missing patterns. Gaps in the prior translate into overconfidence on tables unlike it. This is why careful, diverse generation matters. See Synthetic data generation.

Mixed types, missing values and no names

A Bayesian treatment of tables must handle numeric and categorical columns, missingness and weak or absent metadata. In a pre-trained model, these are covered by what the prior contains: tables with different type mixes, with missing cells set at random or in structured ways, and with column order and names scrambled. The model then learns to infer roles from values and relationships rather than from labels, and to represent the uncertainty that arises when a cell is missing or a role is unclear.

Imputation

Imputing a missing cell is a posterior predictive question: what is the distribution of this cell given the others? A pre-trained tabular model can answer it with a distribution, and sampling several values gives multiple imputation. Using the spread rather than a point value avoids false certainty downstream. See Imputation networks.

Anomaly detection

A row is anomalous when it has low predictive probability under the model, or when the model's epistemic uncertainty at that point is high. These two signals mean different things: low probability under a confident model suggests a true outlier, while high epistemic uncertainty suggests the row is outside what the model has seen. Separating them reduces false alarms. See Anomaly detection.

Continual learning

When new data arrives, a Bayesian view says to treat today's posterior as tomorrow's prior. In practice, with large networks, this is approximated by regularizing toward old weights, replaying old examples or adding small adapters such as LoRA. Uncertainty also helps decide which new examples are most informative to store.

Sector and semantic classification

When a model assigns a table or column to a sector or semantic type, calibrated probabilities allow an "unsure" outcome. Low-confidence cases can be sent for review or answered with a list of candidates rather than a single forced label.

Scale and engineering

Uncertainty methods multiply cost, so the engineering matters: mixed precision to save memory, batched sampling to reuse computation, and caching of the encoded context so that repeated predictions on the same context are cheap. See Mixed precision, Transformers and PyTorch.

Practical guidance and best practices

Pick the method to fit the job

• Need a quick, reliable baseline: train a small deep ensemble, or use a tree ensemble with quantile outputs.
• Need guaranteed coverage on intervals: add conformal prediction on top of any model.
• Have very little data: use a pre-trained tabular predictor or a model with a strong prior.
• Need to find unfamiliar inputs: use epistemic uncertainty, or distance in an embedding space, and validate it.
• Have tight latency: use a single model with temperature scaling, or cache ensemble outputs.

A practical workflow

1. Build a strong point-prediction baseline first.
2. Measure calibration on a held-out set.
3. Add the cheapest uncertainty method that fixes what is wrong.
4. Evaluate with proper scoring rules, not only accuracy.
5. Stress-test with shifted data, such as a different time period or region.
6. Document what uncertainty means, so users do not misread it.

Data and evaluation hygiene

• Keep calibration data separate from training and test data.
• Split by group or by time when rows are related, otherwise uncertainty will look better than it is.
• Evaluate on shifted data, since uncertainty is most valuable when the world changes.
• Report coverage and width together.
• Use enough held-out rows for tail probabilities to be measurable.

Communicating uncertainty

Handling tabular quirks

• Standardize numeric columns and treat heavy tails with transforms.
• Add missingness indicators so the model can learn from absence.
• Use embeddings for high-cardinality categories, with dropout or noise to avoid overfitting.
• Keep a record of rows used as context for any in-context predictor, so results can be reproduced.

A weather forecaster analogy

Consider two forecasters. The first says "it will rain tomorrow". The second says "there is a seventy percent chance of rain". Over a year, you can check which one is honest. If it rained on about seventy of every hundred days she gave that figure, she is calibrated. Bayesian deep learning aims to build the second forecaster. The network does not only name the most likely answer; it also says how much to trust it, and that statement can be tested against reality.

Comparison of common approximation families

Where each uncertainty type matters

• Aleatoric uncertainty matters when the outcome is truly noisy, such as a customer's choice. More data will not remove it. It should shape the width of intervals.
• Epistemic uncertainty matters when the model lacks knowledge, such as a new region of the feature space. More data does help. It should trigger caution, a request for a label, or a human review.
• Distribution shift is a special case: the model faces inputs unlike its training data, and epistemic uncertainty should grow.

Active learning and uncertainty

A model that knows what it does not know can ask for help. In active learning, the system picks the rows with the highest epistemic uncertainty and sends them to a person for labeling. This often reaches a target accuracy with fewer labels than random sampling. The same signal supports triage in production: confident cases pass automatically and uncertain cases go to review.

Bayesian optimization as a sibling

Bayesian optimization uses a probabilistic model of an expensive function to decide where to try next. It balances exploring uncertain regions and exploiting good ones. It appears in hyperparameter search, which links it to Optuna and to the broader topic of Bayesian optimization. The probabilistic surrogate there is often a Gaussian process, but a Bayesian neural network or an ensemble can play the same role.

Proper scoring rules in plain words

A scoring rule is a way to grade a probabilistic forecast. A proper rule is one where the best strategy is to report your honest belief.

• Log loss punishes confident wrong answers very hard.
• The Brier score is the squared distance between the predicted probability and the outcome.
• The continuous ranked probability score grades a whole predictive distribution for a numeric target.
• Interval score and coverage grade prediction intervals.

Accuracy alone ignores uncertainty, so a model can look excellent on accuracy and still be badly calibrated.

A decision table for tabular projects

Sanity checks for any uncertainty estimate

1. Plot a reliability diagram and check that it hugs the diagonal.
2. Check coverage of the ninety percent interval on held-out data. It should be near ninety percent.
3. Check that uncertainty rises on shifted data, for example when a column is corrupted on purpose.
4. Check that uncertainty falls as the training set grows for the epistemic part.
5. Check that errors concentrate in the high-uncertainty bins. If they do not, the signal is not useful.
6. Compare against a trivial baseline such as an empirical interval from residuals.

Why classic neural networks are overconfident

A network trained with cross-entropy is rewarded for pushing the probability of the correct class toward one. Over many epochs it learns to be sharp, even where it should hesitate. Large modern networks are often more overconfident than small ones. Temperature scaling, label smoothing and ensembling all soften this. None of them is a full Bayesian treatment, yet each one is cheap and helps in practice.

Further reading inside this site

For the foundations see probability, statistics and variational inference. For how uncertainty meets in-context tabular models, see TabPFN and /model/architecture.

Common mistakes

1. Reading softmax scores as calibrated probabilities. Raw outputs are often overconfident.
2. Judging by accuracy alone. Two models with equal accuracy can differ greatly in calibration.
3. Using one run. Uncertainty estimates vary with seeds. Average over several.
4. Too few ensemble members. Three to five is a minimum for a stable estimate.
5. Calibrating on the test set. It inflates results. Hold out separate calibration data.
6. Ignoring distribution shift. Calibration on last year's data may fail on this year's.
7. Mixing up the two uncertainty types. Aleatoric noise will not shrink with more data.
8. Trusting test-time dropout too much. It is a rough signal, not a full posterior.
9. Overlooking leakage. Leaked features make uncertainty look tiny while the model fails in production.
10. Wide, useless intervals. Coverage without sharpness is not helpful. Check both.
11. Forgetting the prior. With small tables the prior or pre-training distribution drives the answer.
12. Skipping simple baselines. Quantile trees or conformal wrappers often match fancier methods.

More pitfalls to avoid

• Reporting only accuracy and skipping calibration, which hides overconfidence.
• Using ensemble members that are too similar, for example the same seed with a tiny change, which gives little diversity.
• Reading the spread of an ensemble as a guaranteed interval. It is a signal, not a promise of coverage.
• Tuning calibration on the same data used to report it. Keep a separate calibration split.
• Ignoring the cost at prediction time, then finding the service too slow once many samples are needed.
• Treating test-time dropout as exact Bayesian inference. It is an approximation with known limits.
• Forgetting that uncertainty must be re-checked after the data distribution shifts.
• Presenting probabilities to users without explaining what they mean, which breeds false trust or needless doubt.
• Skipping a simple baseline, such as residual-based intervals, that may do nearly as well.
• Using heavy methods on tiny tables where a prior-fitted model or a Gaussian process is already well calibrated.

Each of these is easy to catch with the sanity checks listed earlier, so build them into the evaluation script from day one. - Stopping at one uncertainty number when separate aleatoric and epistemic views would guide action better. - Skipping a check of how the estimate behaves on rows with missing values. - Forgetting to document the method, the number of samples and the seed, so results cannot be reproduced.

Frequently asked questions

What is Bayesian deep learning?

It is the practice of building neural networks that represent uncertainty by treating weights or predictions as probability distributions. Instead of one answer, the model produces a range of answers with probabilities, and the width of that range reflects what the model knows and does not know about the data and its own parameters.

What is the difference between aleatoric and epistemic uncertainty?

Aleatoric uncertainty comes from randomness in the data itself and cannot be reduced by gathering more examples. Epistemic uncertainty comes from limited knowledge about the model and shrinks as more data is seen. Separating them tells you whether to collect more data or accept that the outcome is inherently noisy.

Are deep ensembles Bayesian?

Not strictly, but they behave like a rough approximation to a Bayesian posterior by sampling different good solutions from different random starts. They are simple, parallel and often give excellent uncertainty in practice, so they are widely used as a strong baseline even where the theory is not exact.

Does test-time dropout give real uncertainty?

It gives a cheap signal that can be interpreted as an approximation to Bayesian inference under certain assumptions. In practice its quality varies with the architecture and dropout rate, and it is usually less reliable than an ensemble. Treat it as a rough indicator and validate it against calibration and error-detection checks.

What does calibrated mean?

A model is calibrated if its stated probabilities match observed frequencies. Among all cases where it says 70 percent, about 70 percent should turn out to be positive. Calibration is checked on held-out data, with binned plots or proper scoring rules, and can often be improved with simple post-processing.

How is conformal prediction related?

Conformal prediction wraps any model, Bayesian or not, and uses held-out calibration data to produce intervals or prediction sets with a guaranteed long-run coverage rate under mild assumptions. It does not tell you about model internals, but it is a practical way to make uncertainty statements trustworthy.

How do tabular foundation models relate to Bayesian methods?

When a model is trained on many tables sampled from a prior and asked to predict hidden targets, its best possible output is the Bayesian posterior predictive for that prior. A pre-trained tabular transformer therefore approximates Bayesian inference in a single forward pass, which gives calibrated probabilities and distributions on new tables with no retraining.

Can Bayesian methods help with small tabular datasets?

Yes. With few rows, priors matter and uncertainty is large, so a method that averages over plausible explanations avoids overfitting and reports honest doubt. Pre-trained tabular predictors, Gaussian-process style models and regularized ensembles are all good candidates for small tables.

Is Bayesian deep learning too expensive for production?

It depends on the method. Full sampling is costly, but a small ensemble, a last-layer approximation, conformal wrapping or a pre-trained predictor with cached context can be fast enough for many services. Measure latency and memory early, and pick the cheapest approach that delivers acceptable calibration.

How do I know if the uncertainty is any good?

Test it. Check calibration on held-out data, measure interval coverage against the nominal level, evaluate with proper scoring rules, and see whether high uncertainty actually predicts errors and shifted inputs. If it does not rank mistakes better than chance, the uncertainty is decoration rather than information.

Glossary

• Aleatoric uncertainty: noise in the data that more examples cannot remove.
• Bayesian neural network: a network with a probability distribution over its weights.
• Brier score: the mean squared error of predicted probabilities.
• Calibration: agreement between predicted probabilities and observed frequencies.
• Conformal prediction: a method that gives intervals or sets with guaranteed coverage.
• Credible interval: a range containing a stated share of posterior probability.
• Deep ensemble: a group of networks trained independently and averaged.
• Epistemic uncertainty: uncertainty about the model that shrinks with more data.
• Out-of-distribution: unlike the data the model was trained on.
• Posterior: the updated distribution after seeing data.
• Posterior predictive: the distribution of new outputs averaging over the posterior.
• Prior: the distribution assumed before seeing data.
• Proper scoring rule: a score that is best when probabilities are honest.
• Sharpness: how narrow predictive distributions are.
• Temperature scaling: a single-parameter adjustment of output confidence.
• Test-time dropout: keeping dropout active at prediction time to get varied outputs.

Summary

Bayesian deep learning gives neural networks a way to express what they do not know. It splits uncertainty into noise in the data and doubt about the model, approximates the posterior with variational weights, ensembles, dropout or last-layer methods, and judges the results with calibration checks and proper scoring rules. The best choice is usually the cheapest method that delivers calibrated, useful intervals.

For tabular foundation models, the Bayesian view is close to the heart of the design. Pre-training on tables drawn from a prior makes the network an amortized predictor of the posterior predictive distribution, which supports probabilistic outputs, multiple imputation, anomaly scoring that separates outliers from unfamiliar inputs, and honest confidence in sector and semantic classification. Continual learning and engineering choices then keep that behavior reliable as data and workloads change. Related pages: Variational inference, Statistics, Probability, Bayesian optimization and /model/benchmarks.

Related

It belongs to Foundations in my stack, under AI and machine learning. Others in the same group: