Probability

Overview

Probability is the branch of mathematics that describes uncertainty. It gives precise rules for talking about events that may or may not happen, for combining evidence, and for saying how likely each outcome is. In machine learning, almost every model can be read as a statement about probability: a classifier outputs the chance of each class, a regression model describes a distribution over a number, and a generative model describes a distribution over whole datasets.

Understanding probability explains why loss functions look the way they do. Cross entropy, mean squared error and many others are negative log likelihoods of a probability model. It also explains what calibrated predictions are, how uncertainty is measured, and how models learn from tables with missing values. This article walks from the basic rules to their use in modern tabular models. It connects to Statistics, Information Theory and Bayesian Deep Learning.

What it is

Probability assigns a number between 0 and 1 to each possible outcome of an uncertain situation. Zero means impossible. One means certain. Numbers in between express degrees of belief or long-run frequencies.

Two ways to read a probability

• Frequentist view. A probability is the long-run share of times an event happens in repeated trials. A fair coin lands heads half the time over many flips.
• Bayesian view. A probability is a degree of belief given the information you have. You can speak of the probability that a one-off event is true, and you update it as new evidence arrives.

Both views use the same rules. They differ in what they say the numbers mean and in how they treat unknown quantities.

The basic objects

• Sample space. The set of all possible outcomes. For one die it is the numbers 1 to 6.
• Event. A set of outcomes, such as "the roll is even".
• Probability measure. A rule that gives each event a number and obeys a few axioms.
• Random variable. A quantity whose value depends on the outcome, such as the number shown on the die.
• Distribution. A description of how likely each value of a random variable is.

The three axioms

1. Every probability is at least zero.
2. The probability of the whole sample space is one.
3. For events that cannot happen together, the probability of either is the sum of the probabilities.

Every other rule follows from these. The complement rule, the addition rule and the product rule are all consequences.

Discrete and continuous variables

A discrete variable takes separate values, such as counts or categories. Its distribution is a table of probabilities that sums to one.

A continuous variable can take any value in a range, such as height. Its distribution is described by a density function. The probability of landing in a range is the area under the density over that range. A density value can exceed one, since it is not itself a probability.

Analogy: a discrete variable is like a bag of labelled marbles, where you count how many have each label. A continuous variable is like a smooth layer of sand, where you ask how much sand lies over a given stretch.

Where it came from

Probability began with games of chance. In the seventeenth century, correspondence about fair division of stakes in an interrupted game led to the first systematic rules for counting outcomes. Early work treated probability as a ratio of favorable cases to equally likely cases.

In the eighteenth century, a theorem for updating beliefs from evidence was published after its author's death. It is now known as Bayes' theorem. Another early result, the law of large numbers, showed that averages of repeated trials settle on the expected value. The central limit theorem followed, explaining why sums of many small effects tend to look bell shaped.

In the nineteenth and early twentieth centuries, probability was applied to physics, insurance and statistics. Its logical foundations were fixed in 1933, when a measure-theoretic formulation introduced the axioms in their modern form. That framework unified discrete and continuous cases.

In the twentieth century, probability became the backbone of statistics and of information theory, where entropy measures uncertainty. Later, probabilistic graphical models, Bayesian inference with sampling methods and variational methods made it central to machine learning. Modern generative models, such as variational autoencoders and diffusion models, are explicitly probabilistic. See /model/research for the research context on this site.

How it works

This section builds up the working tools of probability step by step.

Step 1: Counting and basic rules

If outcomes are equally likely, the probability of an event is the number of favorable outcomes divided by the total.

Two simple rules cover most needs.

• Complement rule. The chance that an event does not happen is one minus the chance that it does.
• Addition rule. For two events A and B, P(A or B) = P(A) + P(B) - P(A and B).

Analogy: when counting students who play football or chess, add both groups and subtract those counted twice.

Step 2: Conditional probability

Often you know something and want to update the odds. The conditional probability of A given B is

P(A | B) = P(A and B) / P(B).

It shrinks the world to cases where B holds, then asks how often A also holds.

Example: among emails, 20 percent are spam. Among spam, 60 percent contain the word "offer". Among non-spam, 5 percent do. The word "offer" shifts the odds of spam. Conditional probability makes this precise.

Step 3: Independence

Two events are independent if knowing one tells you nothing about the other. Formally, P(A and B) = P(A) P(B).

Independence is a strong assumption. Many models assume features are independent given the label, or that samples are drawn independently. When this is false, results can mislead.

Step 4: Bayes' rule

Bayes' rule turns a conditional probability around:

P(A | B) = P(B | A) P(A) / P(B).

In words: posterior = likelihood times prior, divided by evidence.

• Prior. What you believed before seeing data.
• Likelihood. How well each hypothesis explains the data.
• Posterior. What you believe after seeing data.
• Evidence. A normalizing constant that makes the result sum to one.

Analogy: a detective starts with a list of suspects and rough hunches (prior). New clues (data) fit some suspects better than others (likelihood). After weighing the clues, the detective has a new ranking (posterior).

Step 5: Random variables and distributions

A random variable maps outcomes to numbers. Its distribution tells you how likely each value is.

Common distributions:

Step 6: Expectation and variance

The expected value is the probability-weighted average of the values. It is the center of mass of the distribution.

The variance is the expected squared distance from the mean. It measures spread. The standard deviation is its square root, in the same units as the variable.

Analogy: if the distribution is a seesaw loaded with weights, the expectation is the balance point and the variance tells you how far out the weights sit.

Useful facts:

• The expectation of a sum is the sum of expectations, always.
• The variance of a sum of independent variables is the sum of variances.

Step 7: Joint, marginal and conditional distributions

A joint distribution describes several variables together, like a whole table of cell counts.

A marginal distribution describes one variable alone, found by summing or integrating out the others.

A conditional distribution describes one variable when the others are fixed to given values.

This trio is the language of tabular data. A table of rows and columns is a sample from a joint distribution. Predicting one column from the others is estimating a conditional distribution. Filling in a missing cell is sampling from, or taking the center of, a conditional distribution.

Step 8: The chain rule of probability

Any joint distribution can be broken into a product of conditionals:

P(x1, x2, x3) = P(x1) P(x2 | x1) P(x3 | x1, x2).

This is how autoregressive models work. They predict each part given the earlier parts. It is also how a sentence or a table row can be generated one piece at a time.

Step 9: Limit theorems

• Law of large numbers. The average of many independent samples approaches the expected value.
• Central limit theorem. The average of many independent samples is approximately normal, whatever the original shape, if the variance is finite.

These explain why sample averages are reliable and why the bell curve is so common.

Step 10: Estimating distributions from data

Given data, you estimate parameters. The most common method is maximum likelihood: choose the parameters that make the observed data most probable.

The likelihood of independent data is a product. Because products of small numbers underflow, we use the sum of logs instead. Minimizing the negative log likelihood is the same as maximizing likelihood.

A Bayesian goes further. Instead of one best parameter, she keeps a whole posterior distribution over parameters and averages predictions over it.

Key concepts and terms

A worked example

A medical test with Bayes' rule

Suppose a condition affects 2 in 1,000 people. A test detects it 95 percent of the time when present (sensitivity). It gives a false alarm 3 percent of the time when absent (false positive rate). You test positive. What is the chance you have the condition?

Use natural frequencies with 100,000 people.

The chance of having the condition given a positive test is 190 / 3,184, about 6 percent.

The result surprises many people. Even with a good test, the base rate is so low that most positives are false alarms. This is the importance of the prior.

The same example in code

prior = 0.002
sens = 0.95
fpr = 0.03

evidence = sens * prior + fpr * (1 - prior)
posterior = sens * prior / evidence
print(round(posterior, 4))   # about 0.0597

Maximum likelihood for a coin

Flip a coin 10 times and see 7 heads. The likelihood of heads probability p is

L(p) = p^7 (1 - p)^3.

The log likelihood is 7 log p + 3 log (1 - p). Setting its derivative to zero gives p = 7/10.

A Bayesian with a Beta(2, 2) prior, which mildly favors fairness, gets a posterior Beta(9, 5). Its mean is 9/14, about 0.64. It is pulled a bit toward 0.5 compared with the maximum likelihood value of 0.7. With more data the two agree.

Cross entropy as negative log likelihood

A classifier outputs probabilities over K classes. For a true class y, the loss is the negative log of the probability given to y.

import numpy as np

probs = np.array([0.7, 0.2, 0.1])   # model output for one row
y = 0                               # true class index
loss = -np.log(probs[y])
print(round(float(loss), 4))        # 0.3567

If the model gave the correct class probability 0.1 instead, the loss would be 2.30. The loss grows quickly as confidence in the wrong answer rises. This is why probabilistic losses reward honest uncertainty.

Sampling from a distribution

import numpy as np
rng = np.random.default_rng(0)

samples = rng.normal(loc=50.0, scale=10.0, size=100000)
print(round(samples.mean(), 2), round(samples.std(), 2))
print(round(float((samples > 70).mean()), 4))   # close to 0.0228

The share above 70 is about 2.3 percent, since 70 is two standard deviations above the mean of a normal distribution.

A check of calibration

A forecaster says 70 percent chance of rain on 100 days. If it rained on about 70 of those days, the forecaster is calibrated. If it rained on 50, the forecaster is overconfident.

To check a model: group predictions into bins by predicted probability, and compare the average prediction in each bin with the actual rate. Large gaps show miscalibration.

Strengths and limitations

Strengths

• A single language for uncertainty. The same rules cover coins, sensors, tables and text.
• Principled learning. Maximum likelihood and Bayesian updating give clear objectives.
• Interpretable outputs. A probability or a distribution tells a decision maker how much to trust a prediction.
• Composable. Joint, marginal and conditional distributions can be combined with the chain rule.
• Foundation for loss functions. Most standard losses are negative log likelihoods.
• Handles missing information. Unknown values can be integrated out or filled in by sampling.

Limitations

• Model dependence. A probability is only as good as the assumed model. Wrong assumptions give confident wrong answers.
• Independence assumptions. Real data often breaks them.
• Computation. Exact Bayesian inference is often intractable, so we use approximations.
• Small samples. Estimates of tails and rare events are unreliable with little data.
• Misread numbers. People often confuse P(A | B) with P(B | A), the base rate fallacy being a classic case.
• Distribution shift. A model calibrated on one population may be miscalibrated on another.
• Overconfidence in deep models. Large neural networks often output probabilities that are too extreme without calibration.

How it compares with alternatives

Probability is not a tool you pick instead of something else. It is a way of framing problems. Still, it helps to compare it with other ways of reasoning about uncertainty and prediction.

A good habit is to ask of any model: what distribution is it assuming, and what happens when that assumption is wrong?

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types such as numbers, categories and dates, missing values and often unhelpful or absent column names. Models of this kind are pre-trained on many synthetic or real tables and then apply what they learned to a new table through in-context learning, without weight updates at prediction time. See TabPFN and FT-Transformer, and /model/architecture for how such a model is organized.

Probability is the foundation of this entire approach. Here is how it appears concretely.

A table is a sample from a joint distribution

Each row is a draw from a joint distribution over columns. Learning from a table means learning something about that distribution.

• Predicting a target column is estimating the conditional distribution of the target given the other columns.
• Describing a single column is estimating a marginal distribution.
• Generating realistic new rows is sampling from the joint distribution.

This framing works for any column types. Numbers use density models, categories use categorical distributions, and mixed tables combine them with the chain rule.

Pre-training as learning to do Bayesian inference

One influential idea is to pre-train a transformer on a huge supply of synthetic tables. Each table is generated by first sampling a random data-generating process, such as a random causal structure with random functions and noise, and then sampling rows from it.

The network is trained to predict held-out labels from the labelled rows it is shown. Under the right loss, the best possible network outputs the posterior predictive distribution: the prediction you would get by averaging over all data-generating processes that could have produced the visible rows, weighted by how well they explain those rows.

Put simply, the network learns Bayes' rule as a function. At test time, feeding it a new table's labelled rows is the same as conditioning on data. No gradient step is needed. This is the probabilistic reading of in-context learning. See TabPFN for a concrete example.

Loss functions are negative log likelihoods

• Classification uses cross entropy, which is the negative log probability of the true class.
• Regression with a normal assumption uses squared error, which is the negative log likelihood up to constants.
• Regression without a fixed shape can use a discretized output distribution, where the model predicts probabilities over bins of the target. The loss is again the negative log probability of the true bin. This lets the model express skew, multiple modes and heavy tails.

Calibration and uncertainty

Decision makers use predicted probabilities, not just labels. A credit or maintenance decision may depend on whether the risk is 5 percent or 40 percent. So a tabular model should be calibrated: its stated chances should match reality.

Tools include reliability diagrams, expected calibration error, and post-hoc methods such as temperature scaling. Models trained with proper scoring rules, such as log loss, tend to be better calibrated than ones trained with other losses. See /model/benchmarks for how models are compared.

Imputation as conditional sampling

Missing values are an everyday problem in tables. In probabilistic terms, a missing cell is a variable we have not observed.

• Single imputation fills in the mean or mode of its conditional distribution.
• Multiple imputation draws several plausible values, so downstream analysis can reflect the uncertainty.
• Missing mechanism matters. Data can be missing completely at random, missing at random given other columns, or missing not at random. The right treatment differs, and the mechanism is a probabilistic assumption.

A model that has learned the joint distribution can fill gaps by conditioning on the observed cells. See imputation networks.

Anomaly detection as low density

An anomaly is a row that the model finds unlikely. If a model assigns a density or likelihood to rows, then a very low value flags an outlier. Variants use reconstruction error or the distance to nearest neighbors, but the idea is the same: rare under the learned distribution means suspicious. See Anomaly Detection.

Continual learning and drift

Data changes over time. In probability terms, the distribution shifts. Continual learning asks the model to update its beliefs about a new distribution without forgetting older ones. Bayesian updating gives the cleanest picture: yesterday's posterior becomes today's prior. Practical models only approximate this.

Sector and semantic classification

When a model decides what a table is about, for example which industry it belongs to, it is estimating a probability over categories given the evidence in headers and values. Reporting the full distribution, not just the top label, lets users see when the model is unsure. Column names, when present, act as extra evidence, and a model that must work without names relies more on the value distributions themselves.

Mixed types and no column names

Without names, a model must infer a column's meaning from its values. Probability helps here: the empirical distribution of a column, such as its range, skew, number of distinct values and share of missing entries, is a fingerprint. Models can learn to read these fingerprints and treat a column of small integers differently from a column of long-tailed positive numbers.

Trees and boosting

Classic methods such as XGBoost and random forests are also probabilistic when used as classifiers, since they output class probabilities. Their raw scores are often poorly calibrated and benefit from a calibration step.

Practical guidance and best practices

Think in distributions

Before modelling, ask what the target looks like. Is it a count, a share, a positive skewed amount, or a category? Choose an output distribution or a loss that fits.

Check base rates

Always know the base rate of the event. A model with 99 percent accuracy on a 1 percent event may be useless, since predicting "never" also scores 99 percent.

Use proper scoring rules

Log loss and the Brier score reward honest probabilities. Accuracy alone does not.

Calibrate

from sklearn.calibration import CalibratedClassifierCV
from sklearn.ensemble import RandomForestClassifier

base = RandomForestClassifier(n_estimators=200, random_state=0)
calibrated = CalibratedClassifierCV(base, method="isotonic", cv=5)
# calibrated.fit(X_train, y_train)
# p = calibrated.predict_proba(X_test)[:, 1]

Compare the reliability curve before and after, on data not used for calibration.

Separate kinds of uncertainty

• Aleatoric uncertainty is noise in the data itself that no model can remove.
• Epistemic uncertainty is lack of knowledge, which more data can reduce.

Report them separately if the decision depends on it.

Match the missing-data treatment to the mechanism

Ask why values are missing. If missingness itself carries information, add indicator columns instead of silently filling.

Validate on the right split

Evaluate probability quality on data that resembles future use. If the future differs from the past, a random split overstates quality. Use time-based or group-based splits.

Keep numbers stable

Work with log probabilities, use stable functions such as log-sum-exp, and clip probabilities away from exactly 0 and 1 before taking logs.

import numpy as np

def log_sum_exp(a):
    m = np.max(a)
    return m + np.log(np.sum(np.exp(a - m)))

Document assumptions

Write down the assumed distribution, independence claims, and the missing-data mechanism. Revisit them when results look odd.

Common mistakes

• Confusing P(A | B) with P(B | A). The base rate fallacy comes from this swap.
• Assuming independence. Treating correlated features or rows as independent makes uncertainty look smaller than it is.
• Ignoring base rates. A positive test for a rare event is usually a false alarm.
• Reading a density as a probability. A density value can be above one. Only areas under it are probabilities.
• Using accuracy as the only metric. Accuracy hides miscalibration and class imbalance.
• Trusting raw model scores as probabilities. Many models output scores that need calibration.
• Averaging probabilities carelessly. For a combined estimate, think about whether to average probabilities, log odds or likelihoods.
• Overfitting priors or likelihoods. A very flexible model with few data can look certain and be wrong.
• Treating missing data as noise. Missingness often depends on other values.
• Mixing up frequency and belief. State clearly whether a number is a long-run rate or a degree of belief.
• Forgetting multiple comparisons. With many tests, some will look significant by chance.
• Taking logs of zero. Clip probabilities or use stable functions.

Frequently asked questions

What is probability in simple terms?

Probability is a number between 0 and 1 that tells you how likely something is. Zero means it cannot happen and one means it is certain. The rules of probability let you combine these numbers to reason about complicated situations, such as the chance that two events both happen or that one happens given the other.

What is the difference between probability and statistics?

Probability starts with a known model and asks what data it would produce. Statistics starts with data and asks what model likely produced it. They are two sides of the same coin, and modern statistics and machine learning use probability as their working language.

What is a random variable?

A random variable is a number whose value depends on chance, such as the result of a die roll or the next sensor reading. It is described by a distribution that says how likely each value is. Mathematically it maps outcomes to numbers, which lets us compute averages and spreads.

What does Bayes' rule do?

Bayes' rule updates a belief when new evidence arrives. It multiplies the prior belief by the likelihood of the evidence and normalizes the result, giving the posterior. It lets you reverse a conditional probability, for example turning the chance of a positive test given illness into the chance of illness given a positive test.

What is likelihood and how is it different from probability?

Probability describes how likely data are when the model parameters are fixed. Likelihood is the same expression viewed the other way: how plausible different parameter values are when the data are fixed. Maximum likelihood picks the parameters that make the observed data most probable.

Why do machine learning losses use logarithms of probabilities?

Taking the log turns a product of many small probabilities into a sum, which is stable and easy to differentiate. Minimizing the negative log likelihood is equivalent to maximum likelihood, and it penalizes confident wrong predictions heavily. Cross entropy is a common example.

What does it mean for a model to be calibrated?

A calibrated model's stated probabilities match real frequencies. Among all cases where it says 30 percent, about 30 percent should turn out positive. Calibration matters whenever someone uses the probability itself to make a decision, not just the top label.

How does probability relate to tabular foundation models?

A table is a sample from a joint distribution, and predicting a column is estimating a conditional distribution. Models pre-trained on many synthetic tables can learn to approximate Bayesian prediction, so that showing them a new table's labelled rows acts as conditioning on data. Probability also underlies their loss functions, calibration, imputation and anomaly scores.

Is a probability the same as a prediction?

Not quite. A prediction is a single guess, such as a label or a number. A probabilistic prediction describes the whole range of outcomes and how likely each is. The second carries more information and lets users judge risk, but it also needs to be calibrated to be trusted.

Do I need advanced math to use probability in practice?

Not at first. A solid grasp of conditional probability, Bayes' rule, expectation, variance and a handful of common distributions covers most everyday work. Calculus and linear algebra help with more advanced models, and libraries handle much of the computation.

Glossary

• Aleatoric uncertainty: uncertainty from noise in the data that cannot be reduced by more data.
• Bayes' rule: the formula that updates a prior with a likelihood to get a posterior.
• Brier score: the mean squared difference between predicted probabilities and outcomes.
• Calibration: agreement between predicted probabilities and observed frequencies.
• Chain rule: breaking a joint distribution into a product of conditionals.
• Conditional probability: the probability of an event given another event.
• Density: a function whose area over a range gives the probability for a continuous variable.
• Entropy: the average surprise of a distribution.
• Epistemic uncertainty: uncertainty from lack of knowledge, reducible with more data.
• Expectation: the probability-weighted average of a random variable.
• Independence: the property that knowing one event tells you nothing about another.
• Joint distribution: the distribution over several variables together.
• Likelihood: the probability of the observed data viewed as a function of the parameters.
• Marginal distribution: the distribution of one variable after summing out the others.
• Maximum likelihood: choosing parameters that make the observed data most probable.
• Posterior: the belief after seeing data.
• Prior: the belief before seeing data.
• Random variable: a numeric quantity whose value depends on chance.
• Variance: the expected squared distance from the mean.

Summary

Probability is the mathematics of uncertainty. Its core tools are random variables and distributions, conditional probability, independence, Bayes' rule, expectation and variance, and the idea of maximum likelihood. Most loss functions in machine learning are negative log likelihoods, and calibrated predictions are those whose stated chances match reality.

For tabular foundation models, probability is the organizing idea. A table is a sample from a joint distribution, prediction is conditional estimation, pre-training on synthetic tables teaches a network to approximate Bayesian inference, and in-context learning is conditioning on the visible rows. Imputation, anomaly detection, calibration and drift all have clear probabilistic readings. To go further, read Statistics, Information Theory, Bayesian Deep Learning and TabPFN, and visit /model/research and /model/benchmarks.

Related

It belongs to Foundations in my stack, under AI and machine learning. Others in the same group: