Statistics

Overview

Statistics is the discipline of drawing reliable conclusions from data that is incomplete, noisy and finite. It gives you tools to estimate quantities, to say how sure you are about those estimates, and to decide whether a difference you observed is real or just luck. In machine learning, statistics is the part that turns a single score on a test set into a claim you can defend.

For work on tables, statistics matters twice. First, every table is a sample: the rows you hold are a small slice of whatever process produced them, so every column mean, every correlation and every model score carries sampling error. Second, modern tabular foundation models learn the shape of many such distributions during pre-training, so the language of distributions, likelihoods and uncertainty is also the language in which these models are built, evaluated and trusted. See Probability for the underlying theory and Bayesian deep learning for the uncertainty-aware side.

What it is

Statistics has two broad halves.

• Descriptive statistics summarize a dataset you already have. Means, medians, spreads, quantiles, histograms and correlations all belong here.
• Inferential statistics use the data you have to say something about data you do not have: the whole population, future rows, or the effect of a change.

A third idea cuts across both: uncertainty. A number without an error bar is only half an answer. Statistics gives several ways to attach that error bar, including classical formulas, resampling, and full probability models.

Populations and samples

A population is everything you care about. A sample is the part you observed. If you want to know the average basket size of every customer, but you only have last month's orders, the last month is a sample. The goal of inference is to move from the sample statistic (a number you computed) to the population parameter (the number you wish you knew).

An everyday analogy: tasting one spoonful of soup to judge the whole pot. The spoonful is the sample. If the soup was not stirred, the spoonful may mislead you. That is sampling bias. If you only take one tiny spoonful, you may be unlucky. That is sampling variance.

Frequentist and Bayesian views

There are two main schools of inference.

• Frequentist methods treat parameters as fixed but unknown and ask how a procedure behaves over many imagined repeats of the experiment. Confidence intervals and p-values come from here.
• Bayesian methods treat parameters as uncertain quantities with a probability distribution that is updated by data. Posterior distributions and credible intervals come from here. See Variational inference for a practical way to compute them.

Both are useful. Practitioners often mix them: a frequentist test for a benchmark, a Bayesian model for a hierarchical problem.

Where it came from

Statistics grew out of several old needs. Governments counted people and goods, which produced descriptive summaries. Gamblers and astronomers asked how to combine repeated, slightly different measurements, which produced the early theory of errors and the method of least squares. In the nineteenth century the study of heredity and of agricultural trials pushed the field toward regression, correlation and the design of experiments.

In the early twentieth century, the core machinery of modern inference was formalized: significance tests, confidence intervals, maximum likelihood estimation and the analysis of variance. In parallel, the Bayesian tradition, which is older in spirit, was kept alive and later revived by cheap computing. Sampling methods such as MCMC made it practical to fit complicated Bayesian models.

From the late twentieth century, statistics and machine learning moved closer. Ideas like regularization, cross-validation, the bootstrap, kernel methods and ensemble learning were developed on both sides. Today the two fields share most of their vocabulary, and the main difference is emphasis: statistics leans toward explanation and uncertainty, machine learning toward prediction at scale.

How it works

This section walks through the standard flow of a statistical analysis, one step at a time.

Step 1: Frame the question

Decide what you want to know. Examples: "What is the mean of this column?", "Is model A better than model B?", "Does this feature change the outcome?" A clear question tells you which quantity to estimate and which comparison to make.

Step 2: Understand how the data was collected

The collection process decides what you can conclude. Randomly sampled rows support claims about the population. Rows scraped from a single region or a single month support claims only about that region or month. Think of a survey that was only answered by people who like surveys: the result describes them, not everyone.

Step 3: Choose a model of the data

A statistical model is a story about how the data could have been generated. A simple example: each row's target is a straight-line function of one feature plus random noise. A model has parameters (the slope, the intercept, the noise size). Estimation means picking good values for them.

Step 4: Estimate

Two classic recipes dominate.

• Method of moments matches simple summaries of the model, such as its mean, to the same summaries in the data.
• Maximum likelihood picks the parameters that make the observed data most probable under the model. It is the foundation of most loss functions in machine learning. Squared error is maximum likelihood under normal noise. Cross-entropy is maximum likelihood for class labels.

Analogy: you find a set of footprints and ask which kind of animal makes this footprint most probable. That is maximum likelihood.

Step 5: Quantify uncertainty

An estimate from a sample will change if you draw a new sample. The spread of that change is the standard error. It shrinks as the sample grows, roughly in proportion to one over the square root of the sample size. This single fact explains why quadrupling your data only halves your error.

You can obtain uncertainty in three main ways:

1. Closed-form formulas, such as the standard error of a mean.
2. Resampling, such as the bootstrap, where you redraw rows with replacement many times and recompute the statistic.
3. Posterior distributions, in a Bayesian model.

Step 6: Test or compare

A hypothesis test asks: if there were really no effect (the null hypothesis), how surprising would my data be? The p-value is the probability of seeing data at least this extreme under the null. A small p-value means the data would be unusual if nothing were going on. It does not tell you the probability that the null is true, and it does not tell you the size of the effect.

Step 7: Check assumptions and report

Every method assumes something: independence of rows, a certain shape of noise, a certain sample size. Check the assumptions, then report the estimate, its uncertainty and the effect size, not just "significant".

Common distributions you will meet in tables

Most columns in a real table look like one of a handful of shapes. Knowing the shape tells you which summary and which model are sensible.

• Normal-like columns cluster around a center with symmetric tails, such as measurement noise or heights.
• Skewed positive columns have a long right tail, such as incomes, prices and waiting times. The log scale often tames them.
• Count columns hold small non-negative integers, such as number of visits. Their spread usually grows with their mean.
• Binary columns hold two outcomes, such as yes or no. Their mean is the share of yes.
• Categorical columns hold labels from a fixed set. Their summary is a table of frequencies.
• Bounded ratios live between zero and one, such as a conversion rate.
• Mixtures combine several groups in one column, which shows up as more than one hump.

Regression as a statistical model

Linear regression is the simplest model that connects a target to features. It says the target equals a weighted sum of the features plus noise. Fitting it by maximum likelihood under normal noise gives ordinary least squares. Each weight has a standard error, so you can ask whether a feature's weight is distinguishable from zero.

import numpy as np

rng = np.random.default_rng(1)
n = 200
x = rng.normal(size=(n, 2))
y = 3.0 + 1.5 * x[:, 0] - 0.5 * x[:, 1] + rng.normal(scale=0.8, size=n)

X = np.column_stack([np.ones(n), x])
beta, *_ = np.linalg.lstsq(X, y, rcond=None)
resid = y - X @ beta
sigma2 = resid @ resid / (n - X.shape[1])
cov = sigma2 * np.linalg.inv(X.T @ X)
se = np.sqrt(np.diag(cov))
for name, b, s in zip(["intercept", "x0", "x1"], beta, se):
    print(name, round(b, 3), "+/-", round(2 * s, 3))

The printed ranges are rough 95 percent intervals for each weight. If a range includes zero, the data does not clearly support that feature.

Key concepts and terms

Summary measures at a glance

Bias and variance

Total prediction error can be split into three parts: bias, variance and irreducible noise. A very simple model has high bias (it cannot capture the pattern) and low variance (it gives similar answers on new samples). A very flexible model has low bias and high variance. The best model sits at a balance. This trade-off explains why regularization, early stopping and ensembling help. See Random forest for a method that lowers variance by averaging.

Type I and type II errors

A type I error is a false alarm: you claim an effect that is not there. A type II error is a miss: you fail to see an effect that is there. Lowering one usually raises the other, unless you collect more data. When you run many tests, false alarms pile up, so you need a correction for multiple comparisons.

Confidence intervals, read correctly

A 95 percent confidence interval does not mean there is a 95 percent chance the truth is inside this particular interval. It means that if you repeated the whole procedure many times, about 95 percent of the intervals built would contain the truth. The distinction is subtle, and many practitioners read it informally as "the plausible range", which is fine as a working summary as long as the procedure is sound.

A worked example

Suppose you compare two models on the same tabular dataset using five-fold cross-validation, and you want to know whether model B is really better than model A.

The numbers

Accuracy per fold (illustrative):

The mean difference is 0.0066. The differences are paired, because both models saw the same folds, so you should analyze the differences directly. Their sample standard deviation is about 0.0058, and the standard error is that divided by the square root of five, about 0.0026. A rough interval is the mean plus or minus two standard errors, about 0.0014 to 0.0118. Zero is outside it, which suggests B is slightly better.

A caution: cross-validation folds share training data, so the differences are not fully independent, and the naive interval is a little too optimistic. Treat it as a guide, not a verdict.

The same analysis in code

import numpy as np

a = np.array([0.812, 0.805, 0.820, 0.809, 0.815])
b = np.array([0.821, 0.813, 0.818, 0.822, 0.820])
d = b - a

mean_d = d.mean()
se = d.std(ddof=1) / np.sqrt(len(d))
print("mean diff", round(mean_d, 4), "se", round(se, 4))
print("rough 95% range", round(mean_d - 2 * se, 4), round(mean_d + 2 * se, 4))

# Bootstrap the mean difference
rng = np.random.default_rng(0)
boots = [rng.choice(d, size=len(d), replace=True).mean() for _ in range(5000)]
lo, hi = np.percentile(boots, [2.5, 97.5])
print("bootstrap range", round(lo, 4), round(hi, 4))

With only five folds, the bootstrap is a weak tool, because it cannot invent variation that the five numbers do not show. A better design is repeated cross-validation, or evaluating on many different datasets.

A second walk-through: how many rows do I need?

Say you want to estimate the share of positive rows in a column and you want the answer within plus or minus 0.02 with high confidence. For a share near one half, the standard error is the square root of p times one minus p, divided by the square root of n. Setting two standard errors equal to 0.02 and solving gives n close to 2,500.

import math

p = 0.5          # worst case for spread
margin = 0.02    # target half-width
z = 2.0          # about 95 percent
n = (z ** 2) * p * (1 - p) / margin ** 2
print(math.ceil(n))   # about 2500

Three lessons follow from this small calculation:

• Halving the margin quadruples the rows needed.
• Rare events need far more rows to pin down, because the share is small and the relative error is large.
• The formula assumes independent rows. If rows come in clusters, the effective sample size is smaller.

Comparing across many datasets

If you compare two methods across dozens of datasets, you cannot treat each dataset's score as an equal-sized draw from a normal distribution, because scales differ. Common practice is to rank methods within each dataset and compare average ranks, then use a rank-based test for paired samples. This is more robust to outlier datasets. See /model/benchmarks for how such comparisons are framed.

Strengths and limitations

Strengths

• Honest uncertainty. Statistics forces you to say how sure you are.
• Well understood guarantees. Many procedures come with proven properties under stated assumptions.
• Interpretability. Simple models with clear parameters are easy to explain and audit.
• Efficiency with small data. When rows are scarce, a good statistical model can beat a large flexible one.
• A common language. Teams in science, finance, health and engineering all read intervals and tests.

Limitations

• Assumptions can fail. Independence, stationarity and a correct model form are often only approximately true.
• p-values are easily abused. Testing many ideas and reporting only the best one produces false findings.
• Classical methods can struggle with very high dimensions. More columns than rows breaks many textbook formulas.
• Prediction is not explanation. A model that predicts well may not reveal cause. See Causal inference.
• Real data is messy. Missing values, duplicates, drift and leakage break neat assumptions.

Statistical thinking versus prediction thinking

Two habits of mind

The best practitioners hold both habits at once.

How it compares with alternatives

These are not rivals. A sensible workflow uses machine learning to predict, and statistics to check that the predictions mean what you think they mean.

Role in tabular foundation models

Tabular foundation models are neural networks pre-trained on a very large number of tables so that, given a new table, they can predict without task-specific training. Statistics sits underneath almost every part of this idea. For the model-side view, see /model/research and /model/architecture.

Pre-training is learning a distribution over tables

A model like TabPFN is trained on many synthetic tables drawn from a generator that mixes simple causal structures, different noise levels, different feature types and different relationships between columns. This is a statistical idea: the model is being trained to approximate the posterior predictive distribution, that is, the distribution of a new row's target given all the labeled rows it has seen. Learning to do this across many generators is a form of amortized Bayesian inference. See Synthetic data generation.

In-context learning is estimation without gradient steps

In a classical workflow, you fit parameters to training rows with an optimizer. In a tabular foundation model, the training rows are passed in as context, and the network produces predictions in one forward pass. The "estimation" step has been moved inside the network's weights during pre-training. You can think of the model as having learned a good general-purpose estimator. See Few-shot learning and Meta-learning.

Row and column structure

Statistics treats rows as samples and columns as variables. A tabular foundation model builds this directly into its design: attention across rows plays the role of comparing samples, and attention across columns plays the role of modeling dependence between variables. Because rows are treated as exchangeable draws, shuffling them should not change predictions. That is a statistical symmetry encoded into the architecture. See Transformers.

Mixed types and missing values

Real tables mix numbers, categories, dates and text-like strings, and they have gaps. Statistics offers mature thinking about this:

• Numeric columns need scaling and robust handling of heavy tails and outliers.
• Categorical columns need encodings that do not invent false order.
• Missing values come in three kinds: missing completely at random, missing at random given other columns, and missing not at random. Which kind you have decides whether simple fills are safe. See Imputation networks.

A foundation model can learn to treat missingness as informative, since in many real tables the fact that a value is missing carries a signal.

No column names

Often a table arrives with no meaningful headers, or with headers that are cryptic codes. A statistical view does not need names: it can infer a column's role from its distribution (counts, bounded ratios, integers with few levels, long tails), from its relationships with other columns, and from its marginal shape. A model pre-trained on many tables learns to read these signatures. When names are present, they can add semantic context, for example in sector or domain classification, but a robust model should still work without them.

Imputation and anomaly detection

Both are statistical tasks in disguise.

• Imputation estimates a missing cell from the conditional distribution of that column given the others. A foundation model can produce that conditional directly, and good practice is to look at its uncertainty, not only a point guess.
• Anomaly detection asks how probable a row is under the learned distribution of normal rows. A low density, or an unusually large prediction error, flags an outlier. See Anomaly detection.

Continual learning and drift

Data distributions change over time. Statistics calls this drift or non-stationarity. A model that is updated continually must guard against forgetting old patterns while adapting to new ones. Statistical monitoring, such as comparing the distribution of incoming rows with the training distribution, tells you when an update is needed.

Calibration and predictive uncertainty

A classifier that says "80 percent" should be right about 80 percent of the time on such cases. That property is called calibration, and it is a statistical check you can run on any tabular model.

• Group predictions into bins by predicted probability.
• In each bin, compare the average prediction with the observed share of positives.
• Plot the two against each other. A well-calibrated model follows the diagonal.

Tabular foundation models that output a full predictive distribution make this check natural, since they can give an interval or a probability instead of a single guess. A held-out calibration set can be used to adjust the outputs further. See Bayesian deep learning for the broader picture.

Why synthetic pre-training needs statistical care

Synthetic tables are only useful if their statistical properties resemble real ones. A good generator varies the number of rows and columns, the mix of types, the strength of dependencies, the amount of noise, the share of missing cells and the presence of outliers. If a property is never sampled, the model may never learn to handle it. Statisticians call this coverage of the prior. The wider and more realistic the prior over tables, the more tables the model can handle without extra training.

Evaluating the models

The biggest practical role of statistics is in evaluation. Benchmarks report averages over datasets and folds. To trust them you need paired comparisons, rank-based summaries, confidence intervals and attention to leakage. A benchmark claim is a statistical claim, and it should be read with an interval in mind. See /model/benchmarks.

Practical guidance and best practices

Before modeling

• Look at the data. Plot every column, check ranges, count missing values and duplicates.
• Write down how the data was collected, and who or what is missing from it.
• Split data into train, validation and test sets before any fitting step, and keep the test set untouched until the end.
• If rows are grouped (the same customer, device or patient appears many times), split by group so that no group appears on both sides.
• If rows are ordered in time, split by time so the model never trains on the future.

During modeling

• Start with a trivial baseline, such as predicting the mean or the majority class. A model that cannot beat it is not learning.
• Use cross-validation to choose among options, and report the spread across folds as well as the mean.
• Fix random seeds, and repeat with several seeds when the method is stochastic.
• Prefer a few planned comparisons to many unplanned ones. If you run many, correct for it.
• Regularize. A bit of shrinkage nearly always helps on small tables.

When reporting

• Give the estimate, the interval and the sample size.
• State the effect size in units people understand, such as points of accuracy or units of error.
• Say what the comparison was, which data was used and what was held out.
• Separate "statistically detectable" from "practically important". A tiny gain on a huge test set can be significant and still not worth shipping.

A short pre-flight checklist

Before you trust any number from a tabular experiment, run through these questions.

• Were the train, validation and test sets separated before any fitting?
• Were scalers, encoders and imputers fit on the training part only?
• Are repeated entities kept on one side of each split?
• Is there a simple baseline in the same table of results?
• Is the metric appropriate for the class balance and the cost of errors?
• Are results shown with spread across folds or seeds?
• Were comparisons paired on identical splits?
• Is the number of ideas tried recorded and acknowledged?
• Is there a check for drift between the training period and the deployment period?
• Does the report say what population the result applies to?

Useful habits for tabular work

Where statistics appears in a tabular pipeline

Reading an interval quickly

• A narrow interval far from zero: a clear, precise effect.
• A wide interval that includes zero: the data cannot decide.
• A narrow interval around zero: the effect, if any, is tiny.
• A wide interval far from zero on one side: probably real but poorly measured.

These four readings cover most results you will meet in model comparisons.

Multiple imputation in one paragraph

Instead of filling each gap once, multiple imputation fills the table several times with draws from a plausible distribution, analyzes each completed copy, and pools the results. The pooled spread includes the extra uncertainty that comes from not knowing the missing values. This idea carries over directly to modern imputers that can sample from a predictive distribution rather than return one number.

Common mistakes

1. Peeking at the test set. Tuning on test data makes the final score optimistic. Keep it sealed.
2. Leakage. A feature that encodes the target, or a preprocessing step fit on all rows, inflates scores. Fit scalers and encoders on training data only.
3. Confusing statistical and practical significance. A tiny effect can have a tiny p-value on a large sample.
4. Reading p-values as probabilities of being right. A p-value is about the data under the null, not about the hypothesis.
5. Ignoring multiple comparisons. Try twenty ideas and one will look good by chance alone.
6. Treating dependent rows as independent. Repeated measures on the same entity shrink your real sample size.
7. Reporting only the mean. Two methods with the same mean can differ greatly in spread and worst cases.
8. Forgetting the base rate. A test with a low false alarm rate can still produce mostly false alarms if the event is rare.
9. Dropping missing rows blindly. If missingness is related to the target, dropping rows biases the result.
10. Over-trusting one split. A single train and test split can be lucky or unlucky. Repeat it.
11. Comparing scores from different splits. Only paired comparisons on the same folds are fair.
12. Extrapolating beyond the data. A fitted trend is not a law outside the range it was fitted on.

Frequently asked questions

What is statistics used for in machine learning?

Statistics is used to estimate quantities from samples, to measure how uncertain those estimates are, and to decide whether differences between models are real. It also supplies the likelihood-based losses that most models are trained on, the cross-validation schemes used to estimate generalization, and the tests used to compare methods on benchmarks.

What is the difference between a population and a sample?

A population is the complete set of cases you want to understand, while a sample is the subset you actually observed. Inference is the act of using the sample to say something about the population. The quality of that statement depends on how the sample was collected, because a non-random sample can mislead no matter how large it is.

What does a p-value really tell me?

A p-value tells you how surprising your data would be if the null hypothesis were true. A small value says the data is hard to reconcile with the null. It does not give the probability that the null is true, it does not measure the size of an effect, and it does not say the effect is important. Use it alongside an effect size and an interval.

What is the difference between a confidence interval and a credible interval?

A confidence interval comes from a procedure that, over many repeats, covers the true value at a stated rate. A credible interval comes from a posterior distribution and directly states the probability that the parameter lies inside it, given the model and prior. In practice, with plenty of data and weak priors, the two often look similar.

How much data do I need to trust a result?

It depends on the noise, the size of the effect you care about and the method. As a rule of thumb, error shrinks with the square root of the sample size, so detecting small effects needs many rows. A power analysis before the experiment, or a look at the width of your interval afterward, tells you whether you have enough.

Why is cross-validation better than one train and test split?

A single split gives one noisy estimate of performance, which can be lucky or unlucky. Cross-validation repeats the split so every row is used for testing once, and the average is more stable. It also shows the spread between folds, which is a rough guide to uncertainty. For grouped or time-ordered data, use grouped or time-based folds.

What is the bias and variance trade-off?

Bias is the error from a model being too simple to capture the true pattern, and variance is the error from a model being too sensitive to the particular sample it saw. Making a model more flexible usually lowers bias and raises variance. Good modeling finds the balance, often through regularization, ensembling or simply getting more data.

How do missing values affect statistical conclusions?

It depends on why the values are missing. If they are missing completely at random, dropping them mostly costs precision. If missingness depends on other observed columns, careful imputation can reduce bias. If it depends on the missing value itself, no simple fix is safe, and you should model the missingness process or report the limitation.

How do tabular foundation models relate to classical statistics?

A tabular foundation model learns, during pre-training on many tables, a general-purpose approximation to the predictive distribution that a statistician would otherwise fit by hand for each dataset. At use time it conditions on your labeled rows in one pass. Classical statistics remains essential for splitting data properly, judging uncertainty and comparing models fairly.

Should I trust a benchmark number?

Trust it as far as its design allows. Look for paired comparisons on identical splits, many datasets rather than a few, intervals or rank-based summaries, and clear handling of leakage and tuning. A single headline number without a spread or a description of the protocol says little about how the method will behave on your own table.

Glossary

• Bias: systematic difference between an estimator's average and the true value.
• Bootstrap: resampling rows with replacement to see how a statistic varies.
• Confidence interval: a range from a procedure that covers the truth at a stated long-run rate.
• Credible interval: a range containing a stated share of the posterior probability.
• Cross-validation: splitting data repeatedly to estimate out-of-sample performance.
• Effect size: the magnitude of a difference or relationship.
• Estimator: a rule that maps a sample to an estimate.
• Exchangeable: a set of rows whose order carries no information.
• Hypothesis test: a procedure that weighs data against a null claim.
• Likelihood: the probability of observed data viewed as a function of parameters.
• Maximum likelihood: choosing parameters that make the observed data most probable.
• Missing at random: missingness explained by other observed columns.
• Multiple comparisons: the inflation of false alarms when many tests are run.
• Null hypothesis: the default claim of no effect or no difference.
• p-value: the probability, under the null, of data at least as extreme as observed.
• Posterior: the updated distribution of a parameter after seeing data.
• Power: the chance that a test detects a true effect of a given size.
• Prior: the distribution of a parameter before seeing data.
• Regularization: a penalty or constraint that reduces variance.
• Standard error: the typical size of an estimator's sampling error.
• Variance: the spread of an estimator across different samples.

Summary

Statistics is how you move from a table of rows to a trustworthy claim. It defines estimation, uncertainty, testing and the bias and variance trade-off, and it supplies the likelihood-based losses behind most models. Good practice means clean splits, honest baselines, paired comparisons, intervals instead of bare means, and care about leakage, dependence and multiple tests.

For tabular foundation models, statistics is both the foundation and the yardstick. Pre-training on many tables is a way of learning a predictive distribution, in-context learning is a learned estimator, and imputation, anomaly detection and drift monitoring are all statistical questions about conditional and marginal distributions. Related pages: Probability, Linear algebra, Optimization, Variational inference, XGBoost and Bayesian optimization.

Related

It belongs to Foundations in my stack, under AI and machine learning. Others in the same group: