Information theory

Overview

Information theory is the mathematical study of measuring, storing and communicating information. Its central idea is that information can be quantified: a surprising event carries more information than an expected one, and the average surprise of a source is called entropy. From this start come tools used everywhere in machine learning, namely cross-entropy, Kullback-Leibler (KL) divergence and mutual information.

These quantities are not abstract extras. Cross-entropy is the standard loss for classification and language modeling. KL divergence appears in variational methods, in policy optimization and in distillation. Mutual information underlies contrastive objectives such as InfoNCE and feature selection. Compression ideas explain why tokenizers work. For models that learn from raw tables, such as TabPFN, information theory gives a precise language for what a column tells you about a target, how much a missing cell hides and how well a model's predicted distribution matches reality.

What it is

Information theory treats a message as a choice among possibilities. The more possibilities there are, and the more evenly likely they are, the more information a message carries when it tells you which one occurred.

Think of the game of twenty questions. If you must find one object among a million equally likely ones, you need about twenty yes or no questions, because each answer halves the options. The number of questions is the information content, measured in bits. If some objects are far more likely than others, a clever player asks about the likely ones first and needs fewer questions on average. Entropy is that average.

The core quantities

Bits and nats

Information is measured in bits when logarithms use base 2, and in nats when they use base e. Machine learning code usually uses natural logs, so losses are in nats. Converting is a constant factor: one nat is about 1.44 bits.

The two views

• Communication view: how few symbols are needed to send data, and how reliably it can be sent through noise.
• Statistical view: how well a probability model describes data, and how much one variable reveals about another.

Both views share the same formulas, which is why a result about coding can become a result about learning.

Where it came from

The field is usually dated to a 1948 paper by Claude Shannon, "A Mathematical Theory of Communication". It defined entropy as a measure of information, proved limits on lossless compression, and proved that reliable communication is possible over a noisy channel up to a maximum rate called capacity. Earlier work on telegraphy and signal transmission by others, including Hartley and Nyquist, had touched related ideas, but Shannon unified them.

Developments

• Source coding: methods like Huffman coding (1952) build prefix codes that approach entropy. Later arithmetic coding came closer for general sources.
• Channel coding: error-correcting codes were designed to approach capacity.
• Kullback and Leibler introduced a measure of divergence between distributions in 1951, now known as KL divergence.
• Kolmogorov, Solomonoff and Chaitin developed algorithmic information theory in the 1960s, which defines complexity through the shortest program that produces a string.
• The minimum description length principle linked model selection to compression: the best model is the one that compresses the data plus the model itself best.
• In statistics, the Akaike information criterion tied model comparison to KL divergence.
• In machine learning, the information bottleneck idea framed learning as compressing inputs while keeping what predicts the target.
• Modern deep learning adopted cross-entropy as the default loss, variational bounds based on KL divergence, and mutual information estimators for self-supervised learning.

For how these threads touch current research, see /model/research.

How it works

Step 1: Surprise of a single event

The information content of an event with probability p is the negative log of p. A coin that lands heads with probability one half gives 1 bit when it lands heads. An event with probability one in a thousand gives about 10 bits. Impossible events would carry infinite surprise, which is why probabilities of zero cause trouble in losses.

Analogy: a weather report saying "it will be dry in a desert" tells you almost nothing. A report saying "it will snow in a desert" tells you a lot. The rarer the event, the more it says.

Step 2: Entropy is average surprise

For a distribution over outcomes, entropy is the probability-weighted average of the surprise. A fair coin has the maximum entropy for two outcomes, 1 bit. A coin that is heads 99 percent of the time has low entropy, about 0.08 bits, because you can nearly always guess the result.

Entropy is highest when all outcomes are equally likely and zero when one outcome is certain. It tells you the minimum average number of bits per symbol that any lossless code can reach.

Step 3: Cross-entropy measures the cost of a wrong model

Suppose reality follows distribution P, but you build your code or prediction around a different distribution Q. The average cost of encoding data from P with a code built for Q is the cross-entropy of P and Q. It is never smaller than the entropy of P. The gap is the KL divergence.

Cross-entropy = entropy of P + KL divergence from P to Q.

This is why minimizing cross-entropy loss during training pushes a model's predicted distribution toward the true one. Because the entropy of the data does not depend on the model, minimizing cross-entropy and minimizing KL divergence are the same optimization.

Step 4: KL divergence compares two distributions

KL divergence is the extra bits wasted when Q is used instead of P. It is zero only when the two distributions match. It is not symmetric: the divergence from P to Q differs from Q to P. It is also not a true distance, since it does not obey the triangle inequality.

Direction matters in practice:

• Forward KL (from the data distribution to the model) punishes a model that assigns near-zero probability where the data has mass. It leads to models that cover all modes, sometimes too broadly.
• Reverse KL (from the model to the data distribution) punishes a model that places mass where the data has none. It leads to models that lock onto one mode.

Step 5: Mutual information measures dependence

Mutual information between X and Y is the reduction in uncertainty about one when you learn the other. It equals the entropy of X minus the entropy of X given Y. It is zero when the variables are independent and grows with dependence of any kind, not only linear.

Analogy: knowing a person's postcode tells you something about their income range. The mutual information between postcode and income measures how much. Correlation would miss a relationship that is strong but not a straight line, while mutual information would capture it.

Step 6: Conditional entropy and information gain

Conditional entropy is the uncertainty left in Y after seeing X. Information gain is the drop in entropy from a split or an observation. Decision tree algorithms use it: at each node, choose the feature split that reduces label entropy the most. Methods like random forest build on trees that use such criteria, and boosted tree libraries such as XGBoost use related impurity and gain measures.

Step 7: Coding and compression

Source coding theory says the best lossless compression rate is the entropy. Frequent symbols get short codes and rare symbols get long ones. A tokenizer built with BPE is a practical, approximate step in that direction. Lossy compression adds the idea of rate-distortion: allow some error and trade it against the number of bits. This frames quantization of model weights as a compression problem, where fewer bits per weight are traded against accuracy loss.

Step 8: Capacity and noise

Channel capacity is the highest rate at which information can be sent with arbitrarily small error. Analogy: a conversation in a noisy room. Speaking slowly and repeating key words lets you get a message through even with noise, at the cost of speed. Shannon proved that there is a best possible trade-off for every channel. In learning, a noisy label or a noisy measurement acts like a channel, and the amount of information that passes limits the best achievable accuracy.

Key concepts and terms

A worked example

A small numeric walk-through

A column in a table holds a category with three values and these probabilities:

• A: 0.5
• B: 0.25
• C: 0.25

Entropy in bits:

• A contributes 0.5 times 1 = 0.5
• B contributes 0.25 times 2 = 0.5
• C contributes 0.25 times 2 = 0.5
• Total entropy = 1.5 bits

An optimal code assigns A one bit and B and C two bits each, giving an average of 1.5 bits per value. A fixed-length code would need 2 bits, so the savings is 25 percent.

Now suppose a model wrongly predicts a uniform distribution, one third each. Its cross-entropy against the true distribution is log2(3), about 1.585 bits. The KL divergence is 1.585 minus 1.5, about 0.085 bits. That is the price of ignorance about the skew.

Computing the quantities in Python

import numpy as np

def entropy(p, base=2):
    p = np.asarray(p, dtype=float)
    p = p[p > 0]
    return -(p * np.log(p)).sum() / np.log(base)

def cross_entropy(p, q, base=2, eps=1e-12):
    p = np.asarray(p, dtype=float)
    q = np.clip(np.asarray(q, dtype=float), eps, 1.0)
    return -(p * np.log(q)).sum() / np.log(base)

def kl(p, q, base=2):
    return cross_entropy(p, q, base) - entropy(p, base)

p = [0.5, 0.25, 0.25]
q = [1/3, 1/3, 1/3]
print(entropy(p))            # 1.5
print(cross_entropy(p, q))   # about 1.585
print(kl(p, q))              # about 0.085

Mutual information between a feature and a target

import numpy as np
from sklearn.feature_selection import mutual_info_classif

rng = np.random.default_rng(0)
n = 5000
x1 = rng.normal(size=n)                 # informative
x2 = rng.normal(size=n)                 # noise
y = (x1**2 + 0.3 * rng.normal(size=n) > 1).astype(int)   # non-linear link

X = np.c_[x1, x2]
mi = mutual_info_classif(X, y, random_state=0)
print(dict(zip(["x1", "x2"], mi.round(3))))

The correlation between x1 and y is near zero here because the relation is symmetric around zero, but mutual information correctly marks x1 as informative. This is a standard reason to prefer it for screening features with non-linear effects.

The link to classification loss

For a classifier outputting probabilities, the usual loss is:

import torch
import torch.nn.functional as F

logits = torch.tensor([[2.0, 0.5, -1.0]])
target = torch.tensor([0])
loss = F.cross_entropy(logits, target)     # = -log softmax(logits)[target]
print(loss.item())

This single number is the surprisal the model assigns to the true class. Averaged over a dataset, it is an estimate of the cross-entropy between the data and the model. Perplexity is its exponential.

Information gain in a tree split

Consider 10 rows: 5 labeled yes and 5 labeled no. The label entropy is 1 bit. A split sends 4 rows (all yes) to the left and 6 rows (1 yes, 5 no) to the right.

• Left entropy = 0.
• Right entropy: probabilities 1/6 and 5/6, about 0.65 bits.
• Weighted entropy after split = 0.4 times 0 + 0.6 times 0.65 = about 0.39 bits.
• Information gain = 1 minus 0.39 = about 0.61 bits.

A split with higher gain separates the classes better, so the tree prefers it.

Strengths and limitations

Strengths

• Universal: one framework for compression, communication, statistics and learning.
• Nonparametric view of dependence: mutual information detects any kind of relationship.
• Principled losses: cross-entropy follows from maximum likelihood.
• Tight bounds: limits like entropy and capacity tell you what is impossible, not only what is hard.
• Interpretable units: bits and nats give a shared scale for uncertainty.
• Useful diagnostics: perplexity, information gain and divergence make model behavior measurable.

Limitations

• Estimation is hard. Mutual information and entropy for continuous or high-dimensional variables are difficult to estimate from finite samples, and estimators can be biased or have high variance.
• Meaning is ignored. Information theory measures quantity and structure, not whether information is useful, true or relevant.
• Asymmetry of KL. The direction chosen changes the behavior of the model, and infinite values arise when supports do not overlap.
• Differential entropy depends on the units of the variable and can be negative, so it must be used carefully.
• Assumes a known or well-estimated probability model. In practice the model is itself learned.
• Lossless limits do not account for computation. A code that reaches entropy in theory may be too slow to use.
• Mutual information is not a causal measure. See causal inference for questions about intervention.

Typical failure modes in practice

How it compares with alternatives

Information measures sit alongside other tools in probability, statistics and optimization. They share ground with variational inference, where a KL divergence is the objective, and Bayesian deep learning, where uncertainty is central.

Role in tabular foundation models

Tabular foundation models learn from raw tables. Information theory describes what these tables contain, how a model should be trained on them and how to judge its uncertainty.

Loss design for cells and labels

Classification heads in models such as TabPFN and TabICL are trained with cross-entropy, so the predicted class probabilities are pushed toward the true conditional distribution. For regression, some tabular models predict a distribution over binned values and use cross-entropy over bins, which gives a full predictive distribution instead of a single number. Scoring a prediction with a log-loss rule is a proper scoring approach: it is minimized in expectation only by the true distribution, which rewards honest uncertainty.

Row and column structure

A table has columns that carry different amounts of information about a target. Mutual information between a column and the label, and conditional mutual information given other columns, describe how useful and how redundant each column is. Two columns that duplicate each other add little beyond the first. Attention across columns can learn to exploit these dependencies, as described in /model/architecture. Rows are treated as samples from a joint distribution, and the model's task in context is to estimate a conditional distribution from them.

Pre-training on synthetic tables

Synthetic data generation for tabular pre-training builds tables from random causal graphs and noise. From an information view, the generator defines a prior over joint distributions. The pre-trained model approximates the Bayesian posterior predictive distribution under that prior, and its training loss is the cross-entropy against the generating process. A richer prior, with more varied dependence structure and noise levels, exposes the model to a wider range of information patterns. See also causal inference for how graph structure shapes dependence.

In-context learning

In in-context learning, the model reads labeled rows and predicts for new rows without updating weights. This can be viewed as amortized inference: a single forward pass approximates the Bayesian update that a conventional method would compute by fitting. The information carried by the context grows with the number of rows, with diminishing returns once the conditional distribution is well pinned down. This explains why more context helps strongly at first and then levels off.

Missing values and imputation

A missing cell hides information. The uncertainty about its true value is the conditional entropy given the observed cells in the row and the patterns in other rows. A good imputation model reduces this entropy as far as the data allows, and a good one reports the remaining uncertainty rather than a single guess. Training by hiding cells and reconstructing them, as in masked cell modeling and with imputation networks, maximizes a lower bound on the mutual information between hidden and visible cells. When values are missing not at random, the pattern of missingness is itself informative, and an explicit mask feature lets the model use it.

Anomaly detection

Surprisal is a natural anomaly score. If a model assigns a very low probability to a row or a cell, that observation has high information content and is a candidate anomaly. Anomaly detection methods based on density estimation use exactly this. Related methods such as isolation forest approximate rarity through how quickly a point is isolated, which links to the idea that rare points need fewer splits, or in coding terms, longer codes.

Self-supervised and contrastive objectives

Contrastive learning, including InfoNCE and SimCLR-style methods, maximizes a lower bound on mutual information between two views of the same example. For tables, views can be created by masking or perturbing columns. The contrastive learning and self-supervised learning pages describe the general method. The information view explains why augmentations matter: they decide which information is kept as shared signal and which is discarded as nuisance.

Continual learning and KL regularization

When a model is updated on new data, a KL penalty between the old and new predictive distributions limits drift. Distillation-based methods such as learning without forgetting use this idea. Weight-space methods like elastic weight consolidation have a link to Fisher information, a curvature measure that comes from the second-order behavior of KL divergence. The same family of ideas appears in trust-region policy optimization such as TRPO and in preference tuning such as DPO, where a KL term holds a model near a reference.

Tokenization and column encoding

Choosing bin edges for a numeric column is a rate-distortion problem. Quantile bins give each bin equal probability, which maximizes the entropy of the binned value and so retains the most information for a fixed number of bins. This is one reason quantile-based encoding is common. For text cells, tokenizer training is a compression problem in the same sense as source coding.

Sector and semantic classification

Assigning a table to a domain or sector is a classification task. The entropy of the predicted class distribution shows how confident the model is, and a high-entropy prediction can trigger review. Heads such as semantic sector heads benefit from calibrated probabilities, where cross-entropy training plus calibration give confidence values that can be trusted.

Summary of the mapping

Results of these choices are summarized in /model/benchmarks.

Practical guidance and best practices

Choose the right quantity

• Use cross-entropy for probabilistic classification and for any model that outputs a distribution.
• Use KL divergence to compare a model with a reference or to keep an updated model close to its predecessor.
• Use mutual information for screening features that may relate non-linearly to a target.
• Use conditional entropy or information gain when reasoning about splits or about what a new column adds.
• Use surprisal for anomaly scoring, then calibrate thresholds on held-out data.

Estimating information measures

• For discrete variables with enough data, count-based estimates are fine. Add smoothing for rare outcomes.
• For continuous variables, prefer nearest-neighbor estimators over crude binning, or use consistent binning for all variables compared.
• Always compare against a baseline from shuffled labels, so you know what noise looks like.
• Report uncertainty with bootstrap intervals, since estimates can be unstable.

Working with logs and numbers

• Clip probabilities away from zero before taking logs.
• Use log-softmax and fused loss functions instead of separate softmax and log.
• Work in log space for products of many probabilities.
• Keep units consistent: all bits or all nats.

Evaluation habits

Regularization with KL terms

• In variational autoencoders, weight the KL term carefully. See VAE and autoencoders.
• Anneal the weight from zero to avoid collapse early in training.
• In preference tuning and policy optimization, set the KL coefficient so the model improves without drifting too far from its reference.

For tabular pipelines

• Rank columns by mutual information with the target as a first screen, then confirm with model-based importance.
• Remove columns that are near-duplicates, using mutual information between columns.
• Log per-column entropy and missingness over time, and alert on changes.
• Quantile-bin skewed numeric columns when using bin-based encoders.
• Keep a record of how each column was encoded, using a data catalog, so training and serving agree, and track structural changes through schema evolution.

Reading the numbers: a quick reference

Monitoring drift with divergence

A simple way to watch a deployed table pipeline is to compare the distribution of each column now with a reference window.

import numpy as np

def binned_probs(x, edges):
    counts, _ = np.histogram(x, bins=edges)
    p = counts + 1e-6          # smoothing avoids zeros
    return p / p.sum()

def js_divergence(p, q):
    m = 0.5 * (p + q)
    def kl(a, b):
        return float((a * np.log(a / b)).sum())
    return 0.5 * kl(p, m) + 0.5 * kl(q, m)

edges = np.quantile(reference_column, np.linspace(0, 1, 11))
score = js_divergence(binned_probs(reference_column, edges),
                      binned_probs(current_column, edges))

Jensen-Shannon is used here because it is symmetric and bounded, which makes thresholds easier to set. Alerts on such scores pair naturally with schema drift detection.

Label smoothing and calibration

Cross-entropy rewards pushing the probability of the correct class toward one. If left unchecked, a model can become overconfident. Two common remedies are label smoothing, which replaces hard targets with slightly softened ones, and temperature scaling after training, which rescales logits to match observed frequencies. Both are small changes with a clear information meaning: they raise the entropy of predictions where the model has no good reason to be certain.

Bits per column as a budget

When designing a table for modeling, think of each column as having a budget of useful bits. A binary flag that is true 1 percent of the time carries at most 0.08 bits per row. A near-unique identifier may carry many bits, but they are about identity, not about the target, and they invite overfitting. Looking at entropy together with mutual information with the target helps separate columns that are rich and relevant from those that are rich and useless.

A short checklist before using an information measure

1. Decide what question you are asking: dependence, difference, uncertainty or cost.
2. Match the measure to the question using the table above.
3. Check whether the variables are discrete or continuous and pick an estimator to match.
4. Add smoothing or clipping where zero probabilities can appear.
5. Run a shuffled baseline to learn what noise looks like.
6. Report units and the direction of any asymmetric measure.
7. Interpret the result as evidence, not proof of causation.

Why the log appears everywhere

The logarithm is not arbitrary. It is the only function that makes information add up for independent events: the surprise of two independent events together is the sum of their surprises. That additivity is why entropies of independent columns add, why log-likelihoods of independent rows add, and why training losses are sums over examples. It also keeps numbers manageable, since probabilities of long sequences shrink to tiny values that would underflow on a computer.

Common mistakes

• Treating KL divergence as a symmetric distance.
• Forgetting that entropy of continuous variables is relative and can be negative.
• Estimating mutual information from small samples and trusting the number.
• Comparing perplexities from models with different tokenizers.
• Using accuracy alone, which hides poor probability estimates.
• Letting predicted probabilities hit exactly zero or one, producing infinite loss.
• Reading mutual information as causation.
• Applying binning inconsistently when comparing information across columns.
• Ignoring the direction of KL when choosing between forward and reverse forms.
• Using a large KL weight too early in training and getting collapsed representations.
• Treating high entropy as a sign of useful information. Noise also has high entropy.
• Mixing units, such as bits in one place and nats in another, when comparing numbers.
• Assuming that compression always preserves what matters for the task.

Frequently asked questions

What is information theory in simple terms?

Information theory is the mathematics of measuring and communicating information. It says that the information in a message depends on how surprising it is, and that the average surprise of a source is its entropy. This gives precise limits on compression and reliable communication, and it supplies the formulas behind many machine learning losses.

What is entropy?

Entropy is the average amount of surprise in a random outcome. It is highest when all outcomes are equally likely and zero when one outcome is certain. In practice it tells you how many bits are needed on average to describe an outcome, and how uncertain a prediction or a data column is.

Why is cross-entropy used as a loss function?

Cross-entropy measures how well a model's predicted probabilities describe the true data. Minimizing it is equivalent to maximum likelihood estimation, and it pushes the predicted distribution toward the true one. It also gives strong gradients when the model is confidently wrong, which speeds learning compared with squared error for classification.

What is KL divergence used for?

KL divergence measures how much one probability distribution differs from another, in terms of extra bits needed when the wrong one is used. It appears as a regularizer in variational autoencoders, as a constraint in policy optimization, as a term in preference tuning and in knowledge distillation. It is not symmetric and is not a true distance.

How is mutual information different from correlation?

Correlation measures only linear association, while mutual information measures any statistical dependence, including curved and complex relationships. A variable can have zero correlation with a target and still carry a lot of information about it. The price is that mutual information is harder to estimate reliably from data.

What does perplexity mean?

Perplexity is the exponential of cross-entropy. It can be read as the effective number of equally likely choices the model is hesitating between at each step. Lower values mean the model is less surprised by the data. Values are only comparable when the units of prediction, such as the tokenizer, are the same.

How does information theory relate to compression and tokenizers?

The entropy of a source is the lower limit on lossless compression, so good codes give short representations to frequent symbols. Subword tokenizers like BPE act as a practical step in that direction by merging frequent pairs, which shortens sequences. The link is conceptual and approximate, since tokenizers also care about model learnability.

How is information theory used in tabular foundation models?

It is used in the training losses, such as cross-entropy for classes and binned regression targets, in contrastive and masked-cell objectives that relate to mutual information, in uncertainty measures for imputation and in surprisal-based anomaly scores. It also informs how numeric columns are binned and how feature relevance is judged.

Can mutual information be negative?

No. Mutual information is always zero or positive, and it is zero exactly when the variables are independent. Estimates from finite samples can come out slightly negative because of estimation error, which is a sign that the estimate is within noise of zero.

Does more information always mean a better model?

Not necessarily. Information that does not relate to the target is noise, and a model can overfit to it. What matters is information relevant to the task. The information bottleneck view says a good representation keeps what predicts the target and discards the rest.

Glossary

• Bit: the unit of information when logarithms use base 2.
• Channel capacity: the maximum rate of reliable communication over a noisy channel.
• Conditional entropy: the uncertainty left in one variable after observing another.
• Cross-entropy: the expected surprisal of data from one distribution under another distribution.
• Differential entropy: the continuous analog of entropy.
• Entropy: the average surprisal of a distribution.
• Fisher information: a measure of curvature of the likelihood, linked to local KL divergence.
• Information bottleneck: a principle of compressing inputs while preserving information about a target.
• Information gain: the reduction in entropy from an observation or a split.
• InfoNCE: a contrastive loss that lower-bounds mutual information.
• Jensen-Shannon divergence: a symmetric, bounded relative of KL divergence.
• KL divergence: the extra cost of using one distribution in place of another.
• Mutual information: the information shared between two variables.
• Nat: the unit of information when logarithms use base e.
• Perplexity: the exponential of cross-entropy.
• Rate-distortion: the trade-off between bits used and error allowed.
• Surprisal: the negative log probability of an event.

Summary

Information theory gives a precise way to measure uncertainty and dependence. Entropy is average surprise, cross-entropy is the cost of using a wrong model, KL divergence is the gap between them and mutual information measures how much one variable reveals about another. These quantities explain why cross-entropy is the standard training loss, why contrastive objectives work, why compression and tokenization are linked and how to reason about features, missing data and anomalies. For tabular foundation models, they shape loss design, pre-training on synthetic tables, in-context prediction, imputation uncertainty and anomaly scores. Related reading includes Probability, Variational inference, Contrastive learning and /model/architecture.

Related

It belongs to Foundations in my stack, under AI and machine learning. Others in the same group: