# What is Information theory? Definition and meaning

Entropy, mutual information and KL divergence, which underlie many training losses.

## Definition

**Information theory**: Entropy, mutual information and KL divergence, which underlie many training losses.

In short: Measuring information, entropy and divergence.

## When it is used

Loss design, contrastive objectives and compression arguments.

## Learn more

The full guide to **[Information theory](/tools/information-theory)** explains how it works, when to use it and how it compares with the alternatives, with examples. The [glossary](/glossary) lists every term on the site from A to Z.

Previous term: [Informatica EDC](/glossary/informatica-edc) · Next term: [Infrastructure as code](/glossary/infrastructure-as-code)
