Özer Cem KELAHMETKELAHMET
Glossary

What is Information theory? Definition and meaning

Entropy, mutual information and KL divergence, which underlie many training losses.

Definition

Information theory: Entropy, mutual information and KL divergence, which underlie many training losses.

In short: Measuring information, entropy and divergence.

When it is used

Loss design, contrastive objectives and compression arguments.

Learn more

The full guide to Information theory explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.

Previous term: Informatica EDC · Next term: Infrastructure as code

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open