# What is Deduplication? Definition and meaning

Detects exact and near duplicates in training data so the model does not memorize them and benchmarks stay clean.

## Definition

**Deduplication**: Detects exact and near duplicates in training data so the model does not memorize them and benchmarks stay clean.

In short: Removing repeated or near-identical data.

## When it is used

Protecting against contamination and wasted compute.

## Learn more

The full guide to **[Deduplication](/tools/deduplication)** explains how it works, when to use it and how it compares with the alternatives, with examples. The [glossary](/glossary) lists every term on the site from A to Z.

Previous term: [Decision trees](/glossary/decision-trees) · Next term: [DeepEval](/glossary/deepeval)
