Glossary
What is Deduplication? Definition and meaning
Detects exact and near duplicates in training data so the model does not memorize them and benchmarks stay clean.
Definition
Deduplication: Detects exact and near duplicates in training data so the model does not memorize them and benchmarks stay clean.
In short: Removing repeated or near-identical data.
When it is used
Protecting against contamination and wasted compute.
Learn more
The full guide to Deduplication explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.
Previous term: Decision trees · Next term: DeepEval