# Deduplication

Detects exact and near duplicates in training data so the model does not memorize them and benchmarks stay clean.

## What it is used for

Protecting against contamination and wasted compute.