Glossary
What is Data parallelism? Definition and meaning
Each device holds a full copy of the model, processes a slice of the batch, and gradients are averaged.
Definition
Data parallelism: Each device holds a full copy of the model, processes a slice of the batch, and gradients are averaged.
In short: Copying the model and splitting the data.
When it is used
The simplest way to scale training across GPUs.
Learn more
The full guide to Data parallelism explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.
Previous term: Data mesh · Next term: Data preprocessing