Özer Cem KELAHMETKELAHMET
Glossary

What is Data parallelism? Definition and meaning

Each device holds a full copy of the model, processes a slice of the batch, and gradients are averaged.

Definition

Data parallelism: Each device holds a full copy of the model, processes a slice of the batch, and gradients are averaged.

In short: Copying the model and splitting the data.

When it is used

The simplest way to scale training across GPUs.

Learn more

The full guide to Data parallelism explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.

Previous term: Data mesh · Next term: Data preprocessing

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open