Glossary
What is Tensor parallelism? Definition and meaning
Divides individual weight matrices so several devices compute one layer together.
Definition
Tensor parallelism: Divides individual weight matrices so several devices compute one layer together.
In short: Splitting the matrices inside a layer across GPUs.
When it is used
Training layers too large for a single GPU.
Learn more
The full guide to Tensor parallelism explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.
Previous term: Temporal · Next term: TensorFlow