Özer Cem KELAHMETKELAHMET
Glossary

What is Tensor parallelism? Definition and meaning

Divides individual weight matrices so several devices compute one layer together.

Definition

Tensor parallelism: Divides individual weight matrices so several devices compute one layer together.

In short: Splitting the matrices inside a layer across GPUs.

When it is used

Training layers too large for a single GPU.

Learn more

The full guide to Tensor parallelism explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.

Previous term: Temporal · Next term: TensorFlow

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open