# What is Tensor parallelism? Definition and meaning

Divides individual weight matrices so several devices compute one layer together.

## Definition

**Tensor parallelism**: Divides individual weight matrices so several devices compute one layer together.

In short: Splitting the matrices inside a layer across GPUs.

## When it is used

Training layers too large for a single GPU.

## Learn more

The full guide to **[Tensor parallelism](/tools/tensor-parallelism)** explains how it works, when to use it and how it compares with the alternatives, with examples. The [glossary](/glossary) lists every term on the site from A to Z.

Previous term: [Temporal](/glossary/temporal) · Next term: [TensorFlow](/glossary/tensorflow)
