# Tensor parallelism

Divides individual weight matrices so several devices compute one layer together.

## What it is used for

Training layers too large for a single GPU.