Glossary
What is DeepSpeed ZeRO? Definition and meaning
A set of memory optimizations that partition training state across devices so much larger models fit.
Definition
DeepSpeed ZeRO: A set of memory optimizations that partition training state across devices so much larger models fit.
In short: Sharding optimizer, gradient and parameter state across GPUs.
When it is used
Training and fine-tuning models that do not fit on one GPU.
Learn more
The full guide to DeepSpeed ZeRO explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.
Previous term: DeepSeek V4.1 · Next term: Delta Lake