Glossary
What is FSDP? Definition and meaning
PyTorch's native method for sharding parameters, gradients and optimizer states across workers.
Definition
FSDP: PyTorch's native method for sharding parameters, gradients and optimizer states across workers.
In short: Fully sharded data parallel in PyTorch.
When it is used
Multi-GPU training of large models with native PyTorch tooling.
Learn more
The full guide to FSDP explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.
Previous term: Format-preserving encryption · Next term: FT-Transformer