Glossary
What is Megatron-LM? Definition and meaning
A library with tensor, pipeline and sequence parallelism that is tuned for NVIDIA GPU clusters.
Definition
Megatron-LM: A library with tensor, pipeline and sequence parallelism that is tuned for NVIDIA GPU clusters.
In short: NVIDIA's framework for training very large models.
When it is used
Large-scale pre-training across many GPUs and nodes.
Learn more
The full guide to Megatron-LM explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.
Previous term: Medallion architecture · Next term: Meta-learning