# Megatron-LM

A library with tensor, pipeline and sequence parallelism that is tuned for NVIDIA GPU clusters.

## What it is used for

Large-scale pre-training across many GPUs and nodes.