Parallelism planner: fit a run on GPUs with data, tensor and pipeline parallelism
A large model does not fit on one GPU, so training splits it. Data parallelism copies it, tensor parallelism splits each layer, pipeline parallelism splits the stack of layers, and ZeRO shards the training state. This planner shows what each choice does to memory and idle time.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The bar shows the memory on the busiest GPU against 80 GB. The grid shows the arrangement: each row is a pipeline stage and each square a group of tensor-parallel GPUs. The side panel reports whether it fits, the pipeline bubble, the number of micro-batches and the gradient traffic.
Things to try
- Set ZeRO to 0 on the 70B model: optimizer state alone overflows the GPU.
- Move to ZeRO 3: each replica shards weights, gradients and optimizer state.
- Use 16 pipeline stages with a small global batch: the bubble grows.
- Raise the context to 32k without recompute: activations take over.
How it works in real systems
Frameworks such as Megatron-LM, DeepSpeed and PyTorch FSDP implement these strategies, and large runs combine all of them. Tensor parallelism usually stays inside one eight-GPU node because it communicates at every layer; pipeline and data parallelism span nodes.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Fully sharded data parallel (ZeRO 3) in PyTorch
# torchrun --nproc_per_node=8 train_fsdp.py
import torch, torch.distributed as dist
from torch.distributed.fsdp import fully_shard, MixedPrecisionPolicy
from transformers import AutoConfig, AutoModelForCausalLM
dist.init_process_group("nccl"); torch.cuda.set_device(dist.get_rank() % 8)
model = AutoModelForCausalLM.from_config(AutoConfig.from_pretrained("Qwen/Qwen2.5-1.5B"))
mp = MixedPrecisionPolicy(param_dtype=torch.bfloat16, reduce_dtype=torch.float32)
for layer in model.model.layers:
fully_shard(layer, mp_policy=mp) # shard weights, grads and optimizer state per layer
fully_shard(model, mp_policy=mp)
opt = torch.optim.AdamW(model.parameters(), lr=3e-4)
Step 2 · Plan memory before launching
def per_gpu_gb(N, gpus, tp, pp, zero, seq, micro, d, L, heads, recompute="selective"):
dp = gpus // (tp * pp); shard = N / (tp * pp)
w = 2 * shard / (dp if zero >= 3 else 1); g = 2 * shard / (dp if zero >= 2 else 1); o = 12 * shard / (dp if zero >= 1 else 1)
sbd = seq * micro * d
act_l = 2 * sbd if recompute == "full" else sbd * 34 / tp if recompute == "selective" else sbd * (34 + 5 * heads * seq / d) / tp
return (w + g + o + act_l * (L / pp) * pp) / 1e9 + 3
print(per_gpu_gb(70.6e9, 256, 8, 4, 1, 8192, 1, 8192, 80, 64))
More simulations
← Numeric precision · All AI Simulations · Run, spikes and checkpoints →