Training infrastructure: GPUs, clusters, distributed training and checkpoints
Training large models is an infrastructure problem as much as a modelling one. This page covers the hardware, the distributed training stack and the practices that keep long runs efficient.
Overview
A training cluster is GPUs joined by fast networks, a scheduler that hands them to jobs, storage that feeds data and saves checkpoints, and software that splits the model across devices. Efficiency is measured by model FLOPs utilisation: the share of the GPUs' peak arithmetic that the run actually uses.
Hardware
| Component | Role | Examples |
|---|---|---|
| Accelerators | The arithmetic | NVIDIA H100, H200, B200, Cloud TPU |
| Intra-node links | GPU to GPU inside a server | NVLink |
| Inter-node network | Server to server | InfiniBand, RoCE |
| Storage | Data loading and checkpoints | Parallel file systems, Amazon S3 |
The software stack
| Layer | Tools |
|---|---|
| Kernels | CUDA, Triton, FlashAttention |
| Communication | NCCL |
| Frameworks | PyTorch, JAX |
| Distributed training | FSDP, DeepSpeed ZeRO, Megatron-LM, Colossal-AI |
| Orchestration | Kubernetes, Slurm, Ray |
Planning a run
Estimate memory per GPU (about 16 bytes per parameter for weights and optimiser state, plus activations), choose the parallelism plan, and estimate time as 6 × parameters × tokens divided by the cluster's effective FLOPs. The model sizing and parallelism planner simulations do this interactively.
Reliability
At thousands of GPUs, hardware fails daily. Save checkpoints often enough that a failure costs minutes, not days; restore the data loader position and random states; detect slow or faulty nodes automatically. The run monitor simulation shows checkpoints and rollbacks.
Cost control
Use spot or pre-emptible capacity for fault-tolerant jobs, right-size clusters to the parallelism plan, profile to raise utilisation, and stop runs early when small-scale experiments predict poor results. Scaling laws let you test ideas cheaply before committing a large budget.
Frequently asked questions
What utilisation is good?
Around 40 to 55% model FLOPs utilisation is typical for large transformer training; well-tuned runs go higher.
Do I need InfiniBand?
For multi-node training of large models, a fast interconnect is essential; for single-node fine-tuning it is not.
Summary
Fast accelerators and networks, a distributed training library, careful memory planning, frequent checkpoints and constant attention to utilisation and cost make large training runs work.
Back to the topic
This page is part of MLOps and AI infrastructure, which maps every concept, tool, simulation and article on the subject.