# Training infrastructure: GPUs, clusters, distributed training and checkpoints

Training large models is an infrastructure problem as much as a modelling one. This page covers the hardware, the distributed training stack and the practices that keep long runs efficient.

## Overview

A training cluster is GPUs joined by fast networks, a scheduler that hands them to jobs, storage that feeds data and saves checkpoints, and software that splits the model across devices. Efficiency is measured by model FLOPs utilisation: the share of the GPUs' peak arithmetic that the run actually uses.

## Hardware

| Component | Role | Examples |
|---|---|---|
| Accelerators | The arithmetic | [NVIDIA H100](/tools/nvidia-h100), [H200](/tools/nvidia-h200), [B200](/tools/nvidia-b200), [Cloud TPU](/tools/cloud-tpu) |
| Intra-node links | GPU to GPU inside a server | NVLink |
| Inter-node network | Server to server | InfiniBand, RoCE |
| Storage | Data loading and checkpoints | Parallel file systems, [Amazon S3](/tools/amazon-s3) |

## The software stack

| Layer | Tools |
|---|---|
| Kernels | [CUDA](/tools/cuda), [Triton](/tools/triton), [FlashAttention](/tools/flashattention) |
| Communication | [NCCL](/tools/nccl) |
| Frameworks | [PyTorch](/tools/pytorch), [JAX](/tools/jax) |
| Distributed training | [FSDP](/tools/fsdp), [DeepSpeed ZeRO](/tools/deepspeed-zero), [Megatron-LM](/tools/megatron-lm), [Colossal-AI](/tools/colossal-ai) |
| Orchestration | [Kubernetes](/tools/kubernetes), Slurm, [Ray](/tools/ray-tune) |

## Planning a run

Estimate memory per GPU (about 16 bytes per parameter for weights and optimiser state, plus activations), choose the parallelism plan, and estimate time as 6 × parameters × tokens divided by the cluster's effective FLOPs. The [model sizing](/ai-simulations/train-model-sizing) and [parallelism planner](/ai-simulations/train-parallelism-planner) simulations do this interactively.

## Reliability

At thousands of GPUs, hardware fails daily. Save checkpoints often enough that a failure costs minutes, not days; restore the data loader position and random states; detect slow or faulty nodes automatically. The [run monitor simulation](/ai-simulations/train-run-monitor) shows checkpoints and rollbacks.

## Cost control

Use spot or pre-emptible capacity for fault-tolerant jobs, right-size clusters to the parallelism plan, profile to raise utilisation, and stop runs early when small-scale experiments predict poor results. [Scaling laws](/ai-simulations/train-scaling-laws) let you test ideas cheaply before committing a large budget.

## Frequently asked questions

### What utilisation is good?

Around 40 to 55% model FLOPs utilisation is typical for large transformer training; well-tuned runs go higher.

### Do I need InfiniBand?

For multi-node training of large models, a fast interconnect is essential; for single-node fine-tuning it is not.

## Summary

Fast accelerators and networks, a distributed training library, careful memory planning, frequent checkpoints and constant attention to utilisation and cost make large training runs work.

## Back to the topic

This page is part of **[MLOps and AI infrastructure](/topics/mlops-ai-infrastructure)**, which maps every concept, tool, simulation and article on the subject.
