Özer Cem KELAHMETKELAHMET
Topics · MLOps and AI infrastructure

Training infrastructure: GPUs, clusters, distributed training and checkpoints

Training large models is an infrastructure problem as much as a modelling one. This page covers the hardware, the distributed training stack and the practices that keep long runs efficient.

Overview

A training cluster is GPUs joined by fast networks, a scheduler that hands them to jobs, storage that feeds data and saves checkpoints, and software that splits the model across devices. Efficiency is measured by model FLOPs utilisation: the share of the GPUs' peak arithmetic that the run actually uses.

Hardware

Component Role Examples
Accelerators The arithmetic NVIDIA H100, H200, B200, Cloud TPU
Intra-node links GPU to GPU inside a server NVLink
Inter-node network Server to server InfiniBand, RoCE
Storage Data loading and checkpoints Parallel file systems, Amazon S3

The software stack

Layer Tools
Kernels CUDA, Triton, FlashAttention
Communication NCCL
Frameworks PyTorch, JAX
Distributed training FSDP, DeepSpeed ZeRO, Megatron-LM, Colossal-AI
Orchestration Kubernetes, Slurm, Ray

Planning a run

Estimate memory per GPU (about 16 bytes per parameter for weights and optimiser state, plus activations), choose the parallelism plan, and estimate time as 6 × parameters × tokens divided by the cluster's effective FLOPs. The model sizing and parallelism planner simulations do this interactively.

Reliability

At thousands of GPUs, hardware fails daily. Save checkpoints often enough that a failure costs minutes, not days; restore the data loader position and random states; detect slow or faulty nodes automatically. The run monitor simulation shows checkpoints and rollbacks.

Cost control

Use spot or pre-emptible capacity for fault-tolerant jobs, right-size clusters to the parallelism plan, profile to raise utilisation, and stop runs early when small-scale experiments predict poor results. Scaling laws let you test ideas cheaply before committing a large budget.

Frequently asked questions

What utilisation is good?

Around 40 to 55% model FLOPs utilisation is typical for large transformer training; well-tuned runs go higher.

Do I need InfiniBand?

For multi-node training of large models, a fast interconnect is essential; for single-node fine-tuning it is not.

Summary

Fast accelerators and networks, a distributed training library, careful memory planning, frequent checkpoints and constant attention to utilisation and cost make large training runs work.

Back to the topic

This page is part of MLOps and AI infrastructure, which maps every concept, tool, simulation and article on the subject.

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open