Training at scale: precision, parallelism, memory and scaling laws
A large model does not fit on one GPU and takes weeks to train. This page explains the techniques that make it possible and the scaling laws that decide how big to go.
Overview
Training at scale is about three limits: memory, speed and budget. Lower-precision numbers save memory and time, parallelism spreads the work across many GPUs, and scaling laws decide how to spend a fixed amount of compute between model size and data.
Memory: where the bytes go
In mixed-precision training with Adam, each parameter needs about 16 bytes: 16-bit weights and gradients, a 32-bit master copy and two optimiser moments. An 8-billion-parameter model therefore needs about 128 GB before activations, more than one 80 GB GPU. Activations grow with batch size and sequence length; gradient checkpointing recomputes them instead of storing them. The model sizing simulation does this arithmetic for any architecture.
Precision
| Format | Use | Note |
|---|---|---|
| FP32 | Master weights, sensitive sums | Full range and precision |
| BF16 | Default compute format | FP32's range, less precision, no loss scaling needed |
| FP16 | Older default | Needs loss scaling to avoid underflow |
| FP8 | Newest GPUs | Per-tensor or per-block scales |
See mixed precision and the numeric precision simulation.
Parallelism
| Strategy | Splits | Communication | Read more |
|---|---|---|---|
| Data parallel | The batch; each GPU has a full copy | Gradients once per step | Data parallelism |
| Sharded data parallel | Weights, gradients and optimiser state | Gather and scatter per layer | FSDP, ZeRO |
| Tensor parallel | Each layer's matrices | Every layer, inside a node | Tensor parallelism |
| Pipeline parallel | The stack of layers | Between stages; idle "bubble" | Pipeline parallelism |
Large runs combine all four, usually with tensor parallelism inside an eight-GPU node and the others across nodes, using libraries such as Megatron-LM and NCCL. The parallelism planner shows memory per GPU for each plan.
Scaling laws
Loss falls predictably as parameters and training tokens grow. For a fixed compute budget, the Chinchilla result puts the optimum at about 20 tokens per parameter; because serving costs dominate for popular models, many are now trained far beyond that on purpose. The scaling laws simulation plots both views.
Keeping a long run alive
Runs log loss, gradient norm and learning rate every step, save checkpoints regularly, skip batches that cause spikes and resume after hardware failures. The run monitor simulation reproduces these events.
Frequently asked questions
How much compute does training take?
About 6 floating-point operations per parameter per token. An 8-billion-parameter model on 15 trillion tokens needs roughly 7×10²³ operations.
Which GPUs are used?
Mostly NVIDIA data-centre GPUs such as the H100, H200 and B200, and Google TPUs.
Summary
Lower precision saves memory, parallelism spreads the work, checkpointing keeps runs alive, and scaling laws tell you how big a model your budget deserves. The nine-step training track covers each stage interactively.
Back to the topic
This page is part of Deep learning, which maps every concept, tool, simulation and article on the subject.