Özer Cem KELAHMETKELAHMET
Topics · Deep learning

Training at scale: precision, parallelism, memory and scaling laws

A large model does not fit on one GPU and takes weeks to train. This page explains the techniques that make it possible and the scaling laws that decide how big to go.

Overview

Training at scale is about three limits: memory, speed and budget. Lower-precision numbers save memory and time, parallelism spreads the work across many GPUs, and scaling laws decide how to spend a fixed amount of compute between model size and data.

Memory: where the bytes go

In mixed-precision training with Adam, each parameter needs about 16 bytes: 16-bit weights and gradients, a 32-bit master copy and two optimiser moments. An 8-billion-parameter model therefore needs about 128 GB before activations, more than one 80 GB GPU. Activations grow with batch size and sequence length; gradient checkpointing recomputes them instead of storing them. The model sizing simulation does this arithmetic for any architecture.

Precision

Format Use Note
FP32 Master weights, sensitive sums Full range and precision
BF16 Default compute format FP32's range, less precision, no loss scaling needed
FP16 Older default Needs loss scaling to avoid underflow
FP8 Newest GPUs Per-tensor or per-block scales

See mixed precision and the numeric precision simulation.

Parallelism

Strategy Splits Communication Read more
Data parallel The batch; each GPU has a full copy Gradients once per step Data parallelism
Sharded data parallel Weights, gradients and optimiser state Gather and scatter per layer FSDP, ZeRO
Tensor parallel Each layer's matrices Every layer, inside a node Tensor parallelism
Pipeline parallel The stack of layers Between stages; idle "bubble" Pipeline parallelism

Large runs combine all four, usually with tensor parallelism inside an eight-GPU node and the others across nodes, using libraries such as Megatron-LM and NCCL. The parallelism planner shows memory per GPU for each plan.

Scaling laws

Loss falls predictably as parameters and training tokens grow. For a fixed compute budget, the Chinchilla result puts the optimum at about 20 tokens per parameter; because serving costs dominate for popular models, many are now trained far beyond that on purpose. The scaling laws simulation plots both views.

Keeping a long run alive

Runs log loss, gradient norm and learning rate every step, save checkpoints regularly, skip batches that cause spikes and resume after hardware failures. The run monitor simulation reproduces these events.

Frequently asked questions

How much compute does training take?

About 6 floating-point operations per parameter per token. An 8-billion-parameter model on 15 trillion tokens needs roughly 7×10²³ operations.

Which GPUs are used?

Mostly NVIDIA data-centre GPUs such as the H100, H200 and B200, and Google TPUs.

Summary

Lower precision saves memory, parallelism spreads the work, checkpointing keeps runs alive, and scaling laws tell you how big a model your budget deserves. The nine-step training track covers each stage interactively.

Back to the topic

This page is part of Deep learning, which maps every concept, tool, simulation and article on the subject.

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open