Numeric precision: fp32, bf16, fp16 and fp8 in training, bit by bit
Large models train in 16-bit or even 8-bit numbers to save memory and time. Each format trades range for precision. This simulation rounds numbers bit-exactly in each format and shows what happens to a whole gradient tensor.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The table shows how your number is stored: the bits, the value that survives and the relative error. The histogram shows the magnitudes of 10,000 gradient values against the range each format can represent; the side panel counts how many underflow to zero or overflow. Loss scaling multiplies the values before the cast.
Things to try
- Type 0.1: no binary format stores it exactly; bf16 is off in the third digit.
- Type 70000: fp16 and both fp8 formats overflow; bf16 rounds it.
- Keep small gradients and look at fp16: a large share flushes to zero.
- Raise loss scaling to about ×65,536: fp16 keeps them, until outliers start to overflow.
How it works in real systems
Mixed-precision training keeps an fp32 master copy of the weights and computes in bf16; fp16 needs dynamic loss scaling. FP8 training, supported on recent GPUs through libraries such as NVIDIA Transformer Engine, keeps a scale for each tensor or block and uses E4M3 for weights and activations and E5M2 for gradients.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · See the formats in PyTorch
import torch
x = torch.tensor([0.1, 70000.0, 1e-8])
for dt in (torch.float32, torch.bfloat16, torch.float16, torch.float8_e4m3fn, torch.float8_e5m2):
y = x.to(dt).to(torch.float32)
print(f"{str(dt):22s}", y.tolist(), torch.finfo(dt).max, torch.finfo(dt).tiny)
Step 2 · Mixed precision with loss scaling
model = torch.nn.Linear(1024, 1024).cuda()
opt = torch.optim.AdamW(model.parameters(), lr=1e-4)
scaler = torch.amp.GradScaler("cuda") # only needed for float16
for step in range(100):
x = torch.randn(32, 1024, device="cuda")
with torch.autocast("cuda", dtype=torch.float16):
loss = model(x).pow(2).mean()
scaler.scale(loss).backward() # multiply before backward so small grads survive
scaler.unscale_(opt)
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
scaler.step(opt); scaler.update(); opt.zero_grad()
print("scale now", scaler.get_scale())
With dtype=torch.bfloat16 drop the scaler: bf16 has fp32's exponent range.