# Numeric precision: fp32, bf16, fp16 and fp8 in training, bit by bit

Large models train in 16-bit or even 8-bit numbers to save memory and time. Each format trades range for precision. This simulation rounds numbers bit-exactly in each format and shows what happens to a whole gradient tensor.

<div data-sim="tfsprec"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

The table shows how your number is stored: the bits, the value that survives and the relative error. The histogram shows the magnitudes of 10,000 gradient values against the range each format can represent; the side panel counts how many underflow to zero or overflow. Loss scaling multiplies the values before the cast.

## Things to try

- Type 0.1: no binary format stores it exactly; bf16 is off in the third digit.
- Type 70000: fp16 and both fp8 formats overflow; bf16 rounds it.
- Keep small gradients and look at fp16: a large share flushes to zero.
- Raise loss scaling to about ×65,536: fp16 keeps them, until outliers start to overflow.

## How it works in real systems

Mixed-precision training keeps an fp32 master copy of the weights and computes in bf16; fp16 needs dynamic loss scaling. FP8 training, supported on recent GPUs through libraries such as NVIDIA Transformer Engine, keeps a scale for each tensor or block and uses E4M3 for weights and activations and E5M2 for gradients.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · See the formats in PyTorch

```python
import torch
x = torch.tensor([0.1, 70000.0, 1e-8])
for dt in (torch.float32, torch.bfloat16, torch.float16, torch.float8_e4m3fn, torch.float8_e5m2):
    y = x.to(dt).to(torch.float32)
    print(f"{str(dt):22s}", y.tolist(), torch.finfo(dt).max, torch.finfo(dt).tiny)
```

### Step 2 · Mixed precision with loss scaling

```python
model = torch.nn.Linear(1024, 1024).cuda()
opt = torch.optim.AdamW(model.parameters(), lr=1e-4)
scaler = torch.amp.GradScaler("cuda")            # only needed for float16
for step in range(100):
    x = torch.randn(32, 1024, device="cuda")
    with torch.autocast("cuda", dtype=torch.float16):
        loss = model(x).pow(2).mean()
    scaler.scale(loss).backward()                # multiply before backward so small grads survive
    scaler.unscale_(opt)
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    scaler.step(opt); scaler.update(); opt.zero_grad()
print("scale now", scaler.get_scale())
```
With `dtype=torch.bfloat16` drop the scaler: bf16 has fp32's exponent range.

<div class="cl-end"></div>

## More simulations

[← Scaling laws](/ai-simulations/train-scaling-laws) · [All AI Simulations](/ai-simulations) · [Parallelism planner →](/ai-simulations/train-parallelism-planner)
