# Scaling laws: how to split a compute budget between parameters and tokens

For a fixed compute budget, you can train a large model on few tokens or a small model on many. Scaling laws predict the loss of each choice before training. This simulation plots them with the published Chinchilla fit.

<div data-sim="tfsscale"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

Each curve is one compute budget; moving along it trades parameters for tokens at the same cost. The dot marks the lowest loss, the ring marks the model size you chose. Choose a number of tokens the model will serve and the side panel finds the cheapest model over training plus inference for the same loss.

## Things to try

- Move your model to the dot: about 20 tokens per parameter.
- Make the model ten times bigger at the same budget: the loss rises because it sees too few tokens.
- Raise the budget by ten: the best model grows by about three, the best token count too.
- Count 100 T served tokens: the cheapest model is smaller and trained on far more tokens.

## How it works in real systems

The curves use the law L(N, D) = E + A/N^α + B/D^β from Hoffmann et al. (2022), with the constants re-estimated by Besiroglu et al. (2024), which reproduce the paper's 20-tokens-per-parameter result. Labs fit their own versions on small runs before committing to a large one, and since inference now dominates the cost of popular models, many are trained on several hundred tokens per parameter on purpose.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · The Chinchilla law and the compute-optimal size

```python
import numpy as np
E, A, B, a, b = 1.8172, 482.01, 2085.43, 0.3478, 0.3658   # Hoffmann et al. 2022, re-fitted by Besiroglu et al. 2024
L = lambda N, D: E + A / N**a + B / D**b

def optimal(C):
    Ns = np.logspace(7, 13.5, 4000)
    Ds = C / (6 * Ns)
    i = np.argmin(L(Ns, Ds))
    return Ns[i], Ds[i], L(Ns[i], Ds[i])

for C in (1e21, 1e23, 1e25):
    N, D, l = optimal(C)
    print(f"C={C:.0e}: N={N/1e9:.1f}B D={D/1e9:.0f}B tokens ({D/N:.0f}/param) loss {l:.3f}")
```

### Step 2 · Fit your own law on small runs

```python
from scipy.optimize import curve_fit
# (N, D, final loss) from a sweep of small runs you trained yourself
runs = np.array([[2e7, 4e8, 3.95], [5e7, 1e9, 3.52], [1e8, 2e9, 3.25], [2e8, 4e9, 3.03], [5e7, 4e9, 3.38], [2e8, 1e9, 3.30]])
f = lambda X, E, A, B, a, b: E + A / X[0]**a + B / X[1]**b
p, _ = curve_fit(f, runs[:, :2].T, runs[:, 2], p0=[1.8, 480, 2000, 0.35, 0.37], maxfev=20000)
print(dict(zip("E A B alpha beta".split(), p.round(3))))
```

<div class="cl-end"></div>

## More simulations

[← Model sizing](/ai-simulations/train-model-sizing) · [All AI Simulations](/ai-simulations) · [Numeric precision →](/ai-simulations/train-numeric-precision)
