Scaling laws: how to split a compute budget between parameters and tokens
For a fixed compute budget, you can train a large model on few tokens or a small model on many. Scaling laws predict the loss of each choice before training. This simulation plots them with the published Chinchilla fit.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
Each curve is one compute budget; moving along it trades parameters for tokens at the same cost. The dot marks the lowest loss, the ring marks the model size you chose. Choose a number of tokens the model will serve and the side panel finds the cheapest model over training plus inference for the same loss.
Things to try
- Move your model to the dot: about 20 tokens per parameter.
- Make the model ten times bigger at the same budget: the loss rises because it sees too few tokens.
- Raise the budget by ten: the best model grows by about three, the best token count too.
- Count 100 T served tokens: the cheapest model is smaller and trained on far more tokens.
How it works in real systems
The curves use the law L(N, D) = E + A/N^α + B/D^β from Hoffmann et al. (2022), with the constants re-estimated by Besiroglu et al. (2024), which reproduce the paper's 20-tokens-per-parameter result. Labs fit their own versions on small runs before committing to a large one, and since inference now dominates the cost of popular models, many are trained on several hundred tokens per parameter on purpose.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · The Chinchilla law and the compute-optimal size
import numpy as np
E, A, B, a, b = 1.8172, 482.01, 2085.43, 0.3478, 0.3658 # Hoffmann et al. 2022, re-fitted by Besiroglu et al. 2024
L = lambda N, D: E + A / N**a + B / D**b
def optimal(C):
Ns = np.logspace(7, 13.5, 4000)
Ds = C / (6 * Ns)
i = np.argmin(L(Ns, Ds))
return Ns[i], Ds[i], L(Ns[i], Ds[i])
for C in (1e21, 1e23, 1e25):
N, D, l = optimal(C)
print(f"C={C:.0e}: N={N/1e9:.1f}B D={D/1e9:.0f}B tokens ({D/N:.0f}/param) loss {l:.3f}")
Step 2 · Fit your own law on small runs
from scipy.optimize import curve_fit
# (N, D, final loss) from a sweep of small runs you trained yourself
runs = np.array([[2e7, 4e8, 3.95], [5e7, 1e9, 3.52], [1e8, 2e9, 3.25], [2e8, 4e9, 3.03], [5e7, 4e9, 3.38], [2e8, 1e9, 3.30]])
f = lambda X, E, A, B, a, b: E + A / X[0]**a + B / X[1]**b
p, _ = curve_fit(f, runs[:, :2].T, runs[:, 2], p0=[1.8, 480, 2000, 0.35, 0.37], maxfev=20000)
print(dict(zip("E A B alpha beta".split(), p.round(3))))