Özer Cem KELAHMETKELAHMET
AI Simulations · Training from scratch

Scaling laws: how to split a compute budget between parameters and tokens

For a fixed compute budget, you can train a large model on few tokens or a small model on many. Scaling laws predict the loss of each choice before training. This simulation plots them with the published Chinchilla fit.

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

Each curve is one compute budget; moving along it trades parameters for tokens at the same cost. The dot marks the lowest loss, the ring marks the model size you chose. Choose a number of tokens the model will serve and the side panel finds the cheapest model over training plus inference for the same loss.

Things to try

  • Move your model to the dot: about 20 tokens per parameter.
  • Make the model ten times bigger at the same budget: the loss rises because it sees too few tokens.
  • Raise the budget by ten: the best model grows by about three, the best token count too.
  • Count 100 T served tokens: the cheapest model is smaller and trained on far more tokens.

How it works in real systems

The curves use the law L(N, D) = E + A/N^α + B/D^β from Hoffmann et al. (2022), with the constants re-estimated by Besiroglu et al. (2024), which reproduce the paper's 20-tokens-per-parameter result. Labs fit their own versions on small runs before committing to a large one, and since inference now dominates the cost of popular models, many are trained on several hundred tokens per parameter on purpose.

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · The Chinchilla law and the compute-optimal size

import numpy as np
E, A, B, a, b = 1.8172, 482.01, 2085.43, 0.3478, 0.3658   # Hoffmann et al. 2022, re-fitted by Besiroglu et al. 2024
L = lambda N, D: E + A / N**a + B / D**b

def optimal(C):
    Ns = np.logspace(7, 13.5, 4000)
    Ds = C / (6 * Ns)
    i = np.argmin(L(Ns, Ds))
    return Ns[i], Ds[i], L(Ns[i], Ds[i])

for C in (1e21, 1e23, 1e25):
    N, D, l = optimal(C)
    print(f"C={C:.0e}: N={N/1e9:.1f}B D={D/1e9:.0f}B tokens ({D/N:.0f}/param) loss {l:.3f}")

Step 2 · Fit your own law on small runs

from scipy.optimize import curve_fit
# (N, D, final loss) from a sweep of small runs you trained yourself
runs = np.array([[2e7, 4e8, 3.95], [5e7, 1e9, 3.52], [1e8, 2e9, 3.25], [2e8, 4e9, 3.03], [5e7, 4e9, 3.38], [2e8, 1e9, 3.30]])
f = lambda X, E, A, B, a, b: E + A / X[0]**a + B / X[1]**b
p, _ = curve_fit(f, runs[:, :2].T, runs[:, 2], p0=[1.8, 480, 2000, 0.35, 0.37], maxfev=20000)
print(dict(zip("E A B alpha beta".split(), p.round(3))))

More simulations

← Model sizing · All AI Simulations · Numeric precision →

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open