From 0 to 100: build a small language model, end to end
Most explanations show one piece of a language model at a time. This project builds a whole one and ships it. A 1.4-million-parameter GPT was trained on the articles and guides of this site; here you walk through every stage of that work, from raw text to a cost estimate for serving it, and the model itself runs in your browser at every step.
Level IntermediateAbout 25 minRead first: Tokenisation, Attention, Evaluation, RAG types1 challenge in the Lab
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The bar at the top goes from 0 to 100% as you complete the eight stages. Each stage uses what the one before it produced, exactly as in a real project. Run 0 → 100 plays them all in order; the numbered tabs let you open any stage and work in it.
- Data. Every article and guide on this site was collected. Code, fragments shorter than ten words, exact duplicates and near duplicates were removed. The counts are the real ones from the run.
- Tokenizer. A byte-pair encoding tokenizer was trained on the cleaned text. Type anything to see the tokens the model receives, and the first merges it learned.
- Training. The recorded loss curves of the real training run: training loss on what it learns from, and held-out loss on text it never sees.
- The model. The trained weights run in your browser. You see the probability of every candidate next token, generate text with your own temperature and top-k, look at what each attention head attends to, and read the logit lens: the model's best guess after each of its four layers.
- Evaluation. Your browser scores a held-out passage token by token. Red tokens surprised the model; the average surprise is the loss and its exponential the perplexity.
- Retrieval (RAG). Your question goes to this site's search. The model then predicts the top passage with and without retrieved context, and you see how much the context lowers its loss.
- Agent. A tool-using loop: think, call a tool, observe, repeat. The planner is a short script so every step can be followed; the tools (search, tokenizer, the model, a calculator) are real.
- Deploy. A real speed measurement of this model on your device, then the arithmetic of serving larger models: memory at each precision, tokens per second and dollars per million tokens.
Things to try
- Run 0 → 100 once without touching anything, then go back to stage 4 and write your own start.
- In stage 2, paste a sentence in another language or a long technical word: it splits into many more tokens, because the tokenizer only saw this site's English.
- In stage 4, generate at temperature 0.3 and then 1.5 with the same start. Low temperature repeats itself; high temperature drifts into nonsense.
- In stage 4, watch the logit lens while you type: early layers often guess a frequent word, later layers a word that fits the sentence.
- In stage 6, ask about a topic the site covers and then about one it does not: the improvement from context shrinks when the passage does not match.
- In stage 8, set 70B at bf16 and compare it with 8B at int4: memory and cost move by an order of magnitude.
How it works in real systems
Large models go through the same stages at a vastly larger scale. Pre-training corpora are filtered with classifiers and deduplicated with MinHash across trillions of tokens; tokenizers have 32,000 to 256,000 entries; models have billions of parameters and are trained on thousands of GPUs for weeks. Evaluation uses held-out perplexity during training and task benchmarks afterwards. Retrieval and tool use are added around the model rather than inside it, and serving cost is driven by memory bandwidth, batch size and the KV cache as much as by raw compute. The shape of the work is the same as here; only the numbers change.
Key formulas
$$\mathcal L=-\frac1N\sum_{t=1}^{N}\log p_\theta(x_t\mid x_{<t}),\qquad \mathrm{PPL}=e^{\mathcal L}$$
The training objective and the evaluation number: the average surprise at each real next token, and the effective number of choices.
$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Big(\frac{QK^\top}{\sqrt{d_k}}\Big)V$$
What every one of the model's 4 layers × 4 heads computes; stage 4 shows the weights.
$$\text{FLOPs per token}\approx 2N,\qquad \text{tokens/s}\approx\frac{\text{FLOP/s}}{2N}$$
The deployment arithmetic of stage 8, with N parameters.
Common mistakes
- Training on text that is also in the test set: here the held-out 5% is cut before training and never seen.
- Judging a small model by whether it states facts: it learns style and structure; facts come from retrieval.
- Estimating serving cost from parameters alone: memory bandwidth and batch size often decide real speed.
Real-world numbers
- This model: 1.4 million parameters, a 1,024-token vocabulary and a 96-token context, trained on this site's articles and guides.
- GPT-2 small (2019) had 124 million parameters and a 50,257-token vocabulary.
- Chinchilla (2022) found about 20 training tokens per parameter to be compute-optimal.
Further reading
- Language Models are Unsupervised Multitask Learners (GPT-2)
- Neural Machine Translation of Rare Words with Subword Units (BPE)
- Training Compute-Optimal Large Language Models (Chinchilla)
- Interpreting GPT: the logit lens
Code lab
The model on this page was trained with the code below, taken from the training script in this site's repository. Run the steps in order in one Python session.
Step 1 · Clean the text
import re, hashlib
def paragraphs(markdown):
t = re.sub(r"```.*?```", "\n", markdown, flags=re.S) # code out
t = re.sub(r"\[([^\]]+)\]\([^)]*\)", r"\1", t) # keep link text
for p in re.split(r"\n\s*\n", t):
p = re.sub(r"\s+", " ", p).strip()
if len(p) >= 60 and len(p.split()) >= 10: # drop fragments
yield p
seen, kept = set(), []
for p in paragraphs(open("article.md").read()):
h = hashlib.md5(p.lower().encode()).hexdigest()
if h not in seen: # drop exact duplicates
seen.add(h); kept.append(p)
print(len(kept), "paragraphs")
Step 2 · Train a byte-pair tokenizer
pre = re.compile(r" ?[A-Za-z]+| ?[0-9]+| ?[^\sA-Za-z0-9]+|\s+")
words = {}
for w in pre.findall("\n".join(kept)):
words[w] = words.get(w, 0) + 1
seqs = {w: [bytes([b]) for b in w.encode()] for w in words}
merges, VOCAB = [], 1024
while 256 + len(merges) < VOCAB:
pairs = {}
for w, c in words.items():
s = seqs[w]
for i in range(len(s) - 1):
pairs[(s[i], s[i + 1])] = pairs.get((s[i], s[i + 1]), 0) + c
if not pairs: break
a, b = max(pairs, key=pairs.get); merges.append((a, b))
for w in words: # apply the new merge everywhere
s, out, i = seqs[w], [], 0
while i < len(s):
if i + 1 < len(s) and s[i] == a and s[i + 1] == b: out.append(a + b); i += 2
else: out.append(s[i]); i += 1
seqs[w] = out
print(merges[:10])
Step 3 · A small GPT
import torch, torch.nn as nn, torch.nn.functional as F
class Block(nn.Module):
def __init__(s, d, h):
super().__init__(); s.ln1, s.ln2, s.h = nn.LayerNorm(d), nn.LayerNorm(d), h
s.qkv, s.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
s.fc, s.fc2 = nn.Linear(d, 4 * d), nn.Linear(4 * d, d)
def forward(s, x):
B, T, D = x.shape
q, k, v = (z.view(B, T, s.h, D // s.h).transpose(1, 2) for z in s.qkv(s.ln1(x)).split(D, 2))
y = F.scaled_dot_product_attention(q, k, v, is_causal=True).transpose(1, 2).reshape(B, T, D)
x = x + s.proj(y)
return x + s.fc2(F.gelu(s.fc(s.ln2(x)), approximate="tanh"))
class GPT(nn.Module):
def __init__(s, vocab=1024, ctx=96, d=160, layers=4, heads=4):
super().__init__(); s.tok, s.pos = nn.Embedding(vocab, d), nn.Embedding(ctx, d)
s.blocks = nn.ModuleList(Block(d, heads) for _ in range(layers)); s.lnf = nn.LayerNorm(d)
def forward(s, idx):
x = s.tok(idx) + s.pos(torch.arange(idx.shape[1]))
for b in s.blocks: x = b(x)
return s.lnf(x) @ s.tok.weight.T # tied input and output embeddings
model = GPT(); print(sum(p.numel() for p in model.parameters()), "parameters")
Step 4 · Train with a held-out split
import math
ids = torch.tensor(token_ids) # the corpus, tokenized with the merges from step 2
n_val = len(ids) // 20; train, val = ids[:-n_val], ids[-n_val:]
opt = torch.optim.AdamW(model.parameters(), lr=3e-3, betas=(0.9, 0.95), weight_decay=0.05)
def batch(src, bs=32, ctx=96):
ix = torch.randint(len(src) - ctx - 1, (bs,))
return torch.stack([src[i:i + ctx] for i in ix]), torch.stack([src[i + 1:i + ctx + 1] for i in ix])
STEPS = 5000
for step in range(1, STEPS + 1):
lr = 3e-3 * min(1, step / 200) * (0.1 + 0.45 * (1 + math.cos(math.pi * step / STEPS)))
for g in opt.param_groups: g["lr"] = lr
x, y = batch(train); loss = F.cross_entropy(model(x).view(-1, 1024), y.view(-1))
opt.zero_grad(); loss.backward(); torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step()
if step % 500 == 0:
with torch.no_grad():
xv, yv = batch(val); vl = F.cross_entropy(model(xv).view(-1, 1024), yv.view(-1)).item()
print(step, round(loss.item(), 3), "held-out", round(vl, 3), "perplexity", round(math.exp(vl), 1))
Step 5 · Generate, and estimate serving cost
@torch.no_grad()
def generate(ids, n=30, temperature=0.8, k=12):
for _ in range(n):
logits = model(ids[:, -96:])[:, -1] / temperature
v, i = logits.topk(k); p = torch.softmax(v, -1)
ids = torch.cat([ids, i.gather(1, torch.multinomial(p, 1))], 1)
return ids
N = sum(p.numel() for p in model.parameters())
for params, gpu_tflops, usd_h in [(N, 50, 1.0), (8e9, 200, 2.5), (70e9, 400, 6.0)]:
tps = gpu_tflops * 1e12 / (2 * params)
print(f"{params:,.0f} params: {tps:,.0f} tokens/s compute-bound, ${usd_h / 3600 / tps * 1e6:.4f} per million tokens")
The compute-bound estimate is an upper bound: real servers are often limited by memory bandwidth, which is why batching and the KV cache matter so much.
More simulations
All AI Simulations · Byte-pair encoding → · Attention → · Training run →