Train a tokenizer: byte-pair merges and the vocabulary-size trade-off
The tokenizer is trained once, before the model, and fixes how every text is cut into tokens. This simulation learns byte-pair encoding merges from a paragraph and shows what each extra merge buys.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The chips show your text split into tokens with the merges learned so far; the chart shows characters per token as merges accumulate. The side panel scales the choice to a real model: every vocabulary entry is a row in the embedding table and, unless the weights are tied, another row in the output layer.
Things to try
- Drag the merges to zero: every character is its own token and the sequence is as long as the text.
- Move to 160 merges: common words such as 'the' and 'tokens' become single tokens.
- Type a word the paragraph never used: it falls back to smaller pieces.
- Switch the scale from 32k to 256k at width 8,192: the embedding table alone passes two billion parameters.
How it works in real systems
Production tokenizers run byte-level BPE or Unigram over a sample of the training corpus with SentencePiece or Hugging Face tokenizers, add special tokens for chat roles and tools, and pick a vocabulary between about 32k and 256k entries. Multilingual and code-heavy models need larger vocabularies; the vocabulary cannot change after pre-training without retraining the embeddings.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Train a byte-level BPE tokenizer
# pip install tokenizers datasets
from tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers
from datasets import load_dataset
ds = load_dataset("wikitext", "wikitext-103-raw-v1", split="train")
tok = Tokenizer(models.BPE())
tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
tok.decoder = decoders.ByteLevel()
trainer = trainers.BpeTrainer(vocab_size=32000, special_tokens=["<|bos|>", "<|eos|>", "<|pad|>"],
initial_alphabet=pre_tokenizers.ByteLevel.alphabet())
tok.train_from_iterator((r["text"] for r in ds), trainer=trainer)
tok.save("tokenizer-32k.json")
print(tok.encode("Tokenizers split rare words into pieces.").tokens)
Step 2 · Compare vocabulary sizes
sample = [r["text"] for r in ds.select(range(2000)) if r["text"].strip()]
chars = sum(len(t) for t in sample)
for v in (8000, 32000, 64000):
t = Tokenizer(models.BPE()); t.pre_tokenizer = pre_tokenizers.ByteLevel()
t.train_from_iterator((r["text"] for r in ds), trainers.BpeTrainer(vocab_size=v, initial_alphabet=pre_tokenizers.ByteLevel.alphabet()))
n = sum(len(t.encode(x).ids) for x in sample)
print(f"vocab {v:6d}: {chars / n:.2f} chars per token · embedding params at d=4096: {v * 4096 / 1e6:.0f} M")