# Train a tokenizer: byte-pair merges and the vocabulary-size trade-off

The tokenizer is trained once, before the model, and fixes how every text is cut into tokens. This simulation learns byte-pair encoding merges from a paragraph and shows what each extra merge buys.

<div data-sim="tfsvocab"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

The chips show your text split into tokens with the merges learned so far; the chart shows characters per token as merges accumulate. The side panel scales the choice to a real model: every vocabulary entry is a row in the embedding table and, unless the weights are tied, another row in the output layer.

## Things to try

- Drag the merges to zero: every character is its own token and the sequence is as long as the text.
- Move to 160 merges: common words such as 'the' and 'tokens' become single tokens.
- Type a word the paragraph never used: it falls back to smaller pieces.
- Switch the scale from 32k to 256k at width 8,192: the embedding table alone passes two billion parameters.

## How it works in real systems

Production tokenizers run byte-level BPE or Unigram over a sample of the training corpus with SentencePiece or Hugging Face tokenizers, add special tokens for chat roles and tools, and pick a vocabulary between about 32k and 256k entries. Multilingual and code-heavy models need larger vocabularies; the vocabulary cannot change after pre-training without retraining the embeddings.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Train a byte-level BPE tokenizer

```python
# pip install tokenizers datasets
from tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers
from datasets import load_dataset

ds = load_dataset("wikitext", "wikitext-103-raw-v1", split="train")
tok = Tokenizer(models.BPE())
tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
tok.decoder = decoders.ByteLevel()
trainer = trainers.BpeTrainer(vocab_size=32000, special_tokens=["<|bos|>", "<|eos|>", "<|pad|>"],
                              initial_alphabet=pre_tokenizers.ByteLevel.alphabet())
tok.train_from_iterator((r["text"] for r in ds), trainer=trainer)
tok.save("tokenizer-32k.json")
print(tok.encode("Tokenizers split rare words into pieces.").tokens)
```

### Step 2 · Compare vocabulary sizes

```python
sample = [r["text"] for r in ds.select(range(2000)) if r["text"].strip()]
chars = sum(len(t) for t in sample)
for v in (8000, 32000, 64000):
    t = Tokenizer(models.BPE()); t.pre_tokenizer = pre_tokenizers.ByteLevel()
    t.train_from_iterator((r["text"] for r in ds), trainers.BpeTrainer(vocab_size=v, initial_alphabet=pre_tokenizers.ByteLevel.alphabet()))
    n = sum(len(t.encode(x).ids) for x in sample)
    print(f"vocab {v:6d}: {chars / n:.2f} chars per token · embedding params at d=4096: {v * 4096 / 1e6:.0f} M")
```

<div class="cl-end"></div>

## More simulations

[← Data pipeline](/ai-simulations/train-data-pipeline) · [All AI Simulations](/ai-simulations) · [Model sizing →](/ai-simulations/train-model-sizing)
