# Tokenisation with byte-pair encoding, merge by merge

Language models do not read letters or words; they read tokens. Train a byte-pair encoding tokeniser on your own text and watch frequent pieces such as 'est' and 'er' become single tokens.

<div data-sim="bpe"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

The text starts as characters, with ␣ marking the end of each word. Each merge finds the most frequent adjacent pair in the whole corpus and joins it everywhere into one new token. The list on the right is the learned vocabulary, in order.

## Things to try

- Run 20 merges and read the list: suffixes such as er, est and ing appear early.
- Type a text in another language and see which pieces it finds.
- Count the tokens before and after: fewer tokens means a shorter, cheaper input for the model.

## How it works in real systems

GPT, Llama and most other language models use BPE or a close relative such as Unigram, trained on terabytes of text with vocabularies of 32,000 to 200,000 tokens, usually on bytes rather than letters so any text can be encoded.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Byte-pair encoding from scratch

Start from characters, count adjacent pairs, merge the most frequent pair, repeat. The list of merges is the tokenizer.

```python
from collections import Counter
def train_bpe(text, n_merges):
    words = Counter(text.lower().split())
    vocab = {tuple(w) + ("</w>",): c for w, c in words.items()}
    merges = []
    for _ in range(n_merges):
        pairs = Counter()
        for sym, c in vocab.items():
            for a, b in zip(sym, sym[1:]): pairs[a, b] += c
        if not pairs: break
        (a, b), n = pairs.most_common(1)[0]
        if n < 2: break
        merges.append((a, b))
        new = {}
        for sym, c in vocab.items():
            out, i = [], 0
            while i < len(sym):
                if i < len(sym) - 1 and sym[i] == a and sym[i + 1] == b: out.append(a + b); i += 2
                else: out.append(sym[i]); i += 1
            new[tuple(out)] = c
        vocab = new
    return merges, vocab
merges, vocab = train_bpe("the lower the newer the lowest newest wider widest", 10)
print(merges)
```

### Step 2 · Encode new text with the learned merges

Apply the merges in the order they were learned.

```python
def encode(word, merges):
    sym = list(word) + ["</w>"]
    for a, b in merges:
        i = 0
        while i < len(sym) - 1:
            if sym[i] == a and sym[i + 1] == b: sym[i:i + 2] = [a + b]
            else: i += 1
    return sym
print(encode("slowest", merges))
```

### Step 3 · Production tokenizers

Real tokenizers work on bytes (so any text, emoji or language can be encoded) and are written in Rust.

```python
# pip install tokenizers tiktoken
from tokenizers import Tokenizer, models, trainers, pre_tokenizers
tok = Tokenizer(models.BPE(unk_token="[UNK]"))
tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
tok.train_from_iterator(["your corpus here", "one string per document"], trainers.BpeTrainer(vocab_size=1000, special_tokens=["[UNK]"]))
print(tok.encode("tokenisation").tokens)

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
ids = enc.encode("Tokenisation splits text into pieces.")
print(len(ids), [enc.decode([i]) for i in ids])
```

<div class="cl-end"></div>

## More simulations

[← Convolution](/ai-simulations/convolution) · [All AI Simulations](/ai-simulations) · [Attention →](/ai-simulations/attention)
