Tokenisation with byte-pair encoding, merge by merge
Language models do not read letters or words; they read tokens. Train a byte-pair encoding tokeniser on your own text and watch frequent pieces such as 'est' and 'er' become single tokens.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The text starts as characters, with ␣ marking the end of each word. Each merge finds the most frequent adjacent pair in the whole corpus and joins it everywhere into one new token. The list on the right is the learned vocabulary, in order.
Things to try
- Run 20 merges and read the list: suffixes such as er, est and ing appear early.
- Type a text in another language and see which pieces it finds.
- Count the tokens before and after: fewer tokens means a shorter, cheaper input for the model.
How it works in real systems
GPT, Llama and most other language models use BPE or a close relative such as Unigram, trained on terabytes of text with vocabularies of 32,000 to 200,000 tokens, usually on bytes rather than letters so any text can be encoded.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Byte-pair encoding from scratch
Start from characters, count adjacent pairs, merge the most frequent pair, repeat. The list of merges is the tokenizer.
from collections import Counter
def train_bpe(text, n_merges):
words = Counter(text.lower().split())
vocab = {tuple(w) + ("</w>",): c for w, c in words.items()}
merges = []
for _ in range(n_merges):
pairs = Counter()
for sym, c in vocab.items():
for a, b in zip(sym, sym[1:]): pairs[a, b] += c
if not pairs: break
(a, b), n = pairs.most_common(1)[0]
if n < 2: break
merges.append((a, b))
new = {}
for sym, c in vocab.items():
out, i = [], 0
while i < len(sym):
if i < len(sym) - 1 and sym[i] == a and sym[i + 1] == b: out.append(a + b); i += 2
else: out.append(sym[i]); i += 1
new[tuple(out)] = c
vocab = new
return merges, vocab
merges, vocab = train_bpe("the lower the newer the lowest newest wider widest", 10)
print(merges)
Step 2 · Encode new text with the learned merges
Apply the merges in the order they were learned.
def encode(word, merges):
sym = list(word) + ["</w>"]
for a, b in merges:
i = 0
while i < len(sym) - 1:
if sym[i] == a and sym[i + 1] == b: sym[i:i + 2] = [a + b]
else: i += 1
return sym
print(encode("slowest", merges))
Step 3 · Production tokenizers
Real tokenizers work on bytes (so any text, emoji or language can be encoded) and are written in Rust.
# pip install tokenizers tiktoken
from tokenizers import Tokenizer, models, trainers, pre_tokenizers
tok = Tokenizer(models.BPE(unk_token="[UNK]"))
tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
tok.train_from_iterator(["your corpus here", "one string per document"], trainers.BpeTrainer(vocab_size=1000, special_tokens=["[UNK]"]))
print(tok.encode("tokenisation").tokens)
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
ids = enc.encode("Tokenisation splits text into pieces.")
print(len(ids), [enc.decode([i]) for i in ids])