BPE

Overview

Byte-pair encoding (BPE) is a method for splitting text into pieces called tokens. It starts with single characters or bytes, then repeatedly merges the most frequent adjacent pair into a new token. After many merges, common words become single tokens, rarer words split into a few familiar pieces, and unknown words can still be spelled out from smaller parts. This makes BPE the most common subword tokenizer for language models.

The idea is simple and powerful. A model needs a fixed vocabulary, but language has endless words. BPE gives a vocabulary of a chosen size that covers frequent text efficiently and handles anything else by falling back to smaller units. Tokenization also matters for tables. When rows, column names and numbers are turned into text for a language model, the tokenizer decides how many tokens each cell costs and how well numbers are represented. This article explains the algorithm step by step, shows a worked example, compares BPE with other tokenizers such as SentencePiece, and describes what it means for tabular foundation models.

What it is

A tokenizer converts a string into a list of integer ids. A model then works on those ids. BPE is one way to decide which pieces get ids.

Think of building words from LEGO bricks. You could use only single-stud bricks (characters), but building takes forever. You could have one custom brick for every possible word, but that needs millions of molds. BPE finds a middle path: it learns a set of medium-sized bricks that fit the text you see most.

Three parts

1. A vocabulary. The list of tokens, such as the, ing, tion, a, b.
2. A merge list. An ordered list of rules like "join e and s into es".
3. An encoding procedure. Apply the merge rules in order to new text.

Training BPE means learning the merge list from a sample of text. Using BPE means applying the learned merges.

Character-level versus byte-level

• Character-level BPE starts from Unicode characters. Rare characters may be unknown if they never appeared in training text.
• Byte-level BPE starts from the 256 possible byte values. Any text in any language, any symbol and any emoji can be encoded, because every string is a sequence of bytes. Nothing is ever unknown.

Byte-level BPE became popular for large language models because it removes the unknown-token problem.

Tokens are not words

A token may be a whole word, part of a word, a space plus a word, a single punctuation mark or a byte. Many tokenizers attach the leading space to the following word, so hello and hello can be different tokens. This detail often surprises newcomers.

Where it came from

BPE began as a data compression technique. In 1994, Philip Gage described a simple compression method that replaced the most common pair of adjacent bytes with a byte not used in the data, then repeated. This reduced file size by shrinking repeated patterns.

Machine translation later needed a way to handle rare and new words. Fixed word vocabularies struggled with names, compounds and inflections. In 2015, Sennrich, Haddow and Birch proposed using BPE to build subword vocabularies for neural machine translation. Their paper, often cited as "Neural Machine Translation of Rare Words with Subword Units", showed that subword units let a model translate rare words by composing known parts.

Soon BPE was adopted widely. GPT-2 used a byte-level variant, which became a standard design for later generative models. Other subword methods also grew up in the same period. WordPiece, used in early large encoder models, chooses merges by a likelihood score rather than raw frequency. The Unigram language model tokenizer, proposed by Kudo in 2018, starts from a large vocabulary and removes pieces. The library SentencePiece packages BPE and Unigram so they can run directly on raw text without pre-splitting into words.

Today most large language models use some form of subword tokenization, and BPE or a close variant is among the most common. The details of vocabulary size and training data differ between models and are often not fully published, so avoid assuming specifics unless a model's documentation states them.

How it works

Training: learning the merges

Step 1: Prepare the training text

Collect a representative text sample. Normalize it if needed. Split it into rough chunks, often words, using a pre-tokenization rule such as splitting on spaces and punctuation. Count how often each chunk appears.

Step 2: Start from the smallest units

Break each chunk into its basic symbols: characters for character-level BPE, bytes for byte-level BPE. Many classic implementations add an end-of-word marker to remember where words stop.

Step 3: Count adjacent pairs

Look at every neighboring pair of symbols in every chunk. Weight each by the chunk frequency. Count them.

Step 4: Merge the most frequent pair

Pick the pair with the highest count. Create a new symbol by joining them. Replace every occurrence of the pair with the new symbol. Add the new symbol to the vocabulary and record the merge rule.

Step 5: Repeat

Recount pairs and merge again. Stop when the vocabulary reaches the target size or when no pair is frequent enough.

Each merge adds exactly one token to the vocabulary. If you start with 256 bytes and run 31,744 merges, you get a vocabulary of 32,000. The target size is a design choice.

Analogy: nicknames

Imagine a group of friends who often say "the New York office". At first they say every word. Soon they say "NY office". Then simply "NYO". Frequent phrases get shorter names. BPE does the same for frequent symbol pairs.

Encoding: applying the merges

To tokenize new text:

1. Pre-tokenize into chunks.
2. Break each chunk into basic symbols.
3. Apply merge rules in the order they were learned. At each step, merge the pair that has the earliest rank in the list.
4. When no learned pair remains, the symbols left are the tokens.
5. Look up each token's integer id.

Because merges are applied in the learned order, the same text always gives the same tokens. This is deterministic.

Decoding

Decoding reverses the lookup: ids to tokens, tokens to bytes or characters, then join. For byte-level BPE, the bytes are decoded as UTF-8 text. With a correct tokenizer, decoding the encoded text returns exactly the original.

Special tokens

Tokenizers include special tokens that are not learned from text: start of sequence, end of sequence, padding, separators and, in chat models, role markers. They are added by hand and reserved ids.

Pre-tokenization matters

Before BPE runs, a rule decides the chunk boundaries. Common rules split on whitespace, separate punctuation and digits, or use a pattern that keeps contractions together. The pre-tokenization rule prevents merges from crossing certain boundaries. For example, preventing a merge between a letter and a digit keeps numbers cleaner.

Vocabulary size trade-offs

Typical vocabularies in modern models range from tens of thousands to a few hundred thousand tokens. Multilingual models often need larger vocabularies, because they must cover several scripts.

Tokens per word

The average number of tokens per word is called fertility. English text with a well-matched tokenizer often needs somewhat more than one token per word. Languages and domains that were rare in tokenizer training can need many more tokens per word, which means higher cost and a shorter effective context.

Key concepts and terms

A worked example

This is a small hand example, then code that does the same thing.

The training data

Suppose a tiny corpus has four words with counts:

Each word is split into characters and ends with a marker </w> that means end of word.

l o w </w>         x5
l o w e r </w>     x2
n e w e s t </w>   x6
w i d e s t </w>   x3

Merge 1

Count adjacent pairs. The pair e s appears 6 times in "newest" and 3 times in "widest", so 9. The pairs s t and t </w> also total 9. Ties are broken by first occurrence in this example. Merge e s into es.

l o w </w>         x5
l o w e r </w>     x2
n e w es t </w>    x6
w i d es t </w>    x3

Merge 2

Now es t appears 9 times. Merge it into est.

n e w est </w>     x6
w i d est </w>     x3

Merge 3

est </w> appears 9 times. Merge into est</w>. This token now means "the word ending -est".

Merge 4 and 5

Next, l o and o w both appear 7 times (5 in "low", 2 in "lower"). Merge l o into lo, then lo w into low.

Result after five merges

Final state of the corpus:

low </w>           x5
low e r </w>       x2
n e w est</w>      x6
w i d est</w>      x3

Now the vocabulary is the original characters plus es, est, est</w>, lo and low.

Encoding a new word

Take the unseen word "lowest". Split to characters: l o w e s t </w>. Apply merges in rank order:

1. Merge 1 joins e s: l o w es t </w>.
2. Merge 2 joins es t: l o w est </w>.
3. Merge 3 joins est </w>: l o w est</w>.
4. Merge 4 joins l o: lo w est</w>.
5. Merge 5 joins lo w: low est</w>.

The word "lowest" never appeared in training, yet it encodes as two tokens, low and est</w>. The model can understand it from its parts. This is the main benefit of BPE.

The same algorithm in Python

from collections import Counter

def get_stats(vocab):
    pairs = Counter()
    for word, freq in vocab.items():
        symbols = word.split()
        for a, b in zip(symbols, symbols[1:]):
            pairs[(a, b)] += freq
    return pairs

def merge(pair, vocab):
    out = {}
    a, b = pair
    for word, freq in vocab.items():
        s = word.split()
        i, new = 0, []
        while i < len(s):
            if i < len(s) - 1 and s[i] == a and s[i + 1] == b:
                new.append(a + b)
                i += 2
            else:
                new.append(s[i])
                i += 1
        out[" ".join(new)] = freq
    return out

vocab = {
    "l o w </w>": 5,
    "l o w e r </w>": 2,
    "n e w e s t </w>": 6,
    "w i d e s t </w>": 3,
}

merges = []
for _ in range(5):
    pairs = get_stats(vocab)
    best = max(pairs, key=pairs.get)
    merges.append(best)
    vocab = merge(best, vocab)

print(merges)

Output:

[('e', 's'), ('es', 't'), ('est', '</w>'), ('l', 'o'), ('lo', 'w')]

Encoding with the learned merges

def encode_word(word, merges):
    symbols = list(word) + ["</w>"]
    for a, b in merges:
        i = 0
        while i < len(symbols) - 1:
            if symbols[i] == a and symbols[i + 1] == b:
                symbols[i:i + 2] = [a + b]
            else:
                i += 1
    return symbols

print(encode_word("lowest", merges))   # ['low', 'est</w>']
print(encode_word("newer", merges))    # ['n', 'e', 'w', 'e', 'r', '</w>']

The word "newer" stays in single characters because the corpus never gave enough evidence to merge them. In a real training run on a large corpus, many more merges would apply.

Byte-level view

In byte-level BPE, the starting symbols are bytes. The word "café" contains the character "é", which is two bytes in UTF-8. Even if no merge ever covered it, the tokenizer can still encode it as byte tokens. That is why byte-level BPE never produces an unknown token.

Strengths and limitations

Strengths

• No unknown words (byte-level). Any string can be encoded.
• Compact vocabularies. A vocabulary of fixed size covers frequent text well.
• Composes new words. Rare and invented words split into known parts.
• Simple and fast. Training and encoding are straightforward and efficient.
• Deterministic. The same input gives the same output.
• Language-agnostic. It works on any script with no hand-written rules, especially at byte level.
• Widely supported. Many libraries and models use it, so tools and tutorials are abundant.

Limitations

• Frequency-driven, not meaning-driven. Merges follow statistics, not grammar. Tokens may not match morphemes.
• Uneven across languages. Languages that were rare in tokenizer training get many more tokens per word.
• Numbers are awkward. A long number may split in arbitrary ways, for example 12345 into 123 and 45, while 12346 splits differently. This can make arithmetic and numeric comparison harder.
• Whitespace and case quirks. A word with and without a leading space can have different tokens. Capitalization can change tokens.
• Sensitive to the training sample. A tokenizer trained on one domain may split another domain badly, such as code, chemistry or tabular records.
• Fixed after training. Changing the tokenizer after a model is trained is hard, because embeddings are tied to token ids.
• Glitch tokens. Rare tokens that appear in tokenizer training data but hardly in model training can behave oddly.
• No built-in sampling variety. Standard BPE gives one segmentation. Some variants add randomness (BPE-dropout) for regularization.
• Longer sequences. Compared with whole-word vocabularies, subwords produce more tokens per text, which uses more context.

How it compares with alternatives

For how to train a tokenizer on your own data, including settings and pitfalls, see tokenizer training. For the architecture that consumes tokens, see Transformers.

BPE versus SentencePiece

They are not direct rivals. BPE is an algorithm. SentencePiece is a library that can run BPE or Unigram on raw text, treats whitespace as a normal symbol and does not require a separate pre-tokenizer. Many models that say they use "SentencePiece" are using BPE inside it.

BPE versus data tokenization

The word tokenization is also used in data security, where a sensitive value is replaced by a random surrogate. That is a different idea. See data tokenization for that meaning. BPE tokenization splits text for a model and has no privacy purpose.

Role in tabular foundation models

BPE belongs to the world of text models, so its role in tabular foundation models is indirect but real. Tabular foundation models learn from raw tables: rows and columns, mixed types, missing values and often no column names. Some of them never use a text tokenizer at all. Others rely on one. Understanding where BPE appears helps you choose between designs.

Two broad routes for tables

1. Serialize the table to text and use a language model. Each row becomes a sentence or a line such as age is 42, income is 51000, city is Lyon. A BPE tokenizer splits this into tokens, and a language model reads them. This route reuses existing large models and tolerates column names, free text and rare categories.
2. Use a model built for tables. Cells are embedded directly as numbers or categories, with attention across rows and columns. Many tabular foundation models are designed this way and do not tokenize with BPE. See TabPFN and FT-Transformer for examples of table-native approaches, and /model/architecture for how such designs treat structure.

Where BPE helps in the text route

• Column names and headers. Names such as cust_eml split into meaningful pieces, cust, _, eml, which a language model may relate to known words.
• Free-text cells. Notes, descriptions and product titles are text, and BPE handles them well.
• High-cardinality categories. Rare category strings are spelled from subwords rather than being unknown.
• Mixed data. Text and categorical values go through one mechanism.

Where BPE hurts

• Numbers. BPE may split a number inconsistently. The values 1000, 1001 and 1010 might break into very different token sequences, even though they are numerically close. A model must learn numeric relations from token patterns, which is difficult. Table-native models avoid this by treating numbers as numbers.
• Token cost. A table with many rows and columns can produce a very long token sequence. The context window limits how many example rows can be placed in a prompt for in-context learning. Verbose serialization such as repeating column names for every row wastes tokens.
• Missing values. A missing cell must be written as some text, like NaN, null or empty. Different choices produce different tokens, and the model may treat them inconsistently.
• Order sensitivity. Text models are sensitive to the order of columns and rows in the serialized string, whereas tables are naturally unordered sets of rows and columns.
• No column names. If column names are absent or meaningless, text serialization loses its main advantage. A tokenizer cannot add meaning that is not there.

Pre-training on synthetic or real tables

Table-native pre-training on synthetic tables usually does not involve a text tokenizer, since the generated cells are numbers and category codes. Pre-training on real tables as text does involve one, and the tokenizer's training data should include table-like strings such as delimiters, numbers and identifiers. Otherwise common table patterns will be split inefficiently. See /model/research for the research landscape.

In-context learning

In the text route, labeled example rows and query rows are packed into the prompt. Tokenization decides how many examples fit. Compact, consistent formats stretch the context. In table-native models, in-context learning happens over cell embeddings, and the limiting factor is the number of rows and columns the architecture can attend over, not tokens.

Imputation, anomaly detection and sector classification

A language model reading serialized rows can be asked to fill a blank or to flag a strange value, but numeric precision depends on how numbers were tokenized. For dedicated imputation, see imputation networks. For anomaly detection from values alone, see anomaly detection. For classifying a table or column into a semantic category from content, see semantic sector heads. Where text is available, BPE helps map strings to a shared space. Where only numbers are available, table-native encoders usually do better.

Hybrid designs

Some systems combine both: a table-native encoder for numeric and categorical cells and a text encoder for column names and free text. The text encoder uses a subword tokenizer such as BPE. This gives numeric fidelity plus semantic hints from names.

Evaluation

When comparing approaches, token counts matter as a cost measure, and numeric handling matters as an accuracy measure. See /model/benchmarks for how comparisons are framed.

Practical guidance and best practices

Match the tokenizer to your domain

If you train your own tokenizer, include samples from the target domain. For tables, include numbers, identifiers, dates, delimiters and typical column names. See tokenizer training.

Choose vocabulary size deliberately

Pick a size that balances sequence length and embedding cost. Test a few sizes on held-out text and compare tokens per character and downstream results. Do not choose by habit alone.

Use byte-level fallback

If you need full coverage, use a byte-level base or a byte-fallback option so that unseen characters are encoded as bytes instead of an unknown token.

Handle numbers explicitly

For tasks that depend on numbers, consider options such as splitting digits into single tokens, scaling numbers into a fixed format, or using a table-native model that embeds numbers directly. Consistent formatting helps: fixed decimals, no thousands separators and a clear sign.

Keep serialization compact and consistent

• Use a fixed column order.
• Use short, stable separators.
• Decide one representation for missing values and keep it.
• Avoid repeating long column names for every row if you can put them once in a header.

Test round trips

Check that decoding an encoded text returns the original. Check edge cases: empty strings, strange Unicode, very long strings, leading and trailing spaces.

Count tokens before you run

Estimate cost and context needs by tokenizing a sample. A table that looks short may be very long in tokens.

Keep the tokenizer with the model

A model and its tokenizer are a pair. Use the exact tokenizer files that came with the model, including special tokens and normalization settings. Mixing tokenizers silently breaks results.

Be careful with normalization

Unicode normalization, lowercasing and whitespace rules change tokens. Apply the same normalization at training and inference time.

Watch for leakage in special strings

Prompts that contain special tokens or role markers typed by users can confuse a model. Sanitize inputs when needed.

Use randomized segmentation only for training

Variants such as BPE-dropout add random splits as a regularizer. They help robustness but should be turned off at inference for stable output.

Monitor tokenization drift

If your data changes, such as new product codes or a new language, check that tokens per record has not jumped. A sudden rise signals mismatch.

Common mistakes

• Confusing BPE with a model. It is a preprocessing step with no knowledge of meaning.
• Changing the tokenizer after training a model. Token ids no longer match learned embeddings.
• Assuming one token equals one word. It often does not, and costs depend on tokens.
• Ignoring leading spaces. word and word may differ.
• Training the tokenizer on the wrong data. A tokenizer trained on prose splits tables badly.
• Treating numbers as ordinary text. Digits split inconsistently and hurt numeric tasks.
• Using different normalization at inference. Results shift quietly.
• Forgetting special tokens. Missing separators or padding settings cause errors.
• Choosing a huge vocabulary without need. Rare tokens are poorly learned and the embedding table grows.
• Not testing multilingual coverage. A tokenizer may be fine in one language and costly in another.
• Mixing up security tokenization and text tokenization. The two share a word but serve different purposes.
• Counting characters when budgeting context. Count tokens.
• Serializing tables in different formats for training and prompting. The model sees a shift.

Frequently asked questions

What does BPE stand for?

BPE stands for byte-pair encoding. It started as a compression method that replaced frequent pairs of bytes with a new symbol, and was later adapted to build subword vocabularies for language models.

How does BPE decide which tokens to create?

It counts how often each adjacent pair of symbols occurs in the training text and merges the most frequent pair into a new token. It repeats this many times until the vocabulary reaches the desired size. Frequent character sequences therefore become single tokens.

What is the difference between BPE and byte-level BPE?

Standard BPE starts from characters and may meet characters it has never seen. Byte-level BPE starts from the 256 byte values, so any text in any language can be represented. This removes the unknown-token problem at the cost of splitting rare characters into several byte tokens.

How is BPE different from WordPiece and Unigram?

BPE merges the most frequent pair at each step. WordPiece chooses merges by a likelihood-based score, and Unigram starts with a large vocabulary and removes pieces to maximize likelihood, which also allows several possible segmentations. In practice they give similar subword tokens, and libraries such as SentencePiece support more than one method.

Why do language models have trouble with numbers?

Because a tokenizer may split numbers into inconsistent chunks, nearby numbers can look unrelated at the token level. The model must infer numeric structure from these pieces, which is hard. Splitting digits individually or using a format that is consistent helps, and table-native models avoid the issue by embedding numbers directly.

Does a tabular foundation model need a BPE tokenizer?

Not necessarily. Models that embed cells directly as numbers and categories do not need one, while approaches that turn rows into text for a language model do. Hybrid systems may use a tokenizer only for column names and free text.

How big should my vocabulary be?

There is no single answer. Smaller vocabularies give longer sequences but cover rare words by composition, and larger ones give shorter sequences but include rarely seen tokens. Many models use tens of thousands to a few hundred thousand tokens, and the best size depends on languages, domain and model size.

Can I add new tokens to an existing tokenizer?

You can, but the model has no learned embedding for them, so you must train or fine-tune to give them meaning. Adding tokens without training them usually does not help. Replacing the whole tokenizer is far more disruptive.

Is BPE deterministic?

Yes, standard BPE encoding is deterministic, because merges are applied in a fixed learned order. Some training variants use randomness for regularization, but at inference the same text gives the same tokens.

How do I count tokens for cost or context planning?

Run the model's own tokenizer on a representative sample and measure tokens per record, then multiply by the number of records. Character or word counts are only rough guides, and the difference can be large for tables, code and non-English text.

Glossary

• BPE: byte-pair encoding, a merge-based subword algorithm.
• Byte fallback: representing unknown characters as raw bytes.
• Byte-level: operating on bytes rather than characters.
• Context window: the maximum number of tokens a model can process at once.
• Decoding: converting token ids back to text.
• Embedding: a learned vector for a token.
• Encoding: converting text to token ids.
• Fertility: average tokens per word.
• Merge rule: a learned instruction to join two symbols.
• Normalization: cleaning text before tokenizing.
• Pre-tokenization: splitting text into chunks before merging.
• Serialization: writing a row or table as text.
• Special token: a reserved control token.
• Subword: a unit smaller than a word.
• Token: a unit of text with an integer id.
• Tokenizer: the component that maps text to tokens.
• Unigram: a probabilistic subword method that prunes a large vocabulary.
• Unknown token: a placeholder for text the tokenizer cannot represent.
• Vocabulary: the set of tokens known to the tokenizer.
• WordPiece: a subword method that selects merges by likelihood.

Summary

BPE builds a subword vocabulary by repeatedly merging the most frequent adjacent pair of symbols. Applying the learned merges in order gives a deterministic, compact way to encode any text, and the byte-level version never meets an unknown character. It is simple, fast and the foundation of many language model tokenizers, with known weaknesses around numbers, uneven treatment of languages and sensitivity to its training data.

For tabular foundation models, BPE matters when tables are turned into text. It helps with column names, free text and rare categories, but it handles numbers and missing values poorly and spends context on verbose serialization. Table-native models avoid these issues by embedding cells directly, and hybrids use a tokenizer only where text meaning helps. Choose the route based on your data, and keep tokenizer and model together. Related reading: SentencePiece, tokenizer training, Transformers, TabPFN, and the model pages /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Foundation models from scratch in my stack, under Foundation models and applied ML. Others in the same group: