Tokenizer training

Overview

Tokenizer training is the process of learning a vocabulary of small pieces, called tokens, from a body of example data, so that any future input can be written as a short sequence of those pieces. The result is a fixed table that maps raw input to integer ids and back. A language model never sees raw text. It sees these ids. So the quality of the tokenizer sets a ceiling on how efficiently and how cleanly the model can read its input.

A trained tokenizer is a compression scheme tuned to a data distribution. Common strings get a single token. Rare strings are spelled out from smaller parts. Methods such as BPE and SentencePiece fit this vocabulary by counting and merging. The same idea reaches beyond text. Numbers, dates, category labels and whole table cells can all be turned into tokens, which matters for models that learn from raw tables such as TabPFN, TabICL and TabSTAR.

What it is

A tokenizer has two parts. The first is a set of rules that cut an input string into pieces. The second is a vocabulary that gives every piece an integer id. Training is the step that decides what goes into the vocabulary and, for some algorithms, how the cutting rules behave.

Think of a shipping company that must pack goods into boxes. It could use one box size for everything, which wastes space. Or it could study what it ships and design a small set of box sizes that fit the common items snugly. Tokenizer training is that study. The goods are text, numbers or cells. The boxes are tokens.

What training produces

• A vocabulary: a list of tokens, each with an id.
• A set of merge rules or piece scores that say how to segment new input.
• Special tokens such as padding, start of sequence, end of sequence, unknown and mask.
• Normalization settings, such as how to treat case, accents and whitespace.
• Optional pre-tokenization rules, such as splitting on spaces or on digits.

What training does not do

Tokenizer training does not train a neural network. It is a separate, usually cheap, preprocessing stage that runs before model pre-training. Its output is frozen. Once a model is trained with a given tokenizer, changing the vocabulary breaks the link between ids and the learned embeddings.

Levels of granularity

Most modern systems use subword or byte-level subword units.

Where it came from

Early language processing used word-level vocabularies. Every distinct word got an id, and anything outside the list became an unknown token. This failed for rare words, names, typos and morphologically rich languages, where one root can produce thousands of forms.

Character-level models avoided the unknown word problem but made sequences very long and hard to model. The field then looked for a middle path.

Subword ideas

Byte pair encoding began as a data compression technique. It repeatedly replaces the most frequent pair of adjacent symbols with a new symbol. In the mid 2010s, researchers in machine translation adapted it to build subword vocabularies, so that rare words could be split into known parts. This work is usually cited as "Neural Machine Translation of Rare Words with Subword Units" (2016).

Other lines of work followed:

• WordPiece, used in early large pre-trained encoders, chooses merges by a likelihood criterion instead of raw frequency.
• The Unigram language model tokenizer (2018) starts with a large candidate vocabulary and prunes it, keeping the pieces that best explain the data.
• SentencePiece (2018) packaged these algorithms into a library that works directly on raw text without requiring pre-split words, which made it practical for languages that do not use spaces.
• Byte-level BPE, popularized by large generative models, applies merges over bytes so that every possible input can be encoded.

Beyond text

As models began to handle code, math, genomic strings and tables, researchers noticed that a tokenizer trained on prose handled these poorly. Numbers were cut at arbitrary places. Code indentation was wasteful. This pushed work on domain-specific tokenizers and on tokenizer-free designs that read bytes or numeric values directly. Tabular foundation models took a related path: many of them skip text tokenization of numbers and embed numeric values directly, while others serialize rows as text. See /model/research for how this fits the wider literature.

How it works

The details differ by algorithm, but the pipeline has the same stages.

Step 1: Collect a representative corpus

The tokenizer will only be efficient for data that looks like its training sample. If the sample is mostly English prose, then code, other languages and numeric strings will be split into many tiny pieces.

An analogy: a person who learns shorthand by writing only recipes will have symbols for "tablespoon" and "simmer", but will write legal contracts slowly. The shorthand reflects what was practiced.

Good practice is to sample the corpus in the same proportions as the intended model training mix, after cleaning and deduplication. Duplicate-heavy data distorts frequency counts and can create odd vocabulary entries, such as long boilerplate strings.

Step 2: Normalize

Normalization makes surface variations consistent. Common choices:

• Unicode normalization, so that visually identical characters have one form.
• Optional lowercasing. Many modern models keep case.
• Whitespace handling. Some tokenizers mark word starts with a special symbol so that spaces are preserved and decoding is lossless.
• Control character removal.

These choices are baked in. They must be applied identically at training and inference time.

Step 3: Pre-tokenize

Pre-tokenization splits text into coarse chunks before subword learning. Typical rules split on whitespace and punctuation, and often split digits apart. Merges are then learned only inside chunks, never across them. This stops the vocabulary from filling with strings that straddle words, such as the end of one word glued to the start of the next.

Step 4: Learn the vocabulary

Three families dominate.

Byte pair encoding

1. Start with a vocabulary of single characters or bytes.
2. Count every adjacent pair of symbols in the corpus.
3. Merge the most frequent pair into a new symbol and add it to the vocabulary.
4. Repeat until the vocabulary reaches the target size.

Each merge is recorded in order. At inference time, the tokenizer applies the same merges in the same order to new text.

Analogy: imagine labelling common letter pairs on a keyboard. First you add a single key for "th" because it is everywhere. Later you add a key for "the", then for "there". Keys that save the most typing come first.

WordPiece

Similar to BPE, but instead of picking the most frequent pair, it picks the pair that most increases the likelihood of the training data under a simple language model. The effect is that pairs are favoured when they occur together more often than their parts would suggest.

Unigram language model

1. Start with a very large set of candidate pieces, for example all frequent substrings.
2. Fit probabilities to the pieces with expectation maximization.
3. For each piece, estimate how much the overall likelihood would drop if it were removed.
4. Prune the pieces whose removal costs least, keeping a fraction each round.
5. Stop at the target vocabulary size.

Unigram can produce several valid segmentations of the same word, each with a probability. This allows a regularization trick called subword sampling, where a random segmentation is used during training so the model becomes robust to different splits.

Step 5: Add special tokens and reserved slots

Special tokens mark structure: beginning and end of a sequence, padding, separators, masks, and in chat or structured models, role markers. Many teams also reserve unused ids for later additions so the embedding matrix does not need to be resized.

Step 6: Freeze, test and package

After training, the vocabulary and rules are saved. Tests check round trips (encode then decode returns the original), coverage of important strings, and efficiency metrics such as tokens per word or bytes per token.

The role of information theory

Tokenization is a compression problem. A good vocabulary lowers the average number of tokens needed to encode the data, at the price of a bigger embedding table. The link to information theory is direct: frequent events deserve short codes, rare events can have long ones. BPE's greedy merging is a practical, approximate version of this principle, though it is not an optimal code.

Key concepts and terms

A worked example

A tiny BPE run by hand

Suppose the corpus has these words with counts:

• "low" appears 5 times
• "lower" appears 2 times
• "newest" appears 6 times
• "widest" appears 3 times

Write each word as characters with an end marker. The starting symbols are l, o, w, e, r, n, s, t, i, d and the end marker.

Count pairs. The pair "e s" occurs in newest and widest, so 6 + 3 = 9 times. The pair "s t" also occurs 9 times. The pair "l o" occurs 5 + 2 = 7 times.

Merge 1: join "e" and "s" into "es" (the first tie is broken by order). Merge 2: join "es" and "t" into "est", because "est" is now in 9 words. Merge 3: join "l" and "o" into "lo". Merge 4: join "lo" and "w" into "low".

After four merges, "newest" becomes "n e w est", "widest" becomes "w i d est", and "low" is a single token. The word "lowest", which never appeared, can still be written as "low est". That is the point: unseen words are built from known parts.

Training with a library

A short, realistic Python example using a widely used open-source tokenizer library:

from tokenizers import Tokenizer, models, trainers, pre_tokenizers, normalizers

tok = Tokenizer(models.BPE(unk_token="[UNK]"))
tok.normalizer = normalizers.NFKC()
tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)

trainer = trainers.BpeTrainer(
    vocab_size=32000,
    min_frequency=2,
    special_tokens=["[PAD]", "[UNK]", "[BOS]", "[EOS]", "[MASK]"],
)

tok.train(files=["corpus_a.txt", "corpus_b.txt"], trainer=trainer)
tok.save("my_tokenizer.json")

ids = tok.encode("Order total: 1,234.50 EUR").ids
print(len(ids), ids[:10])

The same job in SentencePiece:

import sentencepiece as spm

spm.SentencePieceTrainer.train(
    input="corpus.txt",
    model_prefix="sp",
    vocab_size=32000,
    model_type="unigram",      # or "bpe"
    character_coverage=0.9995,
    byte_fallback=True,
    split_digits=True,
)

sp = spm.SentencePieceProcessor(model_file="sp.model")
print(sp.encode("Order total: 1,234.50 EUR", out_type=str))

Measuring quality

After training, compute a few simple numbers on a held-out sample:

def tokens_per_char(tokenizer, texts):
    total_tokens = sum(len(tokenizer.encode(t).ids) for t in texts)
    total_chars = sum(len(t) for t in texts)
    return total_tokens / total_chars

Lower is better for the same vocabulary size. Compare across domains. If code or numbers show a much higher ratio than prose, the training sample under-represented them.

What happens to numbers

Take the string "1234567". A prose-trained BPE tokenizer might split it as "123", "456", "7". Another number, "1234568", might split as "1234", "568". Two nearly equal numbers receive unrelated token sequences, which makes arithmetic and comparison harder to learn. Splitting digits individually, or in fixed groups from the right, gives a regular structure. This is why many modern tokenizers split digits during pre-tokenization.

Strengths and limitations

Strengths

• Compact input: fewer tokens means shorter sequences, lower compute and a longer effective context.
• Open vocabulary: with byte fallback, any input can be encoded.
• Language flexibility: subwords share parts across related words.
• Cheap to train: a vocabulary can be fitted in minutes to hours on a single machine.
• Deterministic and fast at inference time.
• Easy to inspect: the vocabulary is a plain list.

Limitations

• Frozen after pre-training. Changing it later usually requires retraining or careful embedding surgery.
• Bias toward the training sample. Under-represented languages and domains pay a token tax.
• Splits that ignore meaning. Words and numbers are cut at statistical, not semantic, boundaries.
• Sensitive to whitespace and formatting. Small changes in input can change tokenization.
• Rare token problem. Entries that almost never appear in the model training data get poorly trained embeddings.
• Misleading cost. Token counts differ across languages, so the same sentence can cost very different amounts.
• Weak numeric fidelity. Digits as text lose magnitude and ordering unless handled with care.

Design trade-offs

How it compares with alternatives

The security term data tokenization is unrelated. It replaces sensitive values with surrogate tokens for privacy. Here, tokens are model input units.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table is different from a paragraph in several ways, and tokenizer training ideas have to be adapted to each.

Rows, columns and the grid

Text is a line. A table is a grid with structure along two axes. Rows are usually exchangeable, since shuffling rows should not change meaning. Columns carry different types and meanings. A direct text serialization, such as "age: 42, city: Paris, income: 51000", gives a language model something to read, but it turns the grid into a sequence and creates an arbitrary order. Models such as TabPFN and TabICL avoid this by treating each cell as a unit placed in a row and column structure, and using attention across rows and across columns, for example axial attention. In that design, the cell is the token.

Analogy: reading a spreadsheet by converting it to a paragraph is like reading a chessboard by listing the pieces in alphabetical order. You lose the geometry.

Mixed types

A cell can be a float, an integer, a category, a date, a boolean, a free text string or a missing value. A tokenizer for tables must handle all of them:

Here "tokenizer training" generalizes to "fit the cell encoder". The fitting step learns quantile boundaries, category vocabularies and scaling constants from training data. These are the tabular equivalents of merges.

Numbers as tokens versus numbers as values

A text tokenizer turns "51000" into pieces that carry no built-in sense of size. A numeric cell encoder keeps the value itself, so that 51000 is close to 51001 and far from 510. This is why architectures such as the FT-Transformer use a feature tokenizer that maps each numeric feature to an embedding by multiplying a learned vector by the value, and each categorical feature to a lookup embedding. The phrase "feature tokenizer" is the tabular analog of a vocabulary lookup. Its parameters are learned with the model, not in a separate counting stage, but the aim is the same: convert raw input into vectors the transformer can use.

No column names

Many real tables have meaningless or missing headers, such as "col_17". A tokenizer that relies on header text cannot help. Foundation models trained on synthetic data therefore learn to infer column role from the values and their relationships. Where headers exist and carry meaning, text tokenization of names can add a useful semantic signal, which is part of what text-aware tabular models exploit. Robust systems support both modes and do not fail when names are absent.

Pre-training on synthetic and real tables

Synthetic data generation is central to models like TabPFN, which pre-train on a very large number of artificial tables drawn from random causal structures. In that setting, there is no natural language corpus to learn a vocabulary from. Cell encoders are designed to be distribution-free: values are standardized per column, so the model sees comparable scales regardless of units. For models that pre-train on real tables with text, a subword tokenizer is fitted to the cell text and column names in the corpus, and the same rules about representative sampling and deduplication apply.

In-context learning

In in-context learning, the model reads a labeled set of rows as context and predicts labels for new rows without weight updates. The encoder must therefore be fixed and consistent across tables: the same procedure for turning a cell into a vector has to work on a table the model has never seen. This is the reason tabular models prefer per-table, per-column statistics computed on the fly (for example, standardizing using the context rows) over a global learned vocabulary of values. The tokenizer-like step is applied at inference time, using only the provided data.

Missing values and imputation

Missing cells need their own representation. A common approach is to add a binary indicator per cell and fill the value slot with a neutral placeholder, so that the model can learn that "missing" is informative. This ties into masked cell modeling and imputation networks: during self-supervised pre-training, cells are hidden at random and the model learns to reconstruct them. The mask token in a text tokenizer has the same role as the missing indicator in a cell encoder. A well designed encoding makes the difference between "value is zero" and "value is absent" unmistakable.

Anomaly detection

For anomaly detection, the encoder must preserve extremes. If numeric values are clipped or binned too coarsely, outliers vanish into the top bin. Quantile binning compresses tails; raw value embeddings with robust scaling preserve them. Rare categories also matter: a category seen once is often the anomaly, so a hashing or unknown bucket must not erase the signal.

Sector and semantic classification

When a table must be assigned to an industry or semantic domain, cell and column text becomes valuable. A subword tokenizer fitted on the vocabulary of that domain gives short, meaningful token sequences for terms such as units, codes and abbreviations. Heads that map a table representation to a label, such as semantic sector heads, depend on the encoder producing stable features from both the numeric and the textual parts of the table.

Continual learning

When a model is updated over time, any change to the encoder is dangerous. If a text vocabulary grows, new ids need new embeddings, and old ones must remain stable. Methods such as elastic weight consolidation and experience replay help protect learned knowledge, but they cannot fix a shifted input mapping. This is why vocabularies are frozen or only extended by appending, and why numeric encoders use standardization rules that do not depend on the training corpus.

Summary of the mapping

For how a specific architecture handles this, see /model/architecture, and for measured results see /model/benchmarks.

Practical guidance and best practices

Choosing a vocabulary size

• Start from the model size and sequence budget. Small models do better with smaller vocabularies, because the embedding and output layers consume a larger share of parameters.
• For multilingual models, use a larger vocabulary, so that each language gets reasonable coverage.
• Test a few sizes, then compare fertility on held-out text from every target domain, not only on the average.
• Round the size to a multiple of 64 or 128 for hardware friendly matrix shapes.

Building the training sample

• Mirror the pre-training data mix. If 20 percent of training is code, give code a similar share.
• Remove duplicates and boilerplate before counting.
• Include numeric strings, identifiers and dates in realistic proportions if the model will read them.
• Cap the share of any one source so it cannot dominate merges.
• Keep a held-out sample for evaluation that the trainer never sees.

Settings that matter

• Use byte fallback or byte-level pre-tokenization so there is no true unknown token.
• Split digits individually unless there is a reason not to.
• Keep case unless the task is truly case-insensitive.
• Preserve whitespace information so that decoding is exact.
• Define special tokens up front, and reserve some spare ids.
• Fix the random seed, the library version and the configuration file, and store them with the artifact.

Evaluating a tokenizer

For table-focused work

• Fit scalers and bin edges on the training split only, then freeze them.
• Prefer per-column robust scaling (median and interquartile range) when outliers are common.
• Record every learned statistic in a versioned artifact, so inference matches training. Tools such as a data catalog and schema evolution rules help track what each column meant and how it changed.
• Keep a distinct representation for missing, unknown and zero.
• If headers are meaningful, include them as optional text features, and train with random header dropout so the model also works without them.

Versioning and reproducibility

Treat the tokenizer as a model artifact. Store the file, a hash, the training corpus description, the library version and the evaluation numbers. Publishing it with the model on a hub such as Hugging Face keeps the pair together.

A short decision checklist

1. Write down the data types the model will read: prose, code, numbers, identifiers, tables.
2. Decide whether each type is better served by text tokens or by a value encoder.
3. Collect a sample that mirrors the real mix, then clean and deduplicate it.
4. Pick the algorithm. BPE is the common default, Unigram is useful when sampling matters.
5. Pick a vocabulary size from the model budget and test two or three values.
6. Turn on byte fallback and digit splitting unless there is a strong reason not to.
7. Evaluate on held-out data from each domain and inspect the rarest tokens by eye.
8. Freeze the artifact, version it, and store its hash next to the model weights.

Inspecting a trained vocabulary

Reading the vocabulary is the fastest sanity check. Sort entries by id and scan the first and last few hundred. Early ids hold single characters and bytes. Later ids hold the long merges, which reveal what the corpus was full of. If the late entries are long URLs, license headers or repeated menu text, the sample was not clean enough.

from tokenizers import Tokenizer

tok = Tokenizer.from_file("my_tokenizer.json")
vocab = tok.get_vocab()
by_id = sorted(vocab.items(), key=lambda kv: kv[1])

print("first:", by_id[:20])
print("last:", by_id[-20:])

longest = sorted(vocab, key=len, reverse=True)[:20]
print("longest:", longest)

Look for these warning signs:

• Many tokens that are long and clearly come from one document.
• Tokens that mix letters and digits in odd ways.
• A big share of tokens that never occur in a second, independent sample.
• Missing common words or units that matter to the target task.

When to extend instead of retrain

Appending tokens is safe when the old ids keep their meaning. Typical cases are adding a few domain terms, new special tokens for structure, or role markers. The new embedding rows are initialized, often as the mean of existing embeddings, and trained during a short continued pre-training phase. Retrain from scratch only when the mismatch is large, such as moving from English prose to a new script.

Common mistakes

• Training the tokenizer on a corpus that does not match the target data. The result is long, wasteful splits for the data that matters most.
• Forgetting to deduplicate. Repeated boilerplate becomes dedicated tokens.
• Changing the tokenizer after pre-training without handling embeddings. Ids then point to the wrong vectors.
• Allowing digits to merge freely, then expecting good arithmetic.
• Ignoring whitespace and casing differences between training and serving code.
• Leaving a true unknown token in the system, so rare characters silently lose information.
• Using too large a vocabulary for a small model, which leaves many embeddings undertrained.
• Treating token count as a fair cross-language cost measure.
• Fitting numeric scalers on the whole dataset before splitting, which leaks test information.
• Encoding missing values as zero, which merges two different facts.
• Applying text tokenization to numeric columns by default, when a value encoder would preserve magnitude.
• Not testing the loader and the tokenizer together. Padding and truncation bugs often hide here.

Frequently asked questions

What is tokenizer training?

Tokenizer training is the step where a tokenizer learns its vocabulary and splitting rules from a sample of data. It counts how often symbols appear together and keeps the pieces that represent the data most compactly. The output is a fixed file that maps text to integer ids and back, used before and during model training.

Is tokenizer training the same as training the model?

No. It is a separate and much cheaper process that produces a lookup table and segmentation rules, with no neural network weights. The model is trained afterward on token ids produced by that frozen tokenizer. Changing the tokenizer later breaks the match between ids and the learned embeddings.

How big should the vocabulary be?

It depends on model size, languages and data. Many text models use tens of thousands to a few hundred thousand entries. Larger vocabularies shorten sequences but enlarge the embedding and output layers, and they create more rare tokens. The practical way to choose is to try a few sizes and compare efficiency on held-out data from every target domain.

What is the difference between BPE and Unigram?

BPE builds a vocabulary from the bottom up by repeatedly merging the most frequent adjacent pair. Unigram starts with a large candidate set and prunes it down using a probabilistic model. BPE gives one deterministic segmentation, while Unigram can score several segmentations and sample among them, which is useful for regularization.

Can I reuse an existing tokenizer instead of training one?

Yes, and it is often the right choice when your data resembles the data the tokenizer was trained on. Training your own makes sense when your domain has unusual vocabulary, such as code, chemistry, product codes or non-English text. Reusing a tokenizer is also required when fine-tuning an existing model, since its embeddings are tied to that vocabulary.

How are numbers handled by tokenizers?

Text tokenizers split numbers into pieces, often at arbitrary places, so similar numbers can have unrelated token sequences. Splitting digits one by one or in regular groups makes the structure more consistent. For tabular data, a better option is often to skip text tokenization and feed the numeric value through a learned projection, which preserves magnitude and ordering.

Do tabular foundation models use tokenizers?

They use something that plays the same role. Some serialize rows as text and use a subword tokenizer. Many others treat each cell as a unit, using a numeric value encoder, a categorical embedding and a missing-value indicator, then apply attention across rows and columns. The fitted pieces, such as scaling constants and category vocabularies, are the tabular counterpart of a learned vocabulary.

What happens to missing values in a table?

They need an explicit representation, usually a mask flag combined with a neutral placeholder value. This lets the model tell the difference between a true zero and an absent value. The same idea supports self-supervised training, where cells are hidden on purpose and the model learns to predict them, which also teaches imputation.

Why does a tokenizer affect cost?

Cost often scales with the number of tokens processed, and attention cost grows faster than linearly with sequence length. A tokenizer that needs more tokens for the same content raises compute, memory and, for hosted services, usage counts. Languages or formats that the tokenizer handles poorly are therefore more expensive to process.

Can a tokenizer be changed after a model is trained?

It can, but with effort. You can append new tokens and train their embeddings, or you can retrain with a new vocabulary and a continued pre-training stage. Replacing the vocabulary outright without that work will degrade the model badly, because the learned vectors no longer line up with the ids.

Glossary

• Byte fallback: encoding unknown characters as individual bytes, so nothing is ever unknown.
• Byte pair encoding (BPE): a method that repeatedly merges the most frequent adjacent pair of symbols.
• Cell encoder: the tabular counterpart of a tokenizer, turning a table cell into a vector.
• Character coverage: the share of characters in the corpus that the base alphabet must cover.
• Detokenization: converting token ids back into readable text.
• Embedding: a learned vector assigned to a token or cell.
• Fertility: the average number of tokens per word.
• Merge rule: a recorded instruction to join two symbols into one.
• Normalization: standardizing characters and whitespace before tokenizing.
• Out-of-vocabulary: input that cannot be expressed with the vocabulary.
• Pre-tokenization: coarse splitting that limits where merges can occur.
• Quantile binning: mapping numbers to bins that each hold an equal share of data.
• SentencePiece: a library that trains and applies subword tokenizers on raw text.
• Special token: a reserved token with structural meaning, such as padding or mask.
• Subword: a unit between a character and a whole word.
• Unigram model: a tokenizer that prunes a large candidate vocabulary using likelihood.
• Vocabulary: the full set of tokens and their ids.
• WordPiece: a subword method that picks merges by likelihood gain.

Summary

Tokenizer training fits a vocabulary and splitting rules to a representative sample of data, so inputs become short sequences of integer ids. Methods such as BPE, WordPiece and Unigram trade off sequence length, vocabulary size and coverage, and the result is frozen before the model learns. Quality depends on the sample, on digit and whitespace handling, and on careful evaluation across every target domain. For tabular foundation models, the same thinking applies to cells: numeric values, categories, text and missing flags each need a fitted, frozen and versioned encoding, so that a model can read a table it has never seen. Related reading includes BPE, SentencePiece, Transformers and /model/architecture.

Related

It belongs to Foundation models from scratch in my stack, under Foundation models and applied ML. Others in the same group: