SentencePiece

Overview

SentencePiece is a library that trains and applies subword tokenizers directly on raw text. It does not need the text to be split into words first. It treats a sentence as a plain stream of characters, including spaces, and learns a fixed vocabulary of pieces that can rebuild any input. This makes it a good fit for languages that do not put spaces between words, and for any setting where a single tokenizer must serve many languages or many kinds of strings.

In practice, SentencePiece is used to build the vocabulary of a language model before the model is trained. You give it a large text file and a target vocabulary size. It returns a small model file. That file turns text into integer ids and turns ids back into text. The process is fast, repeatable and fully reversible, which is why it shows up in many multilingual and domain-specific models. For tabular work, the same idea can be used to tokenize column names, category labels and free-text cells so that a model can read them. See also Data tokenization and Tokenizer training.

What it is

SentencePiece is two things at once: a training tool and a runtime encoder. The training tool reads raw sentences and builds a vocabulary of subword units. The runtime encoder uses that vocabulary to split any new sentence into those units and to map each unit to an integer.

The core idea

Think of a box of Lego bricks. A word-level tokenizer gives you one special brick for every possible model you could build, which means you need an enormous box and you still fail on a model you have never seen. A character-level tokenizer gives you only tiny single-stud bricks. You can build anything, but it takes a very long time. A subword tokenizer gives you a medium box with common shapes: plates, bars, corners. Frequent words become a single brick. Rare words are built from a few bricks. Nothing is impossible to build.

Language independence

Many older tokenizers assume that words are separated by spaces and that punctuation can be peeled off with simple rules. These assumptions break for Chinese, Japanese, Thai and many other scripts. They also break for code, chemical formulas, product codes and log lines. SentencePiece avoids the problem by not assuming anything. It treats the input as a sequence of Unicode characters and learns where to cut.

Whitespace as a symbol

SentencePiece replaces each space with a visible marker character, a lower one-eighth block that looks like an underscore. The space becomes part of the text, and the marker is attached to the start of the piece that followed the space. This detail matters because it makes encoding lossless. When the model outputs pieces, you join them and turn the marker back into a space. You get exactly the original string, with no guessing about where spaces belong.

Two main algorithms

SentencePiece supports two subword training methods:

• Byte-pair encoding, often shortened to BPE. It starts from characters and repeatedly merges the most frequent adjacent pair. See BPE.
• Unigram language model. It starts from a large candidate vocabulary and repeatedly removes the pieces that hurt a probabilistic model the least.

Both end with a vocabulary of the size you asked for. Unigram can also produce several different segmentations of the same sentence, each with a probability. That feature is used for a regularization trick described later.

What you get as output

Training produces two files. One has the extension .model and holds the full tokenizer in a compact binary form. The other has the extension .vocab and is a readable list of pieces with scores. You only need the .model file at run time.

Where it came from

SentencePiece was released by researchers at a large technology company as an open source library, with a paper that described it as a simple and language independent subword tokenizer and detokenizer for neural text processing. It appeared in the late 2010s, during the period when neural machine translation and large pre-trained language models were becoming standard.

The problem it solved

Before that time, most text pipelines relied on separate pre-tokenizers. A team would write a rule-based tokenizer for English, another for German, a segmenter for Japanese, and so on. Each had its own quirks. A model trained with one tokenizer could not be reused with another. Reproducing a result meant reproducing an entire chain of external tools.

The subword idea underneath

Subword units themselves came from earlier work on rare word handling in neural translation, where byte-pair encoding was adapted from a data compression method. Another line of work proposed the unigram language model for subword segmentation, including the idea of sampling several segmentations during training as a form of regularization. SentencePiece brought both ideas into one library with a single interface, and made them work on raw text.

Adoption

Since then, SentencePiece has been the tokenizer layer for many well-known encoder, decoder and sequence-to-sequence models. Many model families ship a SentencePiece model file next to their weights. Some newer models use other byte-level tokenizers, but the SentencePiece design (raw text in, lossless pieces out, whitespace as a symbol) shaped how people think about tokenization. For model repositories and distribution see Hugging Face.

How it works

This section walks through the full pipeline in order: normalization, training, encoding, decoding. Each step has an everyday analogy.

Step 1: Normalization

First the text is cleaned in a consistent way. By default SentencePiece applies a Unicode normalization form (commonly NFKC) so that visually identical characters have the same code. It can also collapse repeated spaces and add a dummy space at the start of the sentence.

Analogy: before sorting mail, you remove the envelopes and flatten the pages so that every letter looks the same way up. You do not change the meaning. You only remove accidental differences.

Normalization rules are stored inside the model file. This is important. The same rules are used when you train and when you encode, so you never get a mismatch between the two.

Step 2: Choosing the algorithm and vocabulary size

You pick model_type (unigram, bpe, char or word) and vocab_size. The vocabulary size is the number of pieces in the final list. Typical values range from a few thousand for small domain models to a few hundred thousand for broad multilingual models.

Analogy: this is the size of the toolbox. A tiny box is portable but you need many steps for each job. A giant box handles anything but is heavy and hard to search.

Step 3: Training with BPE

When you choose BPE, training works like this:

1. Start with each character as its own piece.
2. Count how often every adjacent pair of pieces appears in the data.
3. Merge the most frequent pair into a new piece.
4. Repeat until the vocabulary has the requested size.

Analogy: imagine a typist who notices that she keeps typing "t", "h", "e" together. She makes a stamp that prints "the" in one press. Then she notices "the" and a space are common, and makes a bigger stamp. Over time she has stamps for frequent patterns.

BPE encoding then applies the learned merges in order of priority. Encoding is deterministic. The same text always gives the same pieces.

Step 4: Training with the unigram model

The unigram approach runs in the opposite direction:

1. Start with a very large seed vocabulary, made of frequent substrings.
2. Assign each piece a probability using expectation maximization.
3. For every piece, estimate how much the total likelihood would drop if it were removed.
4. Remove a fraction of the pieces that cost the least.
5. Repeat until the vocabulary reaches the target size.

Analogy: you start with a huge wardrobe and want a suitcase. You try removing each item and ask, "How much worse is my trip without this?" You drop the items you will miss least, keep doing it, and end with a packed suitcase.

Single characters are always kept, so any text can still be encoded.

Step 5: Encoding with Viterbi

For a unigram model, the encoder finds the segmentation with the highest total probability. It uses dynamic programming (the Viterbi algorithm) over all ways to cut the sentence. This is fast because it reuses partial results.

Analogy: planning the cheapest route through a city where every street has a toll. You do not try all routes blindly. You keep the cheapest known cost to reach each intersection.

Step 6: Subword regularization

Because the unigram model gives a probability to every segmentation, you can sample from the top candidates instead of always using the best one. During training of a downstream model, the same sentence may then appear as slightly different token sequences in different epochs. The model learns to be robust to segmentation noise.

Analogy: a student who sees the same words written in different handwriting learns to read the letters, not just one font.

Step 7: Mapping to ids and special tokens

Each piece has an integer id. SentencePiece reserves ids for special symbols: unknown, beginning of sentence, end of sentence and padding. You can also add user-defined symbols, such as a separator token or a task marker. These are matched before ordinary segmentation, so they are never split.

Step 8: Byte fallback

With byte fallback turned on, characters that are not in the vocabulary are broken into their UTF-8 bytes, and each byte has its own piece. The unknown token almost disappears. Any string can be encoded and decoded back exactly.

Analogy: if you cannot find a pre-cut part, you fall back to raw material and cut it yourself, one grain at a time.

Step 9: Decoding

Decoding concatenates the pieces and replaces the whitespace marker with a normal space. Because the space was part of the text from the start, no language-specific detokenizer is needed.

Cost and speed

Training on tens of millions of sentences takes minutes to hours on a CPU, depending on the algorithm and the settings. Encoding runs at high speed, with a native C++ core and wrappers for Python and other languages. For very large corpora you can train on a sampled subset, and use the input_sentence_size option and shuffling to keep memory under control.

Other options worth knowing

Several smaller options change the result in useful ways.

• max_sentence_length sets the longest line that training will accept. Longer lines are skipped.

• shuffle_input_sentence randomizes which lines are sampled when the corpus is larger than input_sentence_size.

• split_by_whitespace stops a piece from crossing a space. Turn it off if you want pieces that span short phrases, such as common multi-word names.

• split_by_unicode_script keeps pieces from mixing scripts, so a Latin letter and a Cyrillic letter never share a piece.

• split_by_number separates digits from letters, which helps with codes such as A100B.

• treat_whitespace_as_suffix moves the marker to the end of a piece. This suits some languages and code-like text better.

• add_dummy_prefix controls whether a space is placed before the first word. It makes the first word and a later word look the same.

• remove_extra_whitespaces trims leading, trailing and repeated spaces during normalization.

• control_symbols are reserved ids that the tokenizer never produces from text, which is useful for markers you insert yourself.

• hard_vocab_limit can be turned off so that training does not fail when the corpus cannot supply enough distinct pieces for the requested size.

Why a tokenizer is also a compressor

A tokenizer is a kind of compression scheme. A good vocabulary turns a long string of characters into a short list of ids with little loss of information. The same idea sits behind classic data compression, and it explains why frequent patterns get their own pieces. The better the match between the training text and the real text, the shorter the id list becomes. Short lists are cheap to process. They also let more content fit in the fixed context window of a model.

Tokenization and the embedding layer

After tokenization, each id is looked up in an embedding table to get a vector. The table has one row per piece. A vocabulary of thirty-two thousand pieces and vectors of one thousand numbers gives about thirty-two million values in that layer alone. A larger vocabulary raises this cost. It also means each row is trained by fewer examples. This trade-off is the main reason vocabulary size is a design choice, not a default.

Key concepts and terms

A worked example

This example trains a tokenizer on a small file and uses it in Python. The file name and settings are placeholders.

Training

import sentencepiece as spm

spm.SentencePieceTrainer.train(
    input="corpus.txt",            # one sentence per line
    model_prefix="demo_tok",       # writes demo_tok.model and demo_tok.vocab
    vocab_size=8000,
    model_type="unigram",          # or "bpe"
    character_coverage=0.9995,     # high for rich scripts, 1.0 for small alphabets
    byte_fallback=True,
    pad_id=0, unk_id=1, bos_id=2, eos_id=3,
    user_defined_symbols=["<sep>", "<col>", "<val>"],
)

Encoding and decoding

sp = spm.SentencePieceProcessor(model_file="demo_tok.model")

text = "Customer churn rate by region"
pieces = sp.encode(text, out_type=str)
ids = sp.encode(text, out_type=int)

print(pieces)
# example shape of output: ['▁Customer', '▁ch', 'urn', '▁rate', '▁by', '▁region']
print(ids)
print(sp.decode(ids))   # 'Customer churn rate by region'

The exact pieces depend on your training data, so the list above only shows the form. The key point is that the leading marker tells you where spaces were, and that decode returns the original string.

Sampling for regularization

for _ in range(3):
    print(sp.encode(text, out_type=str,
                    enable_sampling=True, alpha=0.1, nbest_size=-1))

Each call may give a different split of the same sentence. Use this only during training of a downstream model, never at inference.

Numeric walk-through of BPE

Suppose the toy corpus has these word counts:

Start from characters. The most frequent pair is e followed by s, which appears in newest and widest: 6 + 3 = 9 times. Merge to es. Next the pair es and t appears 9 times. Merge to est. Then l and o appear 7 times in low and lower. Merge to lo. Then lo and w give low. After a few merges the vocabulary has est and low as pieces, so the unseen word lowest can be written as low + est. This is how a subword vocabulary handles words it has never seen.

Using it for column names and categories

A table has headers such as order_total_usd and categories such as Premium Annual. You can train a SentencePiece model on a file of such strings and use it to turn them into ids:

names = ["order_total_usd", "customer_segment", "Premium Annual", "Basic Monthly"]
for n in names:
    print(n, "->", sp.encode(n, out_type=str))

With user-defined symbols like <col> and <val> you can build a flat sequence such as <col> customer segment <val> Premium Annual and feed it to a text encoder. This is one way to give a tabular model semantic information about columns.

Strengths and limitations

Strengths

• Works on raw text with no language-specific pre-tokenizer.
• Lossless round trip between text and ids.
• One file holds vocabulary and normalization, which makes results reproducible.
• Supports both BPE and unigram, and a character or word mode for baselines.
• Subword regularization can improve robustness for small and noisy datasets.
• Fast native implementation with bindings for common languages.
• Byte fallback makes the unknown token rare.
• Good fit for multilingual vocabularies, code-like strings and identifiers.

Limitations

• The vocabulary is fixed after training. New domains with new terms may be split into many pieces.
• Numbers are often cut in inconsistent ways, which hurts arithmetic and numeric reasoning.
• Training is frequency based, so rare languages in a mixed corpus can be under-represented unless you balance the data.
• Tokenization does not know meaning. Related words may share no pieces.
• Changing the tokenizer later means retraining or at least re-embedding the model.
• Normalization defaults may change text you wanted to keep, such as case or special Unicode forms. You must set rules deliberately.
• Very large vocabularies need large embedding tables, which cost memory.

How it compares with alternatives

A quick rule of thumb: if your text has spaces and a single language, many tokenizers work. If you have many scripts, mixed content or identifiers, a language-independent tool such as SentencePiece reduces the engineering work. If you want guaranteed coverage of any bytes, use byte fallback or a byte-level tokenizer. For the merge-based method in detail see BPE.

Role in tabular foundation models

Tabular foundation models learn from rows and columns rather than sentences. Even so, tokenization appears in several places, and SentencePiece is a natural building block in some of them.

Column names and text cells

Real tables have header strings, free text notes, product names, addresses and categorical labels. Some models are built to use these strings, and they need a way to convert them to vectors. A subword tokenizer gives them a fixed vocabulary that handles typos, abbreviations and new brand names without breaking. A tokenizer trained on a mixed collection of headers and cell values will learn pieces such as id, date, amt and _usd that recur across tables.

Some tabular foundation models intentionally ignore column names, so that the same model can work on anonymous data. In that case, tokenization of names is not used at all. Both designs exist, and the right choice depends on whether semantic hints help for the task. For a broader view of how models turn mixed inputs into tokens, see Data tokenization.

Mixed types

A row mixes numbers, categories, dates, booleans and text. Numeric values are usually handled with scaling or binning and are not sent through a text tokenizer, because splitting digits into pieces loses magnitude. Categorical values may be encoded with ids, learned embeddings or text. SentencePiece fits the text and the high-cardinality categorical cases, particularly when the labels are descriptive strings. See FT-Transformer for a model that turns each feature into its own token.

Missing values

A tokenizer can represent a missing value with a dedicated user-defined symbol, so that "missing" is a visible token rather than an empty string. Models that do imputation benefit when missingness is explicit. For more on this, see Imputation networks.

Pre-training on synthetic or real tables

When a model is pre-trained on many generated tables, any text fields are usually generated from limited vocabularies. When it is pre-trained on real tables, a subword tokenizer built from a broad sample of headers and values helps generalize to new schemas. See Synthetic data generation and TabPFN for an example of a model trained mostly on synthetic tables.

In-context learning

In-context learning means the model sees labeled example rows and a query row in the same input and predicts without weight updates. If rows are serialized as text, the number of tokens per row decides how many examples fit in the context window. A tokenizer with lower fertility on your data means more rows fit. This is a direct, practical link between vocabulary design and model quality. You can read more on the architecture side at /model/architecture.

Semantic and sector classification

Classifying a company, product or transaction into a sector or category depends heavily on text such as names and descriptions. A multilingual SentencePiece vocabulary lets one classifier read descriptions in several languages and spellings. The classifier can then be combined with the numeric and categorical columns of the row.

Anomaly detection and continual learning

For anomaly detection on log-like or text-heavy tables, a subword tokenizer makes rare strings representable, so that unusual messages produce unusual token patterns rather than a flood of unknown tokens. For continual learning, changing the tokenizer is hard, because embeddings are tied to ids. A good practice is to train the vocabulary on broad data once, reserve spare ids, and add new user-defined symbols only when required. See Anomaly detection and Continued pre-training.

For results and background on tabular foundation models more generally, see /model/research and /model/benchmarks.

Practical guidance and best practices

Prepare the training text

• Use data that looks like what the model will see. Mix sources in the proportions you care about.
• Remove obvious noise such as binary blobs and broken encodings.
• Keep one sentence or one record per line.
• Balance languages. If one language has ten times more text, sample the others up or the first one down.
• Include samples of numbers, dates, codes and identifiers if they matter in your task.

Choose settings with care

Measure tokenizer quality

• Compute fertility on a held-out sample. Lower is better as long as pieces stay meaningful.
• Check the rate of unknown tokens. It should be near zero with byte fallback.
• Read a sample of segmentations by eye. Look for strange splits in names and codes.
• Compare compression across languages. If one language needs many more tokens, it will cost more and fit less context.
• Run a small downstream experiment. The best tokenizer is the one that helps your task.

Keep it reproducible

• Store the .model file with the model weights.
• Record the SentencePiece version and the training command.
• Never retrain the tokenizer silently. Even a small data change alters ids.
• Write a test that encodes a fixed set of strings and compares the ids with a saved result.

Tips for table-oriented use

• Add separators such as <col> and <val> as user-defined symbols so the model can tell names from values.
• Lowercase or normalize header strings consistently, and split snake_case and camelCase before training or use rules that treat underscores as separators.
• Do not send raw numbers through the tokenizer if magnitude matters. Use numeric encodings.
• Cap very long text cells before tokenization to keep rows within the context length.
• Use a dedicated missing token rather than an empty string.

A short checklist before you train

1. Write down what the tokenizer must read: plain prose, code, identifiers, headers, or a mix.
2. Collect a sample that matches that mix, and hold out a small part for testing.
3. Decide on special tokens and user-defined symbols first.
4. Pick a normalization rule and write it down.
5. Train two or three vocabulary sizes.
6. Compare fertility, unknown rate and a small downstream score.
7. Freeze the winner, tag it with a version, and store it with the weights.

Handling numbers on purpose

Numbers deserve a clear policy, because tokenizers cut them in uneven ways. One value such as 1234567 may become one piece, two pieces or seven, depending on the training data. That unevenness makes it hard for a model to learn place value.

• Split digits into single pieces when the model must reason about number length or digits.
• Replace numbers by a placeholder token and send the value through a separate numeric channel when the exact magnitude matters.
• Bin numbers into ranges and write the bin as a word when only the rough size matters.
• Keep units, such as a currency code, as ordinary pieces so the model can still read them.

Handling case, punctuation and symbols

Case carries meaning in names, codes and abbreviations. Lowercasing everything shrinks the vocabulary but erases that signal. If you lowercase, do it as an explicit normalization rule so it is applied the same way every time. Punctuation inside identifiers, such as underscores, hyphens and dots, often marks structure. Make sure your tokenizer has pieces for them, or add them as user-defined symbols.

Working with very large corpora

• Sample sentences uniformly rather than taking the first lines of a file, which are often biased.
• Cap the length of a single sentence so that a few giant lines do not dominate training.
• Train on a machine with enough memory for the seed vocabulary of the unigram method.
• Use several threads where the library allows it, and fix the random seed for repeatable results.
• Keep a log of the exact input files and their sizes.

Evaluating on a table-style task

Suppose you want a model to read a row as text. Serialize ten thousand rows in your chosen format. Count tokens per row with each candidate tokenizer. A tokenizer that cuts the average row from 120 to 90 tokens lets you fit about a third more example rows in the same context window. Then check whether accuracy changes. Shorter is not always better, but when accuracy is equal, shorter is cheaper.

Versioning and compatibility

The tokenizer is part of the public contract of a model. If other teams or tools call your model, they depend on the exact ids. Treat any change as a breaking change. Give each tokenizer a version number, keep old versions available, and document the special ids. When you must extend a vocabulary, append new pieces at the end so existing ids stay valid, and initialize the new embedding rows sensibly before continuing training.

Debugging odd tokenization

When a string splits in a strange way, work through these checks in order.

1. Print the pieces and the ids side by side.

2. Check whether normalization changed the input before it was cut.

3. Search the vocabulary file for the expected piece.

4. Check whether a user-defined symbol or control symbol is catching the text.

5. Compare against a second vocabulary size to see whether the odd split is stable.

6. If the string is rare in the corpus, add examples of it and retrain, or add it as a user-defined symbol.

Common mistakes

1. Training on too little text. A tiny file gives a weak vocabulary and many unknown pieces.
2. Training on one language and deploying on many. The model will fragment the others heavily.
3. Forgetting to set the same special ids in the model and the tokenizer. Padding and end tokens will mix up.
4. Changing the normalization rule between experiments and then comparing results.
5. Ignoring numbers. Inconsistent splitting of digits makes numeric fields unstable.
6. Using sampling at inference. Subword regularization is for training only.
7. Setting the vocabulary too large for the data. Many pieces will be seen only a few times and their embeddings will be poor.
8. Setting the vocabulary too small. Sequences get long and the context window fills up.
9. Not testing round trips. Always check that decode returns the original text for a sample.
10. Retraining the tokenizer after the model has been trained, without re-embedding, which silently breaks the model.
11. Treating the whitespace marker as a normal character in downstream string handling.
12. Forgetting that lowercase and uppercase forms can be separate pieces unless you normalize case.

Frequently asked questions

What is SentencePiece used for?

SentencePiece is used to build and apply the vocabulary that a language model reads. You train it on raw text, choose a vocabulary size, and get a model file that turns any string into integer ids and back again. It is popular for multilingual models, for domain-specific text, and for any pipeline where a reproducible, language-independent tokenizer is needed.

Is SentencePiece a tokenizer or an algorithm?

It is a library that implements tokenization algorithms. The library supports byte-pair encoding and a unigram language model, plus character and word modes. People often say "SentencePiece" to mean the unigram variant, because that was the most distinctive part of the original release, but both methods are available under the same interface.

What is the difference between SentencePiece and BPE?

BPE is a training algorithm that builds a vocabulary by merging frequent pairs of symbols. SentencePiece is a toolkit that can run BPE, but also handles raw text, whitespace, normalization and a second algorithm, the unigram model. The main practical difference is that SentencePiece does not need a separate word-splitting step before training.

Why does SentencePiece show a special underscore-like symbol?

That symbol stands for a space. SentencePiece keeps spaces as part of the text so that encoding is lossless. When the pieces are joined and the symbol is replaced with a normal space, you get the exact original sentence. This removes the need for language-specific detokenizers.

How big should the vocabulary be?

There is no single answer. Small domain models often do well with eight to thirty-two thousand pieces, while models that cover many languages use much more. A larger vocabulary shortens sequences but increases the size of the embedding table and leaves rare pieces undertrained. The best method is to try a few sizes and measure fertility and a downstream metric.

Can SentencePiece handle numbers and tabular data?

It can tokenize numbers as text, but the splits may be inconsistent, so numeric columns are usually better handled with scaling, binning or dedicated numeric encoders. SentencePiece is more useful for column names, category labels and free-text cells. A common pattern is to combine text tokens for strings with separate numeric channels for numbers.

What is byte fallback?

Byte fallback is an option that encodes characters missing from the vocabulary as their UTF-8 bytes instead of using an unknown token. It means any input can be represented and decoded back exactly. It costs a few extra pieces in the vocabulary, but it is a good safety net for noisy or rare characters.

What is subword regularization?

It is a training technique in which the same sentence is segmented in different ways across epochs, by sampling from the unigram model. The downstream model sees small variations in tokenization and learns to be less sensitive to them. It helps most on small or noisy datasets and should be turned off at inference.

Can I change the tokenizer after training a model?

Not safely. The model's embeddings are tied to the token ids. If you retrain the tokenizer, the ids change and the model no longer matches. You can extend a vocabulary carefully by adding new pieces and initializing their embeddings, then continue training, but this needs planning. See Continued pre-training.

Does SentencePiece work for tabular foundation models?

It can play a supporting role. Models that read column names or text cells can use a subword tokenizer to turn those strings into tokens. Models that work on anonymous numeric tables may not use a text tokenizer at all. The choice depends on whether the model is meant to use semantic information from the table or to be schema-agnostic.

Glossary

• Byte fallback: encoding unseen characters as bytes instead of unknown tokens.
• BPE: a merge-based method that builds subword units from frequent pairs.
• Character coverage: the fraction of characters in the corpus that the vocabulary must represent directly.
• Detokenizer: the step that turns tokens back into text.
• Embedding table: the matrix that maps token ids to vectors.
• Fertility: average tokens per word, a measure of tokenizer efficiency.
• Lossless tokenization: tokenization from which the exact original text can be rebuilt.
• Normalization: mapping equivalent text forms to one standard form.
• Piece: one entry in the vocabulary.
• Special token: a reserved symbol with a structural role.
• Subword: a unit between a character and a word.
• Unigram language model: a probabilistic vocabulary method that prunes a large seed set.
• Viterbi decoding: dynamic programming that finds the best segmentation.
• Vocabulary size: the number of pieces in the tokenizer.
• Whitespace marker: the visible symbol standing for a space inside pieces.

Summary

SentencePiece trains subword tokenizers directly on raw text and keeps encoding lossless by treating spaces as a symbol. It supports BPE and a unigram model, offers normalization, special tokens, byte fallback and subword regularization, and ships as a single model file that is easy to reproduce. Its main strengths are language independence and simplicity. Its main limits are a fixed vocabulary and awkward handling of numbers.

For tabular foundation models, SentencePiece is useful wherever text meets a table: column names, category labels, free-text cells and descriptive fields used for semantic or sector classification. Numeric values are better handled by dedicated encoders. Choose the vocabulary size by measurement, add structure tokens before training, test round trips, and treat the tokenizer file as part of the model. For related reading, see Tokenizer training, BPE, Data tokenization, Transformers, /model/architecture and /model/research.

Related

It belongs to Foundation models from scratch in my stack, under Foundation models and applied ML. Others in the same group: