Glossary
What is BPE? Definition and meaning
A tokenization algorithm that repeatedly merges the most frequent adjacent symbol pairs into new tokens.
Definition
BPE: A tokenization algorithm that repeatedly merges the most frequent adjacent symbol pairs into new tokens.
In short: Byte-pair encoding.
When it is used
The most common subword tokenizer for language models.
Learn more
The full guide to BPE explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.
Previous term: BM25 · Next term: BrainScaleS