Glossary
What is Tokenizer training? Definition and meaning
Fitting a vocabulary from data so that inputs are represented efficiently.
Definition
Tokenizer training: Fitting a vocabulary from data so that inputs are represented efficiently.
In short: Learning how raw input is split into tokens.
When it is used
Preparing text or structured inputs for a language or foundation model.
Learn more
The full guide to Tokenizer training explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.