Glossary
What is Continued pre-training? Definition and meaning
Keeps training a base model on a large, unlabelled domain corpus before any instruction tuning.
Definition
Continued pre-training: Keeps training a base model on a large, unlabelled domain corpus before any instruction tuning.
In short: Extending pre-training on domain data.
When it is used
Teaching a model a new domain's vocabulary and structure, such as legal or industrial text.
Learn more
The full guide to Continued pre-training explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.
Previous term: Contextual Retrieval · Next term: Contrastive learning