Continued pre-training

Overview

Continued pre-training is the practice of taking an already pre-trained model and training it further on a large body of unlabeled, domain-specific data using the same objective as the original pre-training. For a language model that objective is next-token prediction. The model keeps its general skills and absorbs the vocabulary, facts, style and structure of a new field such as law, medicine, finance, code or industrial logs.

It sits between full pre-training from scratch, which is very expensive, and SFT, which teaches behavior with a small labeled set. When a model keeps misreading the terms of a field, SFT cannot fix it, because the model never learned those terms. Continued pre-training can. It is also called domain-adaptive pre-training or continual pre-training. This article explains how it works, how to mix data, how to avoid forgetting, how to evaluate, and how the idea applies to models that learn from raw tables.

What it is

Continued pre-training means resuming self-supervised training of a base model on new text or new data. "Self-supervised" means the labels come from the data itself. For text, the label for each position is the next token. No human annotation is needed, so the data can be very large.

Think of a doctor who finished general medical school and now moves to a hospital that specializes in marine biology research. SFT would be a short course on how to fill in the hospital's forms. Continued pre-training is spending a year reading the field's literature until its language feels natural. After that year, the doctor still has general medicine, but also thinks in the new vocabulary.

What it teaches

• Vocabulary and phrasing. Rare terms, abbreviations and notation of the field.
• Facts and relations. Knowledge that appears in the domain corpus, to the extent the model can absorb it.
• Structure. Document layouts, formulas, code idioms, table shapes and log formats.
• Style. The register used by experts.

What it does not do

• It does not teach the model to follow instructions. The result is still a base model that continues text. Apply SFT afterward.
• It does not guarantee reliable recall of every fact. Models absorb frequent patterns better than rare facts. For exact lookups, add retrieval, see naive RAG and advanced RAG.
• It does not remove the need for evaluation. A model can sound more fluent in a domain while being no more accurate.

Names you will meet

Where it came from

The idea follows from transfer learning and domain adaptation. In the late 2010s, language models pre-trained on general text were shown to benefit from a second round of pre-training on text closer to the target. Studies on domain-adaptive and task-adaptive pre-training around 2020 found that a model of the BERT family, further pre-trained on biomedical, scientific, news or review text, did better on tasks in that domain than the general model, even when the general model was larger or trained longer.

Domain-specific models followed. Groups released encoders and decoders adapted to science, clinical text, law and finance, and code models were built by continuing the training of general models on source code. As open base models became strong, continuing their pre-training became a standard route for building regional language models, since teams could take a good English base and extend it with a large corpus in another language. Training recipes also began to include a late "mid-training" stage with curated data, which is a close cousin of continued pre-training.

Alongside this, research on continual learning, see EWC, synaptic intelligence and LwF, studied how to learn new data without losing old skills. That work shapes the practical advice on replay and learning rates today. Tooling for large runs comes from Megatron-LM, DeepSpeed ZeRO, FSDP, Colossal-AI and NCCL.

How it works

Step 1: Pick a base model

Choose a base model, not an instruction-tuned one, unless you have a reason. Base models are easier to extend because they have not been narrowed by post-training. Check the license, the size, the context length and the tokenizer.

Step 2: Assemble the domain corpus

Gather large amounts of high-quality domain text. Typical sources are documents, manuals, papers, reports, code and logs that you have the right to use. The corpus is often measured in billions of tokens. A few million tokens can shift style, but little more.

A library analogy fits here. You are stocking a new wing of a library. You want many books, but not duplicates, not damaged copies and not books on the wrong subject.

Step 3: Clean and filter

Remove duplicates, near duplicates, boilerplate, broken encodings, personal data and low-quality text, see deduplication and data tokenization. Use quality classifiers or simple heuristics. Dedup matters a lot: repeated text is memorized and raises the risk of leaking private content.

Step 4: Decide on the tokenizer

If the domain has many terms that the tokenizer splits into long chains of pieces, such as chemical names, code symbols or another language, consider extending the vocabulary, see BPE, SentencePiece and tokenizer training. New tokens need new embeddings, which start untrained and must be learned during the run. Many teams keep the original tokenizer for safety and simplicity.

Step 5: Mix in replay data

Blend the new corpus with a share of data like the original pre-training mix. This replay data reminds the model of its earlier skills. Without it, the model drifts toward the new domain and degrades elsewhere.

Step 6: Set the learning rate and schedule

Use a learning rate lower than the original peak, a short warmup, and a decay. Re-warming to a high rate and decaying again is known to cause a temporary loss spike, and some forgetting. A moderate peak rate with cosine decay is a common compromise.

Step 7: Train at scale

Run the standard causal language modeling loss over packed sequences. Use sharded data parallelism, mixed precision and efficient attention, see data parallelism, tensor parallelism, pipeline parallelism, mixed precision and FlashAttention.

Step 8: Monitor and evaluate

Track loss on a held-out domain set and on a general set. Run task benchmarks for the domain and for general ability. Save checkpoints on a schedule.

Step 9: Post-train

Apply SFT on instructions, then preference tuning such as DPO, to turn the adapted base into a usable assistant or task model.

The objective in one line

For a sequence of tokens x1 to xT, the loss is the average negative log probability of each token given the ones before it:

L = - (1/T) * sum over t of log p(x_t | x_1 ... x_{t-1})

This is the same loss as in original pre-training. Only the data changes.

How the data share changes the outcome

Imagine the final model as a blend of two learners. With almost all new data, the model is mostly the domain learner. With a balanced blend, it keeps both. The next table shows qualitative trends, not measurements.

Key concepts and terms

A worked example

This example continues the pre-training of a small open base model on a domain text corpus using Hugging Face tools. Paths and names are placeholders.

import torch
from datasets import load_dataset, interleave_datasets
from transformers import (AutoModelForCausalLM, AutoTokenizer,
                          TrainingArguments, Trainer,
                          DataCollatorForLanguageModeling)

name = "example-org/base-1b"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.bfloat16)

domain = load_dataset("text", data_files="corpus/domain/*.txt", split="train")
general = load_dataset("text", data_files="corpus/general/*.txt", split="train")

# 70 percent domain, 30 percent replay
mix = interleave_datasets([domain, general], probabilities=[0.7, 0.3], seed=7)

def pack(batch, block=2048):
    ids = tok(batch["text"])["input_ids"]
    flat = [t for seq in ids for t in seq + [tok.eos_token_id]]
    n = (len(flat) // block) * block
    return {"input_ids": [flat[i:i + block] for i in range(0, n, block)]}

mix = mix.map(pack, batched=True, remove_columns=["text"])

args = TrainingArguments(
    output_dir="out/cpt",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=32,
    learning_rate=2e-5,
    lr_scheduler_type="cosine",
    warmup_ratio=0.01,
    max_steps=20000,
    bf16=True,
    gradient_checkpointing=True,
    save_steps=1000,
    logging_steps=50,
)

Trainer(model=model, args=args, train_dataset=mix,
        data_collator=DataCollatorForLanguageModeling(tok, mlm=False)).train()

Measuring the effect with perplexity

Perplexity is the exponential of the average loss. A lower value means the model finds the text more predictable. Suppose the base model has a perplexity of 14 on held-out domain text and 9 on general text. After continued pre-training with a good mix, the domain value might fall toward 8 while the general value stays near 9 or rises a little. If the general value jumps to 15, forgetting is strong and you should raise the replay share or lower the rate. The numbers here are illustrative only.

Perplexity is useful but incomplete. Always follow it with task evaluation, because lower perplexity does not guarantee better answers.

A small run to validate the plan

Before a big run, do a pilot with a fraction of the data and a small model. Check that the loss goes down, the general loss stays stable, and the data pipeline has no bugs. Fix problems at small scale, where they are cheap.

Scaling and compute planning

A continued pre-training run is mostly an engineering problem of keeping accelerators busy and data flowing. This section gives a feel for the sizes involved.

A rough compute estimate

A common rule of thumb says training compute is about six times the number of parameters times the number of tokens. For a model with 7 billion parameters and a corpus of 20 billion tokens, that is about 8.4e20 floating-point operations. Divide by the sustained throughput of your cluster to estimate time. Real runs reach only a fraction of peak speed, often somewhere between a third and a half, so build in a margin. Treat this as a planning estimate and not a promise.

Parallelism choices

Most mid-size runs need only sharded data parallelism. Reach for tensor and pipeline splits when the model or the context is very large.

The data pipeline

At this scale, the input pipeline matters. Pre-tokenize the corpus once, store it in shards, and stream it with a deterministic order, so you can resume at an exact position. Use a data loader with enough workers so that accelerators never wait. Log the throughput in tokens per second and watch for dips, which usually mean a storage or network issue.

Mixing sources in detail

A corpus is rarely uniform. It may contain manuals, reports, web pages, code and tables. Each source has its own quality and its own role.

Source weighting

Give each source a sampling weight. High-quality, high-value sources can be repeated a few times, while low-quality sources get a small weight. Repeating data a small number of times is usually harmless, but many repeats lead to memorization.

Quality filtering

Simple rules go a long way: minimum length, ratio of symbols, language detection, and removal of repeated lines. A small classifier trained to separate good samples from bad ones can add more. Always inspect random samples by eye after filtering. Numbers alone do not show what went wrong.

Staging the data

Some teams order the data in stages. Early stages use broad data, and later stages use the highest-quality domain data while the learning rate decays. This late stage, sometimes called annealing, tends to have outsized influence on final quality, so it deserves the cleanest data.

Evaluation in depth

What to measure

1. Domain perplexity on held-out domain text.
2. General perplexity on held-out general text.
3. Domain tasks such as question answering, extraction or classification in the domain, run few-shot so that the base model can be tested without SFT.
4. General benchmarks for reasoning, knowledge and language, see lm-evaluation-harness.
5. Safety and bias checks, since new data can shift behavior.

Few-shot probes for base models

A base model cannot follow instructions, so evaluate it by showing a few worked examples in the prompt and asking it to continue the pattern, see few-shot learning. This works before any SFT and gives a fair view of what the pre-training added.

Watching for contamination

Search the corpus for your evaluation items before training. If an exact or near match is present, remove it from the corpus or from the evaluation. A model that has seen the test set will look better than it is.

Strengths and limitations

Strengths

• Teaches the language of a field in a way that SFT cannot.
• Uses unlabeled data, which is cheap and plentiful in most organizations.
• Builds on an existing strong model, so it needs a small fraction of the compute of training from scratch.
• Improves downstream tasks in the domain, including after later SFT.
• Can extend a model to a new language or modality-specific format.
• Fits into a pipeline: continue pre-training, then SFT, then preference tuning.

Limitations

• Still expensive. Billions of tokens and many accelerator hours are common.
• Forgetting risk. Without replay and a careful rate, general ability can fall.
• Needs a large, clean corpus. Noisy data adds noise to the model.
• Gains in facts are uneven. Rare facts are absorbed poorly.
• Needs a new round when the base model changes.
• Hard to evaluate. Domain benchmarks are scarce, and perplexity can mislead.
• Possible memorization of sensitive text if the corpus is not filtered.

How it compares with alternatives

A common ordering is to try prompting and retrieval first, continue pre-training when vocabulary and structure are the problem, then SFT for behavior.

Role in tabular foundation models

Tabular foundation models learn from raw tables, with rows and columns, mixed types, missing values and often no column names. Table-native models such as TabPFN, TabICL, TabSTAR and XTab are pre-trained on large numbers of tables, many of them synthetic, then used with in-context learning. Continued pre-training is the natural way to extend such a model after its first training run. See /model/research and /model/architecture.

What continued pre-training means for a table model

The self-supervised objective for tables is not next-token prediction in the plain sense. It is predicting hidden parts of a table from the visible parts, such as masked cells, held-out target columns or missing rows. This is close to masked cell modeling and to self-supervised learning on tables. Continued pre-training then means resuming that objective on a new pool of tables.

New pools of tables

Mixed types and missing values

Real tables mix numbers, categories, dates, free text and identifiers. Continued pre-training can raise the share of hard cases: heavy-tailed numbers, high-cardinality categories, and columns where missingness depends on the value itself. It can also increase the variety of missing patterns, whether completely random, random given other columns, or tied to the missing value itself. A model that has seen all three patterns during training is less likely to assume the easiest one at deployment.

No column names

Many production tables lack meaningful headers. Continued pre-training that randomly drops, shuffles or anonymizes column names teaches the network to infer a column's role from its values: ranges, spread, repeated entries, formats and relations with other columns. This is important for semantic tasks such as sector classification, where names might be unavailable or misleading.

Imputation and anomaly detection

Imputation is the same task as the pre-training objective: predict hidden cells. So extra pre-training on tables with realistic masks directly improves it, see imputation networks. For anomaly detection, the model learns what a normal row looks like given its neighbors, and unusual rows have low likelihood or large reconstruction error, see anomaly detection and autoencoders.

Continual learning without forgetting

A table model that keeps receiving new sectors faces the same trade-off as a language model. The advice carries over.

• Keep a replay buffer of earlier tables, including synthetic ones.
• Lower the learning rate for later rounds.
• Run a fixed regression suite after each round, see /model/benchmarks.
• Version each round and keep the ability to roll back.
• Consider adapters per sector when the sectors differ strongly, see adapters.

Language-model route

If a language model reads tables as text, continued pre-training on serialized tables, schema descriptions and data documentation can teach it table structure and the vocabulary of column names. This helps it parse and describe tables, though it is a weaker fit than a table-native network for numeric prediction. See transformers for the shared building blocks.

Practical guidance and best practices

Planning

• Define success in advance with domain tasks and a general regression suite.
• Estimate the token budget. Small corpora of a few million tokens call for a light-touch method such as LoRA or just SFT.
• Decide whether you need vocabulary changes. Avoid them unless the benefit is clear.
• Plan the post-training steps now, since they affect which base model to start from.

Data

• Keep provenance records of every source and its license or permission status.
• Deduplicate within and across sources, and against evaluation sets.
• Remove personal and secret data before training, see presidio and GDPR.
• Balance sources so that no single noisy source dominates.
• Hold out a domain validation set drawn from the same sources but never trained on.

Training

• Use a moderate peak learning rate, below the original peak, with a short warmup and a decay.
• Include replay data, often a quarter to a third of the mix.
• Checkpoint often and evaluate at each checkpoint on both domain and general sets.
• Use sharding and mixed precision, see DeepSpeed ZeRO and FSDP.
• Run on suitable hardware, see NVIDIA H100, H200 and B200, with fast links.
• If the run is interrupted, resume from a checkpoint with the data position restored, so the model does not see the same examples twice by accident.

After training

• Run SFT and preference tuning on the adapted base.
• Re-run the full evaluation suite on the final model.
• Quantize for serving if needed, see AWQ and GPTQ.
• Track the model, data and configs together, see MLflow.

Choosing between continued pre-training and cheaper options

Ask four questions. Is the failure about vocabulary or structure? Do you have a large unlabeled corpus? Can you afford the compute? Do you have a way to measure the gain? If the answers are yes, yes, yes and yes, continue pre-training. If the failure is about format or tone, use SFT. If the failure is about missing facts, add retrieval.

Data governance for large corpora

Large corpora raise legal and ethical questions. Keep a register of sources with their rights and restrictions. Remove personal data and secrets before training, since a model can reproduce what it has seen. Document how people can ask for removal, see GDPR and KVKK, and think about data lineage so you can trace which sources went into which model version.

Re-running when the base changes

Because the work is tied to one base model, a new base release means a new run. Make the pipeline repeatable: versioned corpus, versioned filters, a config file for every run and a script that reproduces the whole flow. The corpus is the lasting asset, and the weights are an output you can regenerate.

Worked scenario: an industrial domain

Consider a team that wants a model to understand maintenance logs, sensor descriptions and equipment manuals. The base model reads general English well but stumbles on part codes, abbreviations and unit notation. The plan could look like this.

1. Collect manuals, logs and reports, then remove duplicates and personal data.
2. Measure the base model's perplexity on a held-out slice, and run a few-shot extraction test.
3. Continue pre-training on a blend with about a quarter general replay data.
4. Re-measure. Domain perplexity should fall, and the general score should hold.
5. Apply SFT with prompts such as "Extract the fault code and action from this log line".
6. Add retrieval over the manuals for exact lookups, see hybrid search.
7. Monitor in use, and send reviewed failures back into the next data version.

The same pattern fits predictive maintenance and industry 4.0 settings, where text, sensor tables and logs appear side by side.

Language and locale adaptation

Another frequent use is extending a model to a language that was thin in its first training run. The work involves a large corpus in the new language, often a vocabulary check for how many pieces each word needs, and a replay share of the original language to avoid losing it. Evaluate in both languages, and test code-switching, where text mixes the two.

How long to train

There is no universal answer. Watch the held-out domain loss. When it flattens while general loss starts to rise, the run has reached the point of diminishing returns. A cleaner and more diverse corpus usually helps more than extra passes over the same data.

Checkpoint selection

Evaluate several checkpoints and compare them on the domain and general suites. Averaging the weights of the last few checkpoints often gives a more stable model than the final one alone. Keep the chosen checkpoint, its config and its evaluation report together.

Monitoring during the run

Track loss, gradient norm, learning rate and throughput. A sudden loss spike often points to a bad data shard or a numeric issue. Skipping the offending batch and resuming from the last good checkpoint is a common fix.

Common mistakes

• Using an instruction-tuned model as the start without a plan. Raw text training can erode instruction behavior. Start from a base model or plan to redo post-training.
• No replay data. The model forgets general skills.
• Learning rate too high. Spikes and forgetting appear fast.
• Dirty or duplicated corpus. The model memorizes junk and may leak private text.
• Evaluating only perplexity. Fluency is not accuracy.
• Contamination. Evaluation sets that appear in the corpus give false gains. Filter against them.
• Changing the tokenizer lightly. Adding tokens without enough training leaves poorly learned embeddings.
• Expecting instruction following. Continued pre-training does not teach it. Apply SFT.
• Ignoring licenses. Using data without clear rights creates legal risk.
• Skipping a pilot. Bugs in data loading found after a week of training are costly.
• For tables, only easy masks. Training with random masks and deploying on structured gaps overstates imputation quality.
• For tables, always showing headers. The model then fails when headers are missing or unreliable.

Frequently asked questions

What is continued pre-training?

Continued pre-training is further training of an already pre-trained model on a large unlabeled corpus using the original self-supervised objective, so the model adapts to a new domain, language or data format while keeping its general abilities.

How is continued pre-training different from SFT?

Continued pre-training uses large amounts of unlabeled text to teach vocabulary, facts and structure, and the model remains a text continuer. SFT uses a small set of labeled prompt and answer pairs to teach behavior and format. They are complementary, and a common order is continued pre-training first and SFT afterward.

How much data do I need?

Meaningful domain adaptation usually needs hundreds of millions to billions of tokens of clean text. A few million tokens can shift style slightly, but for that scale a parameter-efficient method or plain SFT is often more cost-effective. Quality and diversity matter as much as the raw count.

How do I stop the model from forgetting what it knew?

Mix replay data similar to the original training data into every batch, use a modest learning rate, avoid a full re-warm to the original peak, and track a general benchmark throughout. If general scores fall, increase the replay share or lower the rate.

Should I start from a base model or an instruction model?

A base model is usually the better start, because it has not been narrowed by post-training, and you can apply SFT and preference tuning afterward. Starting from an instruction model can work for light adaptation but may degrade its instruction following unless you repeat post-training.

Do I need a new tokenizer?

Usually not. Keeping the original tokenizer is simpler and safer. Extending the vocabulary can help when the domain or language is split into very many pieces, but new token embeddings must be learned, so you need enough data and a longer run.

Can continued pre-training add new facts reliably?

It adds knowledge to a degree, especially for facts that appear often in the corpus, but recall of rare facts is unreliable. If exact and current facts matter, combine it with retrieval so the model can look information up.

Does continued pre-training apply to tabular foundation models?

Yes. For a table-native model, it means resuming the self-supervised objective, such as predicting masked cells or held-out targets, on a new pool of tables. This can add sectors, harder missing-value patterns, tables without headers and wider schemas, while a replay set protects earlier skills.

How do I evaluate a continued pre-training run?

Use held-out domain text for perplexity, domain task benchmarks for accuracy, and a general regression suite for forgetting. Add manual review of generated samples. Compare each checkpoint against the base model, and choose the one with the best balance.

Is continued pre-training better than LoRA?

Not always. It changes the weights broadly and can absorb more from a large corpus, but it costs much more. LoRA on domain text is lighter and sufficient for modest shifts. Test both on a small pilot with your own evaluation set before committing.

Glossary

• Annealing. Decaying the learning rate at the end of training, often with selected data.
• Base model. A model trained only with the pre-training objective.
• Catastrophic forgetting. Loss of earlier abilities after new training.
• Contamination. Evaluation data leaking into training data.
• Corpus. The collection of data used for training.
• Domain-adaptive pre-training. Continued pre-training on a domain corpus.
• Mid-training. A stage between main pre-training and post-training using curated data.
• Perplexity. The exponential of the average loss, lower is better.
• Replay. Mixing earlier-style data into new training.
• Self-supervised learning. Learning from labels created from the data itself.
• Token budget. The planned number of tokens for a run.
• Tokenizer. The component that splits text into tokens.
• Vocabulary extension. Adding new tokens to a tokenizer.
• Warmup. A short rise in learning rate at the start of training.

Summary

Continued pre-training extends a pre-trained model on a large unlabeled corpus with the original self-supervised objective. It teaches the vocabulary, structure and style of a domain that SFT cannot supply, and it costs far less than training from scratch. Success depends on clean deduplicated data, a replay share to protect general ability, a moderate learning rate, and an evaluation plan that covers both the domain and general skills. Follow it with SFT and preference tuning to produce a usable assistant.

For models that learn from raw tables, the same idea means resuming masked-cell or held-out-target training on new pools of tables: more sectors, richer missing-value patterns, tables without usable headers, and wider schemas. This improves imputation, anomaly detection and sector classification, while replay and regression suites guard against forgetting. For related background, see /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Fine-tuning in my stack, under LLMs, retrieval, and evaluation. Others in the same group: