Özer Cem KELAHMETKELAHMET
Topics · Generative AI and LLMs

How LLMs are trained: pre-training, fine-tuning, preference optimisation and RL

A chat model is made in stages. Pre-training teaches language and knowledge, fine-tuning teaches the assistant format, and preference and reinforcement learning teach what good answers look like.

Overview

Training a large language model is a pipeline. Each stage uses different data and a different objective, and later stages are much cheaper than the first but shape the behaviour people see.

The stages

Stage Data Objective Read more Try it
Data preparation Trillions of tokens of web, code, books, papers Clean, deduplicate, mix Deduplication Simulation
Tokenizer A sample of the corpus Learn a vocabulary Tokenizer training Simulation
Pre-training The full corpus Predict the next token Continued pre-training Simulation
Supervised fine-tuning Example conversations Imitate good answers SFT Simulation
Preference optimisation Pairs of better and worse answers Prefer the better one RLHF, DPO Simulation
Reinforcement learning Problems with checkable answers Maximise verified reward PPO, RLAIF
Evaluation Held-out benchmarks and human tests Measure, compare, decide lm-evaluation-harness Simulation

Preference optimisation

RLHF trains a reward model on human comparisons and then optimises the policy with PPO. DPO skips the separate reward model and optimises the comparisons directly, which is simpler and stable; variants include IPO, KTO, ORPO and SimPO. Constitutional AI uses written principles and model feedback in place of some human labels.

Reinforcement learning with verifiable rewards

For maths and code, an answer can be checked automatically, so the model can be rewarded for correct solutions without human labels. This is how reasoning models learn to think longer and check their work.

Adapting an existing model

Most teams do not pre-train. They fine-tune an open model with LoRA or QLoRA using libraries such as TRL, Unsloth, Axolotl or LLaMA-Factory. The LoRA simulation shows why low-rank adapters are enough for many tasks.

Frequently asked questions

How much does each stage cost?

Pre-training dominates, often more than 90% of compute. Fine-tuning and preference optimisation can run on a few GPUs for small and medium models.

Is DPO better than RLHF?

DPO is simpler and often matches RLHF on preference data; RL with verifiable rewards adds abilities that preference data alone does not teach.

Summary

Data, tokenizer, pre-training, fine-tuning, preferences and reinforcement learning: each stage shapes the model, and the nine-step training track lets you run every one.

Back to the topic

This page is part of Generative AI and LLMs, which maps every concept, tool, simulation and article on the subject.

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open