# How LLMs are trained: pre-training, fine-tuning, preference optimisation and RL

A chat model is made in stages. Pre-training teaches language and knowledge, fine-tuning teaches the assistant format, and preference and reinforcement learning teach what good answers look like.

## Overview

Training a large language model is a pipeline. Each stage uses different data and a different objective, and later stages are much cheaper than the first but shape the behaviour people see.

## The stages

| Stage | Data | Objective | Read more | Try it |
|---|---|---|---|---|
| Data preparation | Trillions of tokens of web, code, books, papers | Clean, deduplicate, mix | [Deduplication](/tools/deduplication) | [Simulation](/ai-simulations/train-data-pipeline) |
| Tokenizer | A sample of the corpus | Learn a vocabulary | [Tokenizer training](/tools/tokenizer-training) | [Simulation](/ai-simulations/train-tokenizer-vocabulary) |
| Pre-training | The full corpus | Predict the next token | [Continued pre-training](/tools/continued-pre-training) | [Simulation](/ai-simulations/train-run-monitor) |
| Supervised fine-tuning | Example conversations | Imitate good answers | [SFT](/tools/sft) | [Simulation](/ai-simulations/train-post-training-sft-dpo) |
| Preference optimisation | Pairs of better and worse answers | Prefer the better one | [RLHF](/tools/rlhf), [DPO](/tools/dpo) | [Simulation](/ai-simulations/train-post-training-sft-dpo) |
| Reinforcement learning | Problems with checkable answers | Maximise verified reward | [PPO](/tools/ppo), [RLAIF](/tools/rlaif) | |
| Evaluation | Held-out benchmarks and human tests | Measure, compare, decide | [lm-evaluation-harness](/tools/lm-evaluation-harness) | [Simulation](/ai-simulations/train-evaluation) |

## Preference optimisation

RLHF trains a reward model on human comparisons and then optimises the policy with PPO. DPO skips the separate reward model and optimises the comparisons directly, which is simpler and stable; variants include [IPO](/tools/ipo), [KTO](/tools/kto), [ORPO](/tools/orpo) and [SimPO](/tools/simpo). [Constitutional AI](/tools/constitutional-ai) uses written principles and model feedback in place of some human labels.

## Reinforcement learning with verifiable rewards

For maths and code, an answer can be checked automatically, so the model can be rewarded for correct solutions without human labels. This is how reasoning models learn to think longer and check their work.

## Adapting an existing model

Most teams do not pre-train. They fine-tune an open model with [LoRA](/tools/lora) or [QLoRA](/tools/qlora) using libraries such as [TRL](/tools/trl), [Unsloth](/tools/unsloth), [Axolotl](/tools/axolotl) or [LLaMA-Factory](/tools/llama-factory). The [LoRA simulation](/ai-simulations/lora) shows why low-rank adapters are enough for many tasks.

## Frequently asked questions

### How much does each stage cost?

Pre-training dominates, often more than 90% of compute. Fine-tuning and preference optimisation can run on a few GPUs for small and medium models.

### Is DPO better than RLHF?

DPO is simpler and often matches RLHF on preference data; RL with verifiable rewards adds abilities that preference data alone does not teach.

## Summary

Data, tokenizer, pre-training, fine-tuning, preferences and reinforcement learning: each stage shapes the model, and the nine-step [training track](/ai-simulations/train-data-pipeline) lets you run every one.

## Back to the topic

This page is part of **[Generative AI and LLMs](/topics/generative-ai-llms)**, which maps every concept, tool, simulation and article on the subject.
