# Post-training: supervised fine-tuning, then preference optimisation with DPO

A pre-trained model predicts text; post-training turns it into an assistant. Supervised fine-tuning imitates good answers, then preference optimisation learns from comparisons. This simulation runs both on a small policy with exact gradients.

<div data-sim="tfspost"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

Each row is a candidate answer with a hidden quality score. The bars show its probability under the base model, after SFT and after DPO. The side panel reports the expected quality across three prompts, the KL divergence from the SFT model and the average answer length.

## Things to try

- Set SFT steps to zero: DPO has to do all the work from the base model.
- Raise β: the policy stays closer to the SFT model.
- Switch the labellers to 'prefer the longer answer': quality falls and length rises.
- Add 30% random labels: DPO still helps, but less.

## How it works in real systems

DPO (Rafailov et al., 2023) optimises the preference objective directly, without training a separate reward model as RLHF with PPO does. Current pipelines combine SFT, preference optimisation and reinforcement learning with verifiable rewards for maths and code, and audit the labels because the policy learns whatever they reward.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Supervised fine-tuning with TRL

```python
# pip install trl datasets
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
ds = load_dataset("trl-lib/Capybara", split="train")
trainer = SFTTrainer(model="Qwen/Qwen2.5-0.5B", train_dataset=ds,
                     args=SFTConfig(output_dir="sft", per_device_train_batch_size=4, num_train_epochs=1, learning_rate=2e-5))
trainer.train()
```

### Step 2 · Preference optimisation with DPO

```python
from trl import DPOTrainer, DPOConfig
pref = load_dataset("trl-lib/ultrafeedback_binarized", split="train")    # prompt, chosen, rejected
trainer = DPOTrainer(model="sft", train_dataset=pref,
                     args=DPOConfig(output_dir="dpo", beta=0.1, per_device_train_batch_size=2, learning_rate=5e-7))
trainer.train()
```

### Step 3 · The DPO loss in five lines

```python
import torch.nn.functional as F
def dpo_loss(pc, pr, rc, rr, beta=0.1):
    # pc/pr: log-probs of chosen/rejected under the policy, rc/rr: under the frozen reference
    return -F.logsigmoid(beta * ((pc - rc) - (pr - rr))).mean()
```

<div class="cl-end"></div>

## More simulations

[← Run, spikes and checkpoints](/ai-simulations/train-run-monitor) · [All AI Simulations](/ai-simulations) · [Evaluation →](/ai-simulations/train-evaluation)
