# What is DPO? Definition and meaning

Optimizes a model directly on preferred versus rejected answers, without training a separate reward model or running reinforcement learning.

## Definition

**DPO**: Optimizes a model directly on preferred versus rejected answers, without training a separate reward model or running reinforcement learning.

In short: Direct preference optimization.

## When it is used

A simpler, more stable alternative to RLHF for preference tuning.

## Where it fits

DPO belongs to **Fine-tuning** in the [Generative AI and LLMs](/topics/generative-ai-llms) topic. These are the other terms in the same group:

- **[Fine-tuning](/glossary/fine-tuning)**: Further training of a pretrained model on new data so it performs a specific task, domain or style better.
- **[Full fine-tuning](/glossary/full-fine-tuning)**: Continues training with all model parameters trainable, rather than a small add-on.
- **[SFT](/glossary/sft)**: Trains a model on curated prompts and ideal responses so it follows instructions in the desired style.
- **[LoRA](/glossary/lora)**: Freezes the model and learns small low-rank update matrices for selected layers, so only a small fraction of parameters change.
- **[QLoRA](/glossary/qlora)**: Combines 4-bit quantization of the frozen base with LoRA adapters, which cuts memory enough to fine-tune large models on one GPU.
- **[PEFT](/glossary/peft)**: A library that implements LoRA, prefix tuning, prompt tuning and related methods on top of Hugging Face models.
- **[RLHF](/glossary/rlhf)**: Trains a reward model from human preference comparisons and then optimizes the language model against it with reinforcement learning.

## Learn more

The full guide to **[DPO](/tools/dpo)** explains how it works, when to use it and how it compares with the alternatives, with examples. The [glossary](/glossary) lists every term on the site from A to Z.

Previous term: [Double DQN](/glossary/double-dqn) · Next term: [DQN](/glossary/dqn)
