DPO

Overview

DPO, short for direct preference optimization, is a method that aligns a language model with human preferences using pairs of answers, one chosen and one rejected. It trains the model with a simple classification-style loss. It does not need a separate reward model and it does not need a reinforcement learning loop. For many teams it is the easiest way to turn a set of preference labels into a better-behaved model.

The key insight is mathematical. The standard RLHF objective has a known best solution, and that solution links the reward to the ratio between the tuned model and a reference model. DPO plugs that link into the preference loss, so the reward model disappears. This article explains the idea in plain terms, shows the loss with a worked example and code, compares DPO with IPO, KTO, ORPO and others, and explains how it relates to models that learn from raw tables.

What it is

DPO is an offline preference-optimization algorithm. "Offline" means it learns from a fixed data set of comparisons, not from fresh samples generated during training. Each training example has three parts: a prompt, a chosen answer and a rejected answer.

Imagine coaching a writer. You show two drafts of the same paragraph and say, "This one is better." You never give a numeric score and you never build a scoring machine. After many such comparisons, the writer's habits shift toward the style you prefer. DPO works the same way. It directly shifts the model's probabilities toward chosen answers and away from rejected ones.

What makes it different

• No reward model. The model itself plays the role of an implicit reward model.
• No sampling loop. Training is a standard supervised pass over stored pairs.
• A reference model. A frozen copy of the starting model keeps the tuned model from drifting too far.
• One main knob. The parameter beta controls how strongly the model is tied to the reference.

What problem it solves

RLHF with PPO works, but it needs a reward model, a critic, a reference model and a policy, plus careful tuning. DPO keeps the goal of RLHF, which is to maximize preference while staying close to a reference, and removes most of the machinery. It reduces memory use, shortens code and makes runs easier to repeat.

Typical position in a pipeline

1. Pre-train a base language model.
2. Run supervised fine-tuning on demonstrations so the model follows instructions.
3. Collect or generate preference pairs.
4. Run DPO starting from the SFT model, with the SFT model also serving as the reference.

Where it came from

DPO was introduced in a 2023 paper titled "Direct Preference Optimization: Your Language Model is Secretly a Reward Model". The authors showed that the constrained reward maximization problem used in RLHF can be solved in closed form. They then rewrote the preference loss in terms of the language model's own log probabilities.

The background includes several earlier ideas:

• The Bradley-Terry model from 1952, which describes how score differences translate into the probability that one item beats another.
• Reinforcement learning from human preferences work from 2017, which brought pairwise human judgments into deep learning.
• The language-model RLHF papers of 2020 and 2022, which made the three-stage pipeline of SFT, reward model and PPO standard.
• Earlier work on KL-regularized control, which gave the form of the optimal policy.

Why it spread quickly

DPO arrived when many open-model teams wanted to align models without large reinforcement learning infrastructure. Its loss could be implemented in a few lines, ran with ordinary training code and often matched RLHF on common benchmarks. Libraries for fine-tuning added it quickly, and it became a default option for preference tuning, often combined with adapters such as LoRA and QLoRA.

What followed

After DPO, researchers studied its weaknesses and proposed variations. A general theoretical analysis in 2023 introduced the family of objectives that includes IPO. Work in 2024 produced KTO, which uses single labeled answers, and ORPO, which merges SFT and preference optimization. Other variants adjust length handling, add margins, or run DPO in repeated online rounds.

Timeline of related ideas

How it works

Step 1: define the target

RLHF seeks a policy that maximizes expected reward while keeping a KL penalty to the reference policy. For this objective, the best policy has a neat form:

pi*(y | x) = (1 / Z(x)) * pi_ref(y | x) * exp( r(x, y) / beta )

Here x is the prompt, y is the answer, r is a reward and Z is a normalizing constant that makes probabilities sum to one. In words, the best policy takes the reference model and boosts answers in proportion to their exponentiated reward.

Step 2: invert the relationship

Rearranging that formula gives the reward in terms of the policies:

r(x, y) = beta * log( pi*(y | x) / pi_ref(y | x) ) + beta * log Z(x)

The reward is beta times the log ratio of the tuned and reference probabilities, plus a term that depends only on the prompt. Think of it as a weather forecaster's confidence: how much more likely does the tuned model consider this answer than the old model did?

Step 3: plug into the preference loss

The Bradley-Terry model says the probability that the chosen answer beats the rejected one is the logistic function of the reward difference. In that difference, the prompt-only term cancels. What remains is:

loss = - log sigmoid( beta * [ log(pi(y_w|x)/pi_ref(y_w|x)) - log(pi(y_l|x)/pi_ref(y_l|x)) ] )

Here y_w is the winning (chosen) answer and y_l is the losing (rejected) answer. This is a binary cross-entropy loss on an implicit reward margin.

Step 4: train with ordinary gradient descent

For each batch, you compute four log probabilities by running the answers through both models:

• the tuned model on the chosen answer
• the tuned model on the rejected answer
• the reference model on the chosen answer
• the reference model on the rejected answer

The reference values can be computed once and cached, since the reference never changes. The loss pushes up the chosen log ratio and pushes down the rejected log ratio.

Step 5: understand the gradient

The gradient has an intuitive shape. It increases the probability of the chosen answer and decreases the probability of the rejected one. Each example is weighted by how wrong the implicit reward currently is. If the model already ranks the pair correctly with a large margin, the weight is small. If it ranks them backwards, the weight is large. The model spends effort where it is most mistaken.

An analogy is a student reviewing flashcards. Cards already known well are skipped quickly. Cards answered wrongly come back more often.

Step 6: choose beta

Beta sets the strength of the tie to the reference.

• A small beta, such as 0.05, lets the model move far from the reference.
• A larger beta, such as 0.5, keeps it close.
• Common practice uses values around 0.1, then tunes.

Beta acts like the stiffness of a rubber band connecting the model to its starting point.

There is no universal best value. It depends on data quality, learning rate and model size, so treat the table as a starting map and verify on held-out data.

Step 7: evaluate

After training, check preference accuracy on held-out pairs, win rates against the SFT model by human or judge model comparison, answer length, and general capability tests. DPO can change length and style, so look beyond the average score.

Key concepts and terms

Log probability of a sequence

An answer is a sequence of tokens. Its log probability is the sum of the log probabilities of each token given the ones before. Longer answers tend to have lower total log probability, because there are more terms. DPO uses the sum, not the average, which is one reason length effects appear.

A worked example

Numeric walk-through

Suppose beta is 0.1. For a prompt, the log probabilities are:

The reward margin is beta times the difference of log ratios: 0.1 times (2.0 minus (-1.0)), which equals 0.3. The loss is the negative log of sigmoid(0.3). Sigmoid(0.3) is about 0.574, so the loss is about 0.555.

Now imagine a tuned model that has not changed from the reference. Both log ratios are zero, the margin is zero, sigmoid(0) is 0.5 and the loss is about 0.693. That is the starting loss for every pair. As training proceeds, the margin grows and the loss falls below 0.693. If the loss drops very close to zero quickly, the model may be overfitting.

Code sketch

The sketch below shows the loss function in plain PyTorch. Model loading and data utilities are placeholders.

import torch
import torch.nn.functional as F

def sequence_logprob(model, prompt_ids, answer_ids):
    """Sum of token log probs of the answer given the prompt."""
    logits = model(prompt_ids + answer_ids).logits
    logp = torch.log_softmax(logits[:, :-1], dim=-1)
    targets = (prompt_ids + answer_ids)[:, 1:]
    token_logp = logp.gather(-1, targets.unsqueeze(-1)).squeeze(-1)
    return token_logp[:, len(prompt_ids) - 1:].sum(dim=-1)

def dpo_loss(policy, reference, batch, beta=0.1):
    pi_w = sequence_logprob(policy, batch.prompt, batch.chosen)
    pi_l = sequence_logprob(policy, batch.prompt, batch.rejected)
    with torch.no_grad():
        ref_w = sequence_logprob(reference, batch.prompt, batch.chosen)
        ref_l = sequence_logprob(reference, batch.prompt, batch.rejected)
    margin = beta * ((pi_w - ref_w) - (pi_l - ref_l))
    loss = -F.logsigmoid(margin).mean()
    accuracy = (margin > 0).float().mean()
    return loss, accuracy

Data format

Preference data is usually stored as one JSON object per line:

{"prompt": "Explain what a primary key is.",
 "chosen": "A primary key uniquely identifies each row in a table...",
 "rejected": "It is a key. Use it."}

Typical training settings

method: dpo
base_model: example.internal/sft-checkpoint
reference_model: example.internal/sft-checkpoint
beta: 0.1
learning_rate: 5.0e-7
epochs: 1
batch_size: 64
max_length: 1024
adapter: lora        # optional, reduces memory
lora_rank: 16

The learning rate is small because the model is already well trained. One epoch is common, since repeated passes over the same pairs invite overfitting.

Reading the training logs

• Loss should fall from about 0.693 and settle.
• Preference accuracy should rise above 0.5 and often reaches 0.7 to 0.8 on training pairs.
• Chosen reward (beta times chosen log ratio) should not fall too much. If both chosen and rejected log probabilities fall, the model may be becoming worse at generating anything.
• Margin should grow gradually.
• Output length on held-out prompts should be sampled and checked.

A known quirk: both probabilities can fall

DPO only cares about the difference between chosen and rejected. It can lower the probability of both, as long as the rejected one drops more. In practice this can cause the model to put mass on odd, unseen answers. Remedies include a small SFT term on the chosen answers, lower learning rates, or alternative losses.

Variants and extensions

Researchers have proposed many tweaks to the basic loss. They are worth knowing, even if you start with plain DPO.

• Length-normalized DPO. Divides log probabilities by answer length so that long answers are not favored or punished just for having more tokens.
• DPO with a margin. Adds a fixed margin so the chosen answer must beat the rejected one by a set amount.
• DPO with an SFT term. Adds a small supervised loss on the chosen answer, which keeps its probability from falling.
• Label smoothing (conservative DPO). Treats labels as slightly noisy, which softens the loss on mislabeled pairs.
• Reference-free variants. Remove the reference model to save memory, at the cost of losing the anchor.
• Online or iterative DPO. Regenerate pairs from the current model every round, which brings back some of the exploration that RLHF has.
• Weighted DPO. Gives higher weight to pairs with high annotator agreement or large quality gaps.

A second worked example: building pairs from model samples

Suppose you want to improve a model that answers questions about data schemas. You have 5,000 prompts and no preference labels. Here is a practical way to build pairs.

1. Sample four answers per prompt from the SFT model at moderate temperature.
2. Score the four answers with a judge, which may be a larger model or a rule-based checker.
3. Take the highest-scoring answer as chosen and the lowest-scoring as rejected.
4. Drop pairs where the score gap is small, because those labels are mostly noise.
5. Spot check 100 pairs by hand.
6. Train with DPO for one epoch and compare with the SFT model.

This procedure uses on-policy samples, which tend to give more useful gradients than pairs drawn from unrelated models. The gap filter in step four is simple but effective.

Debugging a DPO run

Strengths and limitations

Strengths

• Simple. One stage, one loss, no reward model and no critic.
• Stable. Training is supervised, so it avoids many reinforcement learning instabilities.
• Cheap. It needs only the policy and a reference, and reference log probabilities can be cached.
• Works with adapters. Combined with LoRA or QLoRA, it runs on modest hardware. See also PEFT.
• Reproducible. Fixed data and a fixed loss make results easier to repeat.
• Competitive. On many public comparisons it matches or approaches RLHF with PPO.

Limitations

• Offline data limits. It cannot explore. If the preference pairs do not cover the model's current behavior, learning may be poor.
• Distribution shift. Pairs generated by another model may not reflect the tuned model's own outputs.
• Overfitting. With few pairs and many epochs, the loss can approach zero without real gains. This motivated IPO.
• Length bias. Summed log probabilities can reward or punish length.
• Needs paired data. Each prompt needs a chosen and a rejected answer. If you only have thumbs-up and thumbs-down labels, KTO may fit better.
• Reference dependence. The tuned model stays tied to the reference. A weak reference limits the outcome.
• No reusable reward. You cannot easily rank new samples or run best-of-n, unless you use the implicit reward.
• Noisy labels. Mislabeled pairs push the model in the wrong direction, with no reward-model smoothing.

How it compares with alternatives

DPO versus RLHF in practice

• Choose DPO when you already have a good pair data set, limited compute and a need for fast iteration.
• Choose RLHF when you need a reward model for ranking, want online exploration or have verifiable rewards.
• Many teams run DPO first, check gains, and only move to RL for the final polish.

DPO versus IPO

IPO replaces the logistic loss with a squared loss that aims for a fixed margin. This prevents the margin from growing without bound, which helps when preferences are nearly deterministic in the data. If your DPO runs overfit quickly, try IPO.

DPO versus KTO and ORPO

KTO accepts unpaired good and bad labels and is inspired by prospect theory. ORPO adds an odds-ratio term to the SFT loss and needs no reference model, which saves memory. Choose by the data you have and the memory you can spend.

Role in tabular foundation models

DPO is a language-model alignment method, so it is not part of how models that learn from raw tables are pre-trained. Tabular foundation models usually learn from a direct prediction objective on many tables, often synthetic, with masked cells or held-out rows. The objective is already precise, so there is no need for human preference pairs. Still, DPO matters in the tabular world wherever a language model touches tables.

Where DPO does not apply

Consider a model that takes a table with rows, columns of mixed numeric, categorical and date types, and missing entries, and predicts a target for new rows in-context. Quality is measured by accuracy, error or likelihood. TabPFN is an example of a model pre-trained on a large body of generated tables so that it can learn from a new table in a single forward pass. Boosted trees such as XGBoost and LightGBM minimize a fixed loss. None of these use preference pairs. For background on how such models are built and measured, see /model/research, /model/architecture and /model/benchmarks.

Where DPO can help

• Text-to-SQL and table question answering. Build pairs from correct and incorrect queries. A query that returns the right result is the chosen answer, and a query that errors or returns wrong rows is the rejected one. DPO is a natural fit because execution gives cheap, reliable labels.
• Explanations of predictions. Pairs can compare explanations that are accurate and cautious against those that overstate causation.
• Data-cleaning assistants. Experts compare two proposed fixes for missing or inconsistent values and pick the better one.
• Schema and semantic labeling. A language model proposes a meaning for a column that has no name, such as "customer age" or "transaction date". Accepted proposals become chosen answers and corrected ones become rejected answers.
• Anomaly triage. Analysts compare two write-ups of why a row looks unusual, and DPO tunes the writer toward the more useful style.

Conceptual parallels

• Implicit reward and anomaly scores. DPO's implicit reward is a log ratio between two models. Many anomaly detectors also compare a model against a baseline, such as how much less likely a row is under the fitted model than under a reference.
• Reference anchoring and continual learning. The tie to a reference model resembles methods in continual learning that keep updated weights near old ones to limit forgetting.
• Mixed signals. Tabular tasks often have exact metrics, so a hybrid of metric-based labels and human preference labels is realistic. DPO can consume both, since both reduce to chosen and rejected pairs.
• In-context learning contrast. A tabular in-context learner adapts to a new table at inference time without weight updates. DPO changes weights offline. They solve different problems: one adapts to a dataset, the other adjusts behavior.

Mapping tabular tasks to preference data

A concrete data-building recipe for table assistants

1. Collect a set of natural-language questions about sample tables with fake names such as orders and customers.
2. For each question, sample several SQL candidates from an SFT model.
3. Run each against a test database.
4. Label queries that give the right result as chosen and wrong ones as rejected.
5. Add a small number of human-ranked pairs for style questions, such as clarity of the final answer.
6. Train with DPO and compare execution accuracy before and after.

The strict labels come from execution, not from a reward model, which fits the DPO setting well.

Practical guidance and best practices

Data quality

• Use pairs where the difference between chosen and rejected is real and meaningful.
• Remove duplicates and near duplicates.
• Check a random sample by hand. Label noise hurts DPO directly.
• Balance prompt types so one domain does not dominate.
• Prefer pairs sampled from a model close to the one you tune. On-policy pairs usually work better than distant ones.
• Control length. If chosen answers are always longer, the model will learn to be verbose.

Hyperparameters

• Start with beta near 0.1 and test a few values.
• Use a low learning rate, often in the range of 1e-7 to 1e-6 for full fine-tuning.
• Use one or two epochs and watch held-out accuracy.
• Use a warmup and a cosine or linear decay schedule.
• With adapters, learning rates can be higher, but verify stability.

Pipeline tips

• Always run SFT first, and use the SFT model as the reference.
• Cache reference log probabilities to save compute.
• Consider adding a small SFT loss on chosen answers to prevent probabilities from collapsing.
• Try iterative DPO: generate new pairs from the current model, label them, and train again.
• Keep a held-out set of pairs to measure preference accuracy.

Evaluation

• Compare against the SFT model with pairwise human or judge model evaluation.
• Measure length changes and normalize them in comparisons.
• Test general capabilities to catch regressions.
• Test safety behavior with both harmful and benign prompts.

A short pre-flight checklist

• The SFT model works on the target format.
• The reference model equals the starting model, with identical tokenizer and prompt template.
• Prompts, chosen answers and rejected answers use the same chat template as serving.
• The held-out pair set is separate from training.
• Logs include loss, margin, accuracy, chosen and rejected log ratios, and length.
• A human review of sample outputs is planned for each checkpoint.
• A baseline exists, such as the SFT model or a rejection sampling run.
• Checkpoints are saved so that you can roll back to an earlier, less tuned version.

When to move on from DPO

If gains stall, consider these next steps.

• Improve the data first. More careful labels usually beat more epochs.
• Try IPO if overfitting is the issue.
• Try KTO if you can gather many single-answer labels cheaply.
• Try ORPO if memory is the constraint.
• Move to RLHF if you need online exploration or a reusable reward model.

Common mistakes

• Skipping SFT. Starting from a raw pre-trained model leaves the reference weak and the format unlearned.
• Mismatched reference. Using a different reference than the starting model breaks the math, since the log ratio would start nonzero.
• Too high a learning rate. This quickly collapses the model or drives probabilities to extremes.
• Too many epochs. Pairs get memorized, the loss goes near zero and quality falls.
• Ignoring length. Verbose outputs may look better to judges but are not truly better.
• Noisy preference labels. Mixing careless labels with careful ones wastes training signal.
• Using off-distribution pairs only. Pairs from a very different model give weak gradients on the tuned model's own behavior.
• Not checking absolute log probabilities. Both chosen and rejected can fall together.
• Treating the implicit reward as a calibrated score. It is only meaningful relative to the reference.
• Forgetting tokenization and padding masks. Including prompt tokens or padding in the log probability sum adds noise.

Frequently asked questions

What does DPO stand for?

DPO stands for direct preference optimization. It is a training method that adjusts a language model so that it assigns higher probability to answers people preferred and lower probability to answers they rejected, using a simple loss on pairs and without any separate reward model.

Does DPO replace RLHF?

For many practical cases it can replace the full RLHF pipeline, because it uses the same kind of preference data and often reaches similar quality with far less complexity. RLHF still has advantages when you need online exploration, a reusable reward model or rewards from automatic verifiers, so the two methods coexist.

Why does DPO need a reference model?

The reference model anchors the tuned model. The loss is written in terms of how much more likely the tuned model finds an answer than the reference does, which keeps the model from drifting into strange behavior and mirrors the KL penalty in RLHF. The reference is frozen, so its values can be computed once and reused.

What is beta in DPO?

Beta controls how strongly the tuned model is tied to the reference. Small values allow larger changes and sharper preference learning, while larger values keep behavior close to the starting point. A value around 0.1 is a common starting point, and it is usually tuned on held-out data.

How many preference pairs do I need?

There is no fixed number. Useful gains have been reported with tens of thousands of pairs, and smaller sets can help on narrow tasks, but quality, diversity and closeness to the model's own outputs usually matter more than raw count. Start with a modest set, measure held-out accuracy and add data where the model is weakest.

Why does DPO sometimes make answers longer?

The loss uses summed log probabilities and the preference data often contains longer chosen answers, so the model can learn that length correlates with winning. You can reduce this by balancing lengths in the data, adding length normalization or a length penalty, or using variants designed to handle length.

Can I use DPO with LoRA?

Yes. Combining DPO with low-rank adapters is common because it reduces memory and lets a single GPU handle models that would otherwise be too large. With adapters, the reference model can even be the same network with adapters switched off, which saves further memory.

What if I only have thumbs-up and thumbs-down labels?

DPO needs paired answers to the same prompt. If your labels are single answers marked good or bad, a method such as KTO is designed for that format. You can also build pairs by matching a good and a bad answer to the same prompt when your data allows it.

Is DPO used for tabular foundation models?

Not for their core pre-training, which relies on a direct prediction objective on tables. DPO may help language assistants around tabular systems, such as text-to-SQL, explanation writers and data-cleaning helpers, where chosen and rejected answers are easy to define from execution results or expert review.

How do I know DPO is working?

Check that held-out preference accuracy rises, that win rates against the starting model improve under human or judge evaluation, and that answer length and general capabilities stay reasonable. A training loss that falls to nearly zero very fast is a warning of overfitting rather than a sign of success.

Glossary

• Alignment: shaping model behavior to match human intent.
• Bradley-Terry model: a model of pairwise comparisons based on latent scores.
• Chosen answer: the preferred response in a pair.
• Rejected answer: the less preferred response in a pair.
• Beta: the parameter setting how closely the model follows the reference.
• Implicit reward: beta times the log ratio between tuned and reference probabilities.
• KL divergence: a measure of how different two distributions are.
• Log probability: the logarithm of the probability assigned to a sequence.
• Offline training: learning from a stored data set rather than fresh samples.
• On-policy data: samples generated by the model being trained.
• Preference pair: a prompt with a chosen and a rejected answer.
• Reference model: a frozen copy of the starting model.
• Reward margin: the difference in implicit rewards between chosen and rejected.
• SFT: supervised fine-tuning on demonstration data.

Summary

DPO turns preference alignment into a straightforward supervised problem. It uses pairs of chosen and rejected answers, a frozen reference model and a logistic loss on log ratios, so no reward model or reinforcement learning loop is needed. It is simple, stable and cheap, and it works well with adapters, but it depends on good paired data, can overfit and can show length bias. Related methods such as IPO, KTO and ORPO address some of those issues. For tabular foundation models, DPO is not the pre-training method; it is a tool for the language layer around them, including text-to-SQL, explanations and data-cleaning assistants, while the numeric core is judged on metrics as described in /model/benchmarks.

Related

It belongs to Alignment in my stack, under LLMs, retrieval, and evaluation. Others in the same group: