ORPO

Overview

ORPO, short for odds-ratio preference optimization, is a method that teaches a language model to follow instructions and to prefer good answers over bad ones in a single training stage. It adds a small penalty term to the ordinary supervised fine-tuning loss. That term uses the odds ratio between a chosen answer and a rejected answer. Because it needs no frozen reference model and no separate preference stage, it is cheaper and simpler than the usual pipeline of supervised fine-tuning followed by DPO or RLHF.

The core idea is that supervised fine-tuning alone has a side effect: it raises the probability of the chosen answer, but it also tends to raise the probability of similar rejected answers. ORPO adds a gentle push that separates the two. This article explains the idea with everyday analogies, gives the loss, a worked example and code, compares ORPO with IPO, KTO, DPO and RLHF, and explains how it relates to models that learn from raw tables.

What it is

ORPO is a monolithic alignment method. "Monolithic" means one stage does the work that normally takes two. The model starts from a pre-trained base, not from a supervised fine-tuned checkpoint, and is trained on preference pairs with a combined loss.

Each training example has a prompt, a chosen answer and a rejected answer. The loss has two parts:

• An SFT term that raises the likelihood of the chosen answer, just like normal instruction tuning.
• An odds-ratio term that lowers the relative odds of the rejected answer compared with the chosen one.

An analogy

Imagine teaching a child to write polite emails. The first lesson shows good examples to copy, which is supervised fine-tuning. A second lesson shows a rude email next to a polite one and explains why the rude one is worse, which is preference tuning. Many courses teach these in two separate classes. ORPO teaches both in one class: while copying the polite email, the child is also gently told to avoid the rude one.

What problem it solves

The standard recipe has three costs.

1. Two training stages. SFT first, then preference tuning, each with its own hyperparameters and runs.
2. A reference model. DPO, IPO and KTO keep a frozen copy of the starting model in memory or cache its log probabilities.
3. Complexity. RLHF adds a reward model and reinforcement learning on top.

ORPO removes the second stage and the reference model, which lowers memory use and shortens the path from a base model to an aligned one.

What it keeps

• Pairs of chosen and rejected answers as data.
• A logistic-style preference signal.
• Compatibility with adapters such as LoRA and QLoRA.

What it drops

• The frozen reference model.
• The separate SFT stage before preference tuning.
• Any reward model or reinforcement learning loop.

Where it came from

ORPO was introduced in 2024 in a paper titled "ORPO: Monolithic Preference Optimization without Reference Model". The authors started from an observation about supervised fine-tuning. When a model is fine-tuned only on chosen answers, the probability of rejected answers in the same style also rises, because the two answers are often similar. That means SFT alone does not teach the model what to avoid.

The authors then proposed adding a light penalty based on the odds ratio. The odds ratio had been used in statistics for decades to compare the likelihood of two outcomes, and it gives a gentler signal than the probability ratio used by some other methods. They argued that this gentler signal works well alongside SFT, because a stronger penalty might disrupt learning of the basic task.

Background ideas

• Supervised fine-tuning. Training on demonstrations, as in SFT.
• Bradley-Terry model, 1952. Pairwise comparisons from latent scores.
• Odds and odds ratio. Classical statistical quantities used in medicine and social science.
• RLHF and DPO. The 2022 and 2023 developments that made preference tuning standard.

Timeline

Why it caught attention

Memory is a real constraint for many teams. Methods that need a reference model use roughly twice the memory for model weights, unless adapters or caching reduce it. ORPO appealed to people who wanted to align a model on a single machine, in a single run, with a simple objective.

How it works

Step 1: understand odds

If a model assigns probability P to an answer, the odds of that answer are P divided by (1 minus P). Odds measure how many times more likely the answer is than not.

Example: a probability of 0.5 gives odds of 1. A probability of 0.8 gives odds of 4. A probability of 0.2 gives odds of 0.25.

An everyday analogy is horse racing. Bookmakers speak in odds, not percentages. "Four to one" tells you something similar to "80 percent", but the odds scale stretches the high end, so differences near certainty become easier to see.

Step 2: use a length-normalized probability

Probabilities of long answers are tiny, since they multiply many token probabilities. ORPO uses the geometric mean per token, which is the exponential of the average token log probability. This puts short and long answers on a comparable scale:

P(y | x) = exp( (1 / |y|) * sum over tokens of log p(token | previous tokens, x) )

Step 3: compute the odds ratio

For a chosen answer y_w and a rejected answer y_l:

odds(y) = P(y | x) / (1 - P(y | x))
OR = odds(y_w) / odds(y_l)

If OR is above one, the model favors the chosen answer. The larger it is, the stronger the favor.

Step 4: build the odds-ratio loss

The loss turns the log of the odds ratio into a logistic preference loss:

L_OR = - log sigmoid( log OR )

When the model strongly prefers the chosen answer, log OR is large and positive, sigmoid is near one and the loss is small. When it prefers the rejected answer, the loss is large.

Step 5: add the SFT loss

The total loss combines the two terms:

L = L_SFT(y_w) + lambda * L_OR

L_SFT is the usual negative log likelihood of the chosen answer. Lambda is a weight that sets how strong the preference term is. A small lambda keeps the focus on learning the task. A larger lambda pushes harder on the contrast.

Step 6: train from the base model

Since the SFT term is part of the loss, you can start from a pre-trained base model. No earlier SFT stage and no reference model is required. A single pass over the preference data both teaches the format and shapes preferences.

Step 7: evaluate

Check that the model follows instructions, that held-out pairs are ranked correctly, and that answer quality improves under human or judge model comparison. Check for regressions in general capabilities.

A mental picture

chosen answer:   pull probability UP    (SFT term)
rejected answer: push relative odds DOWN (odds-ratio term, light touch)
no reference model, no second stage

Key concepts and terms

Probability ratio versus odds ratio

Other methods, such as DPO, compare probability ratios between the tuned and the reference model. ORPO compares odds of two answers under the same model. The odds ratio changes more gently than the probability ratio when probabilities move, so it is less likely to overwhelm the SFT signal. Without the reference model there is no anchor, so the SFT term itself plays the stabilizing role.

A worked example

Numeric walk-through

Suppose that for one prompt the model gives these length-normalized probabilities:

The odds ratio is 0.667 divided by 0.429, about 1.556. The log of that is about 0.442. Sigmoid(0.442) is about 0.609, so the odds-ratio loss is the negative log of 0.609, about 0.496.

Now suppose training raises the chosen probability to 0.55 and lowers the rejected one to 0.20:

The odds ratio is about 4.89, its log is about 1.587, sigmoid is about 0.830 and the loss is about 0.186. The preference term has fallen a lot, which is what training is meant to do.

For the SFT part, if the chosen answer's average token log probability is minus 0.8, the SFT loss is 0.8. With lambda equal to 0.1, the total loss is 0.8 plus 0.1 times 0.496, about 0.85 at the start, and falls as the chosen probability rises.

Code sketch

The sketch below shows the loss in plain PyTorch. Helper functions are placeholders.

import torch
import torch.nn.functional as F

def mean_token_logprob(model, prompt_ids, answer_ids):
    """Average per-token log probability of the answer given the prompt."""
    logp = token_logprobs(model, prompt_ids, answer_ids)   # [B, T]
    mask = answer_mask(answer_ids)                          # [B, T]
    return (logp * mask).sum(-1) / mask.sum(-1)

def orpo_loss(model, batch, lam=0.1):
    lp_w = mean_token_logprob(model, batch.prompt, batch.chosen)
    lp_l = mean_token_logprob(model, batch.prompt, batch.rejected)

    # log odds = log P - log(1 - P), computed stably
    def log_odds(lp):
        return lp - torch.log1p(-torch.exp(lp).clamp(max=1 - 1e-6))

    log_or = log_odds(lp_w) - log_odds(lp_l)
    l_or = -F.logsigmoid(log_or).mean()
    l_sft = -lp_w.mean()
    return l_sft + lam * l_or, (log_or > 0).float().mean()

Note that there is only one model in the code. No reference model is loaded.

Typical configuration

method: orpo
base_model: example.internal/pretrained-base   # no SFT checkpoint needed
lambda: 0.1
learning_rate: 8.0e-6
epochs: 2
batch_size: 32
max_length: 1024
adapter: lora          # optional
lora_rank: 16

Learning rates for ORPO are usually higher than for DPO, because the model is learning the task and not only adjusting preferences.

Data format

Data looks the same as DPO data:

{"prompt": "Describe what a foreign key does.",
 "chosen": "A foreign key links a column in one table to the primary key of another...",
 "rejected": "It is some kind of link."}

Reading the logs

• SFT loss should fall as in ordinary instruction tuning.
• Odds-ratio loss should fall from about 0.693 toward lower values.
• Accuracy, the share of pairs with log OR above zero, should rise above 0.5.
• Chosen and rejected log probabilities should separate over time.
• Answer length on held-out prompts should be sampled.

Strengths and limitations

Strengths

• Single stage. One run replaces SFT followed by preference tuning.
• No reference model. Less memory, no cached reference log probabilities and simpler code.
• Simple loss. An SFT term plus one extra term with one weight.
• Works from a base model. No need for an SFT checkpoint.
• Adapter friendly. Combines well with LoRA, QLoRA and other PEFT methods.
• Reported to be competitive. In its original evaluation, it matched or beat several alternatives at various model sizes.

Limitations

• No anchor. Without a reference model, nothing explicitly keeps the model near a known good state, other than the SFT term and the small lambda.
• Needs paired data. Each prompt needs a chosen and a rejected answer.
• Data quality dependence. The chosen answers serve as SFT targets, so they must be high quality, not merely better than the rejected ones.
• Hyperparameter sensitivity. Lambda and the learning rate interact.
• Less studied. Fewer large-scale reports than for DPO or RLHF.
• Offline. It cannot explore beyond the stored data.
• Length handling. Length normalization helps, but length and style correlations in data can still leak in.
• One-shot risk. Because stages are merged, a failure is harder to isolate between the SFT part and the preference part.

How it compares with alternatives

ORPO versus DPO

• ORPO folds SFT into the same run and needs no reference model.
• DPO usually starts from a well-trained SFT model and benefits from the reference anchor.
• If your SFT checkpoint is already strong and you want to refine preferences carefully, DPO or IPO is a safe default.
• If you start from a base model with limited memory, ORPO is attractive.

ORPO versus KTO

• KTO works with unpaired binary labels. ORPO needs pairs.
• KTO needs a reference model. ORPO does not.
• If your feedback is thumbs-up and thumbs-down, choose KTO. If you have curated pairs, choose ORPO or DPO.

ORPO versus RLHF

• RLHF supports online exploration and reusable reward models. ORPO is offline.
• ORPO is far simpler and cheaper.
• Teams with verifiers and large budgets may still prefer RLHF-style training.

Role in tabular foundation models

ORPO is a language-model alignment method. It is not part of the pre-training of models that learn directly from raw tables. Those models, including in-context tabular learners trained on large numbers of synthetic or real tables, use a prediction loss on masked or held-out cells. The objective is precise, so no human preference pairs are needed. For background on how such models are built, see /model/research and /model/architecture.

Where ORPO does not apply

A tabular model reads rows and columns of mixed numeric, categorical and date types, copes with missing values and often works without column names. It predicts targets for new rows from labeled context rows. Quality is measured by accuracy, error and calibration. TabPFN is an example of a model trained this way, and boosted trees such as XGBoost, LightGBM and CatBoost optimize fixed losses. None uses ORPO.

Where ORPO fits

ORPO is most useful where a small team builds a language assistant for table work and wants a quick, cheap way to align it.

• Text-to-SQL assistants. Train on prompts with a correct query as chosen and a wrong query as rejected. The SFT term teaches SQL style, and the odds-ratio term teaches what to avoid, all in one run.
• Column semantic or sector labelers. For columns without names, a language model proposes a meaning. Corrected labels give chosen and rejected pairs.
• Data-cleaning explainers. Experts supply a better explanation and a weaker one for the same issue.
• Anomaly report writers. Analysts provide a preferred write-up and a rejected one.
• Schema documentation helpers. A good description is chosen and a vague one is rejected.

Why single-stage tuning suits small table-focused teams

Table assistants are often built by small teams with limited hardware. Skipping the reference model and the second stage reduces memory and the number of experiments. Because the SFT term does the main teaching, ORPO also works when you have no separate SFT data set; the chosen answers serve both purposes.

Conceptual parallels

• Combined objectives. Tabular foundation models also combine objectives. For example, a pre-training loss on masked cells can be paired with an auxiliary loss. ORPO combines a likelihood loss with a contrast loss in the same spirit.
• Contrast and anomaly detection. The odds-ratio term pushes a model to separate good and bad answers. Anomaly detection separates normal and unusual rows. Both rely on contrast.
• No reference and continual learning. Dropping the reference model removes an anchor against forgetting. Continual-learning methods use replay or regularization for that reason, and similar care is wise when tuning with ORPO on a narrow data set.
• Length normalization and mixed types. ORPO normalizes by length so that short and long answers are comparable. Tabular models normalize numeric columns of different scales for the same reason.
• In-context learning contrast. A tabular in-context model adapts to a table without weight updates. ORPO updates weights offline and shapes general behavior.

Mapping table tasks to ORPO

For how the numeric side is evaluated, see /model/benchmarks.

Practical guidance and best practices

Data

• Make sure chosen answers are good in absolute terms, not only better than the rejected ones, since they are also SFT targets.
• Make rejected answers plausible, not obviously silly, so the contrast teaches something.
• Remove duplicates and near duplicates.
• Balance prompt types and lengths.
• Keep a held-out validation set of pairs and prompts.
• Use the chat template that serving will use.

Hyperparameters

• Start with lambda around 0.1 and try a few values on a small grid.
• Use a learning rate suited to SFT, often higher than for DPO, and a warmup.
• Train for a small number of epochs and watch held-out results.
• With adapters, verify that rank and alpha are adequate for learning the task.

Process

• Compare against an SFT-only baseline on the chosen answers to check that the odds-ratio term adds value.
• Compare against a two-stage SFT plus DPO run if resources allow.
• Save regular checkpoints and sample outputs from each.
• Consider mixing a small amount of general instruction data to protect general abilities.
• Inspect refusal behavior in both directions.

Evaluation

• Pairwise human or judge model comparison for quality.
• Task metrics, such as execution accuracy for text-to-SQL.
• Capability benchmarks to catch regressions.
• Safety tests with harmful and benign prompts.
• Length checks to detect verbosity drift.

A short pre-flight checklist

• The base model and tokenizer are what you intend to use in production.
• Chosen answers pass a quality review.
• Prompt formatting in the data matches serving.
• Lambda and learning rate are logged with every run.
• Held-out data is not used for training.
• Baselines exist for comparison.
• Checkpoints are stored for rollback.

A cooking-school analogy

Imagine training a cook in two stages. First the cook copies recipes (supervised fine-tuning). Then a tutor tastes two dishes and says which one is better (preference tuning). ORPO is like a school where the cook practices copying a good dish while also being nudged away from the bad version of the same dish, at the same time, with one teacher. There is no need for a second, frozen copy of the cook to compare against, which saves space in the kitchen.

Memory view

Practical tips from experience with single-stage methods

• Start from a base model with a clean chat template, because the SFT part of the loss learns the format.
• Use data where the chosen answer is clearly better than the rejected one. Noisy pairs hurt more in single-stage training because they also feed the SFT term.
• Keep the odds-ratio weight modest at first and raise it only if the model fails to separate the pair margins.
• Watch the log-probability of the chosen answer. If it falls steadily, the preference term is overpowering the SFT term.
• Compare against a plain SFT baseline on the same data, so you can see what the preference part adds.

Related methods on this site

See DPO, IPO, KTO, RLHF, RLAIF, TRL, Axolotl and LoRA for neighbouring techniques and libraries.

When to prefer a two-stage pipeline

A single stage is attractive for cost, but a separate SFT stage still makes sense when you have a large instruction set that is not paired, when you want to inspect the SFT model before alignment, or when you need to reuse the same SFT checkpoint for several alignment variants. The choice is practical, not ideological.

Common mistakes

• Using weak chosen answers. Chosen answers become SFT targets, so mediocre ones teach mediocre behavior.
• Setting lambda too high. A strong contrast can disrupt learning the basic task.
• Setting lambda too low. The method degenerates into plain SFT.
• Assuming a reference model is needed. Loading one wastes memory and may confuse the loss.
• Mixing summed and normalized log probabilities. ORPO uses length-normalized likelihood, and changing this alters the meaning of lambda.
• Numerical issues near probability one. The odds formula divides by one minus P, so clamp values to avoid infinities.
• Ignoring data balance. If chosen answers are always longer, the model may learn length as a signal.
• Training too long on a small set. The model can memorize pairs.
• Skipping evaluation on general tasks. Narrow training can cause forgetting.
• Comparing against a poorly tuned baseline. Fair comparisons need similar effort on each method.

Guidance for different situations

Frequently asked questions

What does ORPO stand for?

ORPO stands for odds-ratio preference optimization. It is a method that trains a language model on pairs of chosen and rejected answers in a single stage, combining an ordinary supervised fine-tuning loss with a penalty based on the odds ratio between the two answers.

Why does ORPO not need a reference model?

The supervised fine-tuning term already keeps the model anchored to good behavior, and the odds-ratio term only compares the chosen and rejected answers under the same model. Since no log ratio to a frozen model is needed, the reference can be dropped, which saves memory and removes a component from the pipeline.

What is an odds ratio?

Odds are the probability of an event divided by one minus that probability, and an odds ratio compares the odds of two events. In ORPO, it compares how likely the model finds the chosen answer against the rejected one, and a ratio above one means the model favors the chosen answer.

How is ORPO different from DPO?

DPO runs after supervised fine-tuning and uses a frozen reference model to compute log ratios between tuned and reference probabilities. ORPO merges supervised fine-tuning and preference learning into one stage, uses odds within the same model and requires no reference model.

Can I start ORPO from a base model?

Yes, that is one of its main selling points. Because the loss contains a supervised term on the chosen answers, a single run can teach the format and the preferences together, so a separate SFT stage is not required, although a strong SFT checkpoint can still help.

What does the lambda parameter do?

Lambda sets the weight of the odds-ratio term relative to the supervised term. Small values keep the focus on learning the task from chosen answers, while larger values push harder on separating chosen from rejected answers. It is usually tuned on held-out data, with values around 0.1 as a common start.

Does ORPO work with LoRA?

Yes. Adapters work well with ORPO and reduce memory further. Because the method has no reference model, only one network needs to be kept in memory, which makes single-GPU alignment more practical.

Is ORPO better than DPO?

It depends on the setting. ORPO is more convenient and cheaper, and reports show it can match or exceed other methods in some cases, but DPO has a longer track record and can be safer when you have a strong SFT model and high-quality pairs. The best choice is to compare both on your own validation data.

Is ORPO used for tabular foundation models?

Not for their core pre-training, which relies on a direct prediction loss over tables. ORPO is useful for language assistants around tabular systems, such as text-to-SQL helpers and semantic labelers, where chosen and rejected answers are easy to define from execution results or expert review.

What data does ORPO need?

ORPO needs prompts with a chosen answer and a rejected answer for each one. Because chosen answers are also used for supervised learning, they should be high quality, and rejected answers should be plausible mistakes so that the contrast is informative.

Glossary

• Adapter: a small trainable module added to a frozen model.
• Alignment: shaping model behavior to match human intent.
• Chosen answer: the preferred response in a pair.
• Geometric mean: the exponential of the average log value, used for length normalization.
• Lambda: the weight on the odds-ratio term.
• Monolithic training: doing several training goals in one stage.
• Odds: probability divided by one minus probability.
• Odds ratio: the odds of one outcome divided by the odds of another.
• Offline training: learning from stored data.
• Preference pair: a prompt with a chosen and a rejected answer.
• Reference model: a frozen copy of a starting model, not needed in ORPO.
• Rejected answer: the less preferred response in a pair.
• SFT: supervised fine-tuning on demonstrations.
• Sigmoid: a function that squashes any number into the range from zero to one.

Summary

ORPO aligns a language model in one stage by adding an odds-ratio penalty to the standard supervised loss. It needs no reference model, no reward model and no separate preference stage, which makes it light on memory and simple to run, especially with adapters and when starting from a base model. It depends on high-quality chosen answers, paired data and careful tuning of lambda, and it lacks an explicit anchor to a reference. For tabular foundation models, ORPO is not part of pre-training; it is a practical tool for language assistants around them, such as text-to-SQL and semantic labeling helpers. Compare it with DPO, IPO, KTO and RLHF when choosing a method, and see /model/benchmarks for how the numeric side is evaluated.

Related

It belongs to Alignment in my stack, under LLMs, retrieval, and evaluation. Others in the same group: