IPO

Overview

IPO, short for identity preference optimization, is a method for tuning a language model on pairs of preferred and rejected answers. It was designed to fix a specific weakness of DPO: when the training labels look almost certain, DPO can push the model to extreme probabilities and effectively ignore its own regularization. IPO replaces DPO's logistic loss with a squared loss that has a fixed target, so the model stops improving the margin once it reaches that target.

In practice IPO looks very similar to DPO. It uses the same data, the same frozen reference model and nearly the same code. The difference is a single line in the loss. This article explains why that change matters, shows the math in plain language, gives a worked example and code, and compares IPO with RLHF, KTO, ORPO and related methods. It also explains how preference tuning relates to models that learn from raw tables.

What it is

IPO is an offline preference-optimization objective. It trains on a fixed set of examples, each containing a prompt, a chosen answer and a rejected answer. It keeps the tuned model close to a reference model, usually the supervised fine-tuned starting point.

The name comes from a more general framework. A 2023 theory paper on learning from human preferences described a family of objectives, often written as ΨPO, where a function Psi maps preference probabilities into a scale used by the loss. RLHF and DPO correspond to one specific choice, the logarithm-odds mapping. IPO is the case where Psi is the identity function, which means the preference probability is used directly.

The problem IPO targets

Consider what happens when annotators agree on almost every pair in a data set. The empirical probability that the chosen answer wins is then very close to one. DPO's loss encourages the log ratio margin to keep growing as that probability approaches one. With enough steps the margin can become huge. At that point the model assigns tiny probability to rejected answers and the influence of the reference model, the thing that was supposed to act as a regularizer, becomes negligible.

An everyday analogy is a student who gets a gold star for every correct quiz. If the rule is "more confidence is always better", the student may become wildly overconfident after a few perfect scores. A good teacher caps the reward: once the student shows enough confidence, no more stars. IPO is that cap.

What the fix looks like

IPO asks the log-ratio margin to hit a specific target value and penalizes deviations in either direction. Because the loss is squared, being too far above the target is as bad as being too far below. The target depends on a regularization strength parameter. This keeps the policy tied to the reference even when the labels are nearly deterministic.

Where it sits in a pipeline

1. Pre-train a base model.
2. Run supervised fine-tuning.
3. Collect preference pairs.
4. Run IPO instead of DPO, with the SFT model as reference.

Where it came from

The theoretical paper behind IPO appeared in 2023 and was titled "A General Theoretical Paradigm to Understand Learning from Human Preferences". It was a response to the rapid adoption of DPO. The authors pointed out that both RLHF and DPO rely on two assumptions:

1. Pairwise preferences can be replaced by pointwise scores, as in the Bradley-Terry model from 1952.
2. A reward model trained on those scores will generalize well to the policy's own outputs.

They argued that these assumptions can cause overfitting, especially when preference data is deterministic or the number of pairs is small. They then proposed a general objective that works directly with preferences, and derived IPO as a special, simple case that can be optimized with a regression-style loss.

Related history

Why people care

DPO became the default for preference tuning, but practitioners noticed that long training or small data sets often led to degraded outputs. IPO gave a principled explanation and a remedy. It is a good example of how a small change to a loss function can reflect a clear theoretical insight.

How it works

Step 1: start from the DPO quantity

DPO defines an implicit reward as a log ratio between the tuned model and the reference model. For a chosen answer y_w and a rejected answer y_l, the key quantity is:

h = log( pi(y_w|x) / pi_ref(y_w|x) ) - log( pi(y_l|x) / pi_ref(y_l|x) )

When h is positive, the tuned model has shifted probability toward the chosen answer relative to the reference. A larger h means a larger shift.

Step 2: see what DPO does with it

DPO's loss is the negative log of sigmoid(beta times h). The sigmoid saturates, but the loss never reaches exactly zero for finite h, and the gradient always pushes h upward when the label says the chosen answer wins. If every label says so, h keeps growing.

Step 3: define the IPO loss

IPO replaces that with a squared error toward a fixed target:

loss = ( h - 1 / (2 * tau) )^2

Here tau is the regularization strength, comparable to beta in DPO. The target margin is one over two tau. When h equals the target, the loss is zero. If h goes past the target, the loss rises again.

Think of tuning a guitar string. The goal is a specific pitch. Tightening more makes it sharp, which is just as wrong as being flat. IPO tunes the margin to a pitch.

Step 4: interpret tau

• A small tau means a large target margin and a looser tie to the reference.
• A large tau means a small target margin and a tight tie to the reference.

In many libraries the same hyperparameter name, beta, is used for tau, so always check the documentation to see how the loss is defined.

Step 5: why the squared loss helps

The squared loss has three useful properties:

• Bounded optimum. There is a single best margin, so the model has no incentive to run away.
• Smooth gradients. The gradient is proportional to how far h is from the target. Pairs already near the target contribute little.
• Robustness to deterministic labels. Even if every annotator agrees, the target stays finite.

Step 6: training procedure

Training is the same as in DPO.

1. Load the SFT model as both the tuned model and the frozen reference.
2. For each batch, compute log probabilities of chosen and rejected answers under both models.
3. Compute h and the squared loss.
4. Update the tuned model with gradient descent.
5. Evaluate on held-out pairs and with human or judge model comparisons.

A mental picture

DPO:  margin h ---> pushed upward forever (loss -> 0 as h -> infinity)
IPO:  margin h ---> pulled to a target value (loss = 0 only at the target)

Key concepts and terms

Why deterministic labels are a problem

If a pair is labeled the same way by every annotator, the logistic model infers that the chosen answer wins with probability one. The corresponding reward gap would be infinite. A model that tries to match this infinite gap will drift forever. IPO avoids this by never asking for an infinite gap in the first place.

A worked example

Numeric walk-through

Take tau equal to 0.1. The target margin is 1 divided by (2 times 0.1), which is 5.

For one pair, suppose the log ratios are:

Then h equals 3.0 minus (-2.0), which is 5.0. That matches the target, so the loss is (5.0 minus 5.0) squared, which is zero. The gradient is zero as well, and this pair needs no further update.

Now suppose h grows to 9.0 for another pair. The loss is (9.0 minus 5.0) squared, which is 16. The gradient pulls h back down toward 5. DPO would have kept pushing h upward. This is the core difference.

If h is only 1.0 for a third pair, the loss is (1.0 minus 5.0) squared, which is 16. The gradient pushes h up. So the same loss value can mean "too low" or "too high", and the sign of the gradient tells you which.

Code sketch

The sketch below shows how small the difference from DPO is. Model utilities are placeholders.

import torch

def ipo_loss(policy, reference, batch, tau=0.1):
    pi_w = sequence_logprob(policy, batch.prompt, batch.chosen)
    pi_l = sequence_logprob(policy, batch.prompt, batch.rejected)
    with torch.no_grad():
        ref_w = sequence_logprob(reference, batch.prompt, batch.chosen)
        ref_l = sequence_logprob(reference, batch.prompt, batch.rejected)

    h = (pi_w - ref_w) - (pi_l - ref_l)
    target = 1.0 / (2.0 * tau)
    loss = ((h - target) ** 2).mean()

    accuracy = (h > 0).float().mean()
    return loss, accuracy

Compare this with the DPO loss, which would be -torch.nn.functional.logsigmoid(beta * h).mean(). Everything else is shared.

Typical configuration

method: ipo
base_model: example.internal/sft-checkpoint
reference_model: example.internal/sft-checkpoint
tau: 0.1              # sometimes called beta in libraries
learning_rate: 5.0e-7
epochs: 1
batch_size: 64
max_length: 1024
adapter: lora         # optional

Implementation detail: length normalization

Some implementations average log probabilities over tokens before computing h. The original description used sequence-level sums. Averaging changes the scale of h and therefore the meaning of tau. If you switch between summed and averaged log probabilities, re-tune tau.

A second worked example: a small batch

Take tau equal to 0.25, so the target margin is 1 divided by 0.5, which is 2. Four pairs in a batch have these margins h:

The batch loss is the mean, (2.25 + 0 + 2.25 + 9.0) divided by 4, which is 3.375. Pair D dominates, because its margin has the wrong sign. Pair B is perfect and adds nothing. Pair C is over target and is pulled back down. This behavior shows why IPO spends effort on the pairs that are most wrong while quietly restraining pairs that are already too confident.

Variants and related ideas

• Averaged log probabilities. Some code bases divide by length before computing h, which makes the target margin easier to reach for long answers.
• IPO with an SFT term. Adding a small cross-entropy loss on chosen answers keeps their probability from falling.
• Online IPO. Fresh pairs from the current model are labeled each round, which restores some exploration.
• Other ΨPO choices. The general framework allows other mapping functions between preference probability and loss. IPO is the identity choice, and DPO is the logit choice.
• Margin-aware data weighting. Pairs with larger quality gaps can receive higher weight.

Debugging an IPO run

What to monitor

• The mean of h, which should approach the target from below.
• The standard deviation of h. A very wide spread means the model is fitting some pairs far more than others.
• The fraction of pairs with h above the target. If it is large, the model is overshooting.
• Held-out accuracy and win rate against the SFT model.
• Output length and general capability scores.

A tiny simulation of the difference

The following snippet shows, with plain numbers, how the two losses behave as the margin grows. It needs no model, only the loss formulas.

import math

tau = 0.1
beta = 0.1
target = 1.0 / (2.0 * tau)

for h in [0, 2, 5, 10, 20, 40]:
    dpo = -math.log(1.0 / (1.0 + math.exp(-beta * h)))
    ipo = (h - target) ** 2
    print(f"h={h:>3}  dpo_loss={dpo:.4f}  ipo_loss={ipo:.1f}")

The DPO column keeps shrinking toward zero as h increases, so its gradient always says "grow the margin". The IPO column falls to zero at h equal to 5 and then rises sharply. That rise is what restrains the model.

Effect of the regularization strength

These numbers apply to the summed log-ratio formulation. If you average over tokens, the reachable range changes.

Strengths and limitations

Strengths

• Resists overfitting. The finite target prevents runaway margins, especially with deterministic labels.
• Keeps the regularization meaningful. The tie to the reference stays active.
• Simple. One line of difference from DPO, with no reward model or RL loop.
• Stable. Squared losses have well-behaved gradients.
• Works with adapters. It fits with LoRA, QLoRA and other PEFT methods.
• Principled. It comes from a clear theory about when preference learning goes wrong.

Limitations

• Hyperparameter sensitivity. The target margin depends on tau, and a poor choice can underfit.
• Less widely adopted. DPO has more community recipes and tooling experience.
• Needs paired data. Like DPO, it needs a chosen and a rejected answer for each prompt.
• Offline. It cannot explore beyond the stored pairs.
• Length effects remain. It does not remove length bias by itself.
• Reference dependence. A poor reference model limits what can be reached.
• Not always better. On clean, large, noisy-label data sets, DPO can match it, and benefits depend on the data.

How it compares with alternatives

Choosing between DPO and IPO

• If your pairs come from a noisy mixture and you train for one epoch, DPO is often fine.
• If your data is small, clean and highly consistent, or you train for several epochs, IPO is worth testing.
• If DPO training shows a rapidly falling loss, very large margins and degraded samples, try IPO.
• When in doubt, run both on the same split and compare with human or judge evaluation.

Choosing IPO versus the others

• Pick KTO if you only have thumbs-up and thumbs-down labels.
• Pick ORPO if you want to skip the reference model and merge SFT with preference tuning.
• Pick RLHF if you need online exploration or reuse of a reward model.

Role in tabular foundation models

IPO is a language-model alignment technique. It does not belong to the pre-training of models that learn directly from raw tables. Those models, such as in-context tabular learners trained on very large numbers of synthetic or real tables, use a prediction loss on masked or held-out cells. The loss is precise and does not need preference labels. For a view of how such models are built, see /model/research and /model/architecture.

Where preference tuning does not apply

A tabular model reads rows and columns with mixed numeric, categorical and date types, copes with missing values, and may work even when columns have no names. It predicts targets by looking at the context rows. Quality is measured by metrics. TabPFN is a well-known example of this style. Boosted trees like XGBoost and LightGBM also optimize a fixed loss. None needs IPO.

Where IPO can help

IPO helps the language layer that surrounds tabular systems.

• Table question answering and text-to-SQL. Pairs of correct and incorrect queries are easy to form from execution results. Because execution labels are deterministic, this is exactly the setting where DPO can overfit and IPO's capped margin is attractive.
• Explanation writers. Annotators compare two explanations of a model's prediction. If annotators agree strongly, IPO resists runaway margins.
• Semantic and sector classification assistants. A language model suggests a meaning for a nameless column. Corrected suggestions form deterministic pairs.
• Data-cleaning agents. Experts choose between two cleaning plans. A small data set with consistent labels is a typical IPO use case.
• Anomaly report generation. Analysts compare alert write-ups.

Why deterministic labels are common in table tasks

Table tasks often have objective answers: the query either returns the right rows or it does not, the type is either "date" or it is not. These checks create labels with almost no disagreement. That is the situation in which logistic losses push margins to extremes, so a bounded loss like IPO has a natural role.

Conceptual parallels

• Bounded targets and calibration. Tabular models are judged on calibrated probabilities as well as accuracy. IPO's finite target echoes that concern: do not become more confident than the evidence supports.
• Regularization to a prior. The reference model plays the role of a prior. This echoes continual learning, where new learning is kept near old knowledge to avoid forgetting.
• Scores for ranking. The log-ratio margin is a score that ranks answers, a bit like an anomaly score that ranks rows. Both are only meaningful relative to a baseline.
• In-context learning contrast. An in-context tabular learner adapts to a table at inference without changing weights. IPO changes weights offline and adjusts general behavior, not dataset-specific predictions.

Mapping tabular tasks to IPO

See /model/benchmarks for how the numeric side of tabular models is evaluated, which is by metrics rather than by preference.

Practical guidance and best practices

Data

• Check label consistency. IPO helps most when labels are consistent, but inconsistent labels still hurt.
• Remove duplicates and near duplicates.
• Balance prompt types and answer lengths.
• Use pairs close to the tuned model's own distribution when possible.
• Hold out a validation set of pairs and a set of prompts for generation.

Hyperparameters

• Treat tau (or the library's beta) as the main knob. Try a small grid, for example 0.01, 0.1 and 0.5, and compare on held-out data.
• Remember the target margin is one over two tau. A tau of 0.1 gives a target of 5, which is a large log ratio for sums over long answers. Make sure that target is reachable.
• Use a small learning rate and one or two epochs.
• With adapters, raise the learning rate carefully and monitor margins.

Process

• Always start from an SFT model and use it as the reference.
• Cache the reference log probabilities to save memory and compute.
• Compare IPO to a DPO baseline on the same data, so you know whether the change helps.
• Run iterative rounds: generate fresh pairs from the tuned model, label them and train again.
• Inspect samples from several checkpoints, not only the last one.

Evaluation

• Report held-out pair accuracy and the mean margin.
• Use pairwise human or judge model comparison for the final decision.
• Check capability benchmarks to catch regressions.
• Check safety behavior, including false refusals.

A short pre-flight checklist

• The SFT model follows the target format.
• The reference model matches the starting model, with the same tokenizer and chat template.
• Prompt formatting in the pair data matches the format used in serving.
• The held-out pair set is separate and never used for training.
• Logs show loss, mean and spread of h, accuracy, length and sample text.
• A DPO baseline is available for comparison.
• Human review of outputs is scheduled for several checkpoints.
• Rollback checkpoints are stored.

How IPO fits into a larger alignment program

Preference optimization is rarely used alone. A typical program has several layers.

1. Supervised fine-tuning teaches format and basic behavior.
2. Preference optimization such as IPO, DPO or RLHF refines style, helpfulness and safety.
3. Rule-based or verifier-based checks catch factual and format errors.
4. Evaluation with humans and judge models measures progress.
5. Periodic refreshes bring in new prompts and new pairs as usage changes.

IPO belongs in the second layer. It is an option you reach for when the data and the logistic loss do not cooperate.

Common mistakes

• Confusing tau with beta. The two parameters work in opposite directions in some formulations. Read the loss in your library before copying values from DPO.
• Targeting an unreachable margin. If tau is tiny, the target margin is huge and the model may chase it unsuccessfully.
• Assuming IPO fixes bad data. It limits overfitting, not label noise or unrepresentative pairs.
• Skipping SFT. A weak reference and unformatted outputs make any preference method perform poorly.
• Mixing summed and averaged log probabilities. This changes the meaning of the target.
• Training too long. Even with a bounded loss, repeated passes can memorize pairs.
• Reading a low loss as success. Check samples and held-out win rates.
• Ignoring length. IPO does not remove length bias.
• Using the wrong reference. The reference must be the starting model with the same tokenizer and template.
• Comparing runs with different data splits. Differences may come from the split, not the loss.

Guidance for different data situations

Signs that IPO is the right tool

• A DPO run shows loss near zero within a fraction of an epoch.
• Rejected answers receive vanishing probability and samples look strange.
• Held-out win rates fall while training accuracy rises.
• The preference data comes from a verifier, not from humans, so there is no label disagreement.

Signs that IPO will not help

• The main problem is wrong or inconsistent labels.
• The pairs do not match the tuned model's output distribution.
• The reference model is poor.
• The problem is length bias or format drift, which need data fixes.

Frequently asked questions

What does IPO stand for in machine learning?

In this context, IPO stands for identity preference optimization. It is a method for aligning language models with preference pairs and gets its name from the identity mapping used in a general theory of preference learning, rather than from the financial term for a stock market listing.

How is IPO different from DPO?

Both methods use the same pairs and the same log-ratio margin between the tuned model and a reference model. DPO applies a logistic loss that always rewards a larger margin, while IPO applies a squared loss toward a fixed target margin, which prevents the margin from growing without limit and protects the regularization toward the reference.

Why does DPO overfit?

When labels are nearly deterministic, the logistic model implies an infinite reward gap between chosen and rejected answers. Training pushes the margin larger and larger, and the influence of the reference model fades. The tuned model can then assign extreme probabilities and produce degraded text, especially after several epochs or with small data sets.

What is tau in IPO?

Tau is the regularization strength that sets the target margin, which equals one divided by two times tau. A small tau means a large target margin and looser tie to the reference, while a large tau means a small target margin and a tighter tie. Some software calls this parameter beta, so check the formula used.

Does IPO need a reward model?

No. Like DPO, IPO works directly on pairs of chosen and rejected answers, using the model itself and a frozen reference to compute a log-ratio margin. There is no separate reward network and no reinforcement learning loop, which keeps training simple and memory use modest.

Is IPO better than DPO?

It depends on the data and the training length. IPO tends to help when preference labels are very consistent, when data is small or when training runs for multiple epochs. On large, noisy data sets with one epoch, the two often perform similarly, so a direct comparison on your own validation set is the best guide.

Can IPO be used with LoRA or QLoRA?

Yes. Adapter methods work with IPO in the same way as with DPO. The reference model can be the same network with the adapter disabled, which saves memory, and the squared loss does not change how adapters are trained.

Does IPO fix length bias?

Not by itself. The loss changes how margins are targeted, but the data and the use of summed log probabilities can still make long answers look better or worse. Balancing lengths in the data or normalizing log probabilities remains a good idea.

How does IPO relate to tabular foundation models?

It does not train them. Tabular foundation models learn from a direct prediction loss on many tables. IPO is useful for language assistants around them, such as text-to-SQL helpers or explanation writers, where chosen and rejected answers can be formed from execution results or expert review.

What data format does IPO use?

Each example contains a prompt, a chosen answer and a rejected answer, typically stored as one JSON record per line. The chat template used for these records should match the template used when the model is served, so that the log probabilities reflect real usage.

Glossary

• Alignment: making model behavior match human intent.
• Bradley-Terry model: a model that turns latent scores into pairwise win probabilities.
• Chosen answer: the preferred response in a pair.
• Deterministic preference: a pair where every annotator picks the same answer.
• Identity mapping: using a value directly with no transformation.
• Log-ratio margin: the difference between the chosen and rejected log ratios.
• Offline training: learning from stored data.
• ΨPO: a family of preference objectives defined by a mapping function.
• Reference model: a frozen copy of the starting model.
• Regularization: a constraint or penalty that limits how far a model can move.
• Rejected answer: the less preferred response in a pair.
• SFT: supervised fine-tuning on demonstration data.
• Squared loss: a loss equal to the square of the error.
• Target margin: the value the log-ratio margin is trained toward.
• Tau: the regularization strength in IPO.

Summary

IPO is a preference-optimization method that keeps the simplicity of DPO while fixing its tendency to overfit when labels are nearly deterministic. It replaces the logistic loss with a squared loss toward a fixed target margin set by a regularization parameter, so the tuned model stays tied to its reference. It needs paired data, a reference model and careful tuning of tau, and it does not remove length bias or label noise. For tabular foundation models, IPO is not part of pre-training; it is a tool for language assistants around them, such as text-to-SQL and explanation writers, where objective labels are common. The numeric core of such models is evaluated by metrics, as described in /model/benchmarks.

Related

It belongs to Alignment in my stack, under LLMs, retrieval, and evaluation. Others in the same group: