SimPO

Overview

SimPO, short for Simple Preference Optimization, is a method for aligning a language model with human or AI preferences without a reference model. It takes pairs of answers, one preferred and one rejected, and trains the model so that the preferred answer gets a higher score. The score is the average log-probability the model assigns to each token of the answer. That one choice removes the frozen reference copy of the model that DPO needs, which lowers memory use and simplifies the training loop.

The method adds two small ideas on top of this score. First, it divides the log-probability by the answer length, so long answers are not rewarded just for being long. Second, it adds a target margin, a gap that the preferred answer must beat the rejected answer by. Together these make the training signal match how the model actually generates text at inference time. This article explains what SimPO is, where it came from, how it works step by step, and how it fits next to DPO, IPO, KTO and ORPO. It also covers how preference-tuning ideas relate to models that learn from raw tables.

What it is

SimPO is an offline preference optimization objective. "Offline" means it learns from a fixed dataset of pairs that was collected before training. It does not sample new answers during training, and it does not run a reinforcement learning loop like PPO.

Each training example has three parts:

• A prompt, often called x.
• A chosen answer, the one people or a judge preferred. Call it y_w for "winner".
• A rejected answer, the one that lost. Call it y_l for "loser".

The goal is to shift the model so it is more likely to produce answers like y_w and less likely to produce answers like y_l.

The core idea in plain words

Think of a cooking contest. In older methods, a judge scores a dish by comparing the new chef with a fixed "house chef" who stays in the kitchen as a baseline. That baseline is the reference model. SimPO drops the house chef. Each dish is scored on its own merits, by how confident the chef is in each step of the recipe, averaged over the number of steps. A dish is not given extra credit just because the recipe had more steps.

What problem it solves

Preference tuning with a reference model has a cost. You must keep two copies of the model in memory, or precompute the reference log-probabilities for the whole dataset. SimPO removes that cost. It also fixes a mismatch. In DPO the training reward is a log-ratio between the policy and the reference, but at generation time the model ranks candidates by their likelihood, not by that ratio. SimPO makes the training reward the same quantity the model uses when it generates.

Where it sits in the training pipeline

A typical post-training pipeline has three stages:

• Pre-training on large raw corpora.
• Supervised fine-tuning, see SFT, on demonstrations of good answers.
• Preference optimization on pairs of better and worse answers.

SimPO belongs to the third stage. It usually starts from a model that has already gone through SFT. The library most people use to run it is TRL, and it is often combined with parameter-efficient methods such as LoRA through PEFT.

Where it came from

Preference learning for language models began with reinforcement learning from human feedback, covered in RLHF. That recipe trains a separate reward model from human comparisons, then optimizes the language model against it with an algorithm such as PPO. It works, but it is complex. It needs several models, careful tuning, and a lot of memory.

In 2023 the DPO paper, "Direct Preference Optimization", showed that you can skip the explicit reward model and the RL loop. It rewrote the problem as a simple classification-style loss on preference pairs. This made preference tuning much easier and sparked a family of follow-up methods. IPO changed the loss to reduce overfitting. KTO removed the need for paired data. ORPO merged SFT and preference learning into one stage by dropping the reference model with an odds-ratio term.

SimPO appeared in 2024 in the paper "SimPO: Simple Preference Optimization with a Reference-Free Reward". It continued the push toward simplicity. Its authors argued that the reference model and the log-ratio reward in DPO were not needed, and that using the length-normalized average log-probability as the reward, plus a target margin, was enough. The paper was presented at a major machine learning conference in 2024.

Why the field moved this way

Three pressures drove the trend from RLHF to DPO to SimPO:

• Cost: fewer models in memory means cheaper runs and bigger batch sizes.
• Stability: simpler losses have fewer hyperparameters that interact in odd ways.
• Reproducibility: a single supervised-style training script is easier to share and re-run than a full RL system.

For more on how these ideas feed the wider research agenda, see /model/research.

How it works

This section walks through SimPO step by step.

Step 1: Compute the average log-probability of an answer

A language model generates one token at a time. For each token it gives a probability. The log-probability of a whole answer is the sum of the log-probabilities of its tokens. SimPO uses the average instead of the sum.

Formally, for a prompt x and an answer y with |y| tokens, the implicit reward is:

r(x, y) = (beta / |y|) * log pi_theta(y | x)

Here pi_theta is the model being trained, and beta is a scale factor. Dividing by |y| is the length normalization.

An everyday analogy: imagine grading essays by "how smooth is each sentence on average" instead of "how much total smoothness did the essay accumulate". Without the average, a ten-page essay would always beat a one-page essay on total score, even if each sentence were worse.

Step 2: Compare the chosen and rejected answers

For each pair, compute the reward for y_w and the reward for y_l. The difference between them is what the loss pushes up:

delta = r(x, y_w) - r(x, y_l)

If delta is large and positive, the model already prefers the winner, and the loss is small.

Step 3: Add a target margin

SimPO subtracts a positive constant gamma before applying the sigmoid. The preferred answer must beat the rejected answer by at least gamma, not just by any amount:

loss = - log sigmoid( delta - gamma )

Think of a high-jump bar. Clearing the rejected answer by a hair is not enough. The bar is set at gamma. A bigger margin tends to separate good and bad answers more cleanly, which helps the model generalize.

Step 4: Average over the dataset and update

The loss is averaged over all pairs in a batch. Gradients flow only through the policy model. There is no reference model, so there are no extra forward passes through a frozen copy. Then an optimizer such as AdamW updates the weights.

Step 5: Repeat for a few epochs with a small learning rate

Preference tuning needs small learning rates and few epochs. Large updates can wreck the abilities the model learned in SFT. In practice people use learning rates that are a fraction of those used for SFT and train for one to a few epochs.

Putting the loss together

The full SimPO objective over a dataset D is:

L(theta) = - E_{(x, y_w, y_l) in D} [
    log sigmoid( (beta/|y_w|) log pi_theta(y_w|x)
               - (beta/|y_l|) log pi_theta(y_l|x)
               - gamma )
]

There are only two key hyperparameters beyond the usual learning rate: beta and gamma.

Compare to DPO's loss

DPO uses:

loss = - log sigmoid( beta * ( log pi_theta(y_w|x)/pi_ref(y_w|x)
                             - log pi_theta(y_l|x)/pi_ref(y_l|x) ) )

The reference model pi_ref appears in every term. SimPO removes it, adds length normalization, and adds the margin.

What the gradient does

Intuitively, the gradient raises the per-token probabilities of tokens in the chosen answer and lowers those of tokens in the rejected answer. Because of length normalization, each token of a short answer carries more weight than each token of a long one. The margin keeps the gradient active until the gap is big enough, so training does not stop too early on easy pairs.

Key concepts and terms

A worked example

A tiny numeric walk-through

Suppose a prompt asks for a one-line definition of "overfitting". We have two answers.

• Chosen answer: 10 tokens, total log-probability of -12.0 under the model.
• Rejected answer: 30 tokens, total log-probability of -33.0 under the model.

Set beta to 2.0 and gamma to 0.5 for the arithmetic.

Average log-probability of the chosen answer: -12.0 / 10 = -1.2.

Average log-probability of the rejected answer: -33.0 / 30 = -1.1.

Rewards:

• r(chosen) = 2.0 * (-1.2) = -2.4
• r(rejected) = 2.0 * (-1.1) = -2.2

Delta = -2.4 - (-2.2) = -0.2. Subtract gamma: -0.2 - 0.5 = -0.7.

The loss is -log sigmoid(-0.7), which is about 1.10. That is a high loss, because the model currently prefers the wrong answer per token. Training will push the chosen answer's per-token probabilities up and the rejected one's down.

Notice what length normalization did here. On raw sums, the rejected answer scores -33.0 and the chosen answer scores -12.0, so the short answer looks far better. On averages, they are close, and the model must actually improve per-token quality. That is the point of the normalization: it removes a free ride that comes from length alone.

A code example with TRL

The example below shows a typical setup using the TRL library. Check your installed version for exact argument names, since library APIs change over time.

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import CPOConfig, CPOTrainer

model_name = "example-org/example-sft-model"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# A dataset with columns: prompt, chosen, rejected
dataset = load_dataset("json", data_files="prefs.jsonl")["train"]

config = CPOConfig(
    output_dir="out-simpo",
    loss_type="simpo",          # selects the SimPO objective
    cpo_alpha=0.0,              # no extra SFT term
    simpo_gamma=0.5,            # target margin
    beta=2.0,                   # reward scale
    learning_rate=5e-7,
    num_train_epochs=1,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=16,
)

trainer = CPOTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    processing_class=tokenizer,
)
trainer.train()

In recent TRL releases, SimPO is available as a loss type inside a trainer from the CPO family, because the two objectives share the reference-free structure. The exact class and flag names can differ across versions.

Preparing the preference data

The dataset needs three text fields per row. A simple JSON lines file looks like this:

{"prompt": "Define overfitting in one line.", "chosen": "Overfitting is when a model memorizes training noise and fails on new data.", "rejected": "It is a thing that can happen in many different situations when models are used and trained in many ways."}

Good pairs share these traits:

• Both answers respond to the same prompt.
• The chosen answer is better for a clear reason, such as accuracy or clarity.
• The rejected answer is plausible, not nonsense, so the model learns a real distinction.

Choosing beta and gamma

Start from published defaults for your model family if you have them. As a rule of thumb, beta in SimPO is larger than in DPO because the average log-probability is a small number. Sweep gamma over a small grid and watch both quality metrics and average answer length on a held-out set.

Monitoring training

Watch these signals:

• The reward accuracy: the fraction of pairs where the chosen reward exceeds the rejected reward.
• The reward margin: the average gap between chosen and rejected rewards.
• The average length of sampled outputs.
• The log-probability of chosen answers. If it drops sharply, the model may be degrading.

Strengths and limitations

Strengths

• No reference model, so lower memory and faster steps than DPO.
• A training reward that matches the generation-time likelihood.
• Length normalization reduces the habit of producing long, padded answers.
• Only a few hyperparameters.
• Easy to run with standard libraries.

Limitations

• Sensitive to hyperparameters. Beta, gamma and the learning rate interact, and a poor choice can cause the chosen log-probability to collapse.
• Without a reference anchor, the model can drift further from the starting point. There is no built-in pull back to the SFT model.
• Needs paired data. If you only have thumbs-up and thumbs-down labels, KTO fits better.
• Offline only. It cannot explore new answers the way online methods can.
• Results depend on data quality. Noisy preference labels hurt every method of this family.
• Length normalization is not a full cure for length bias. It changes the pressure but does not remove it.

When the drift risk matters

DPO's reference model acts like a leash. SimPO has no leash, so it relies on small learning rates, few epochs and a good SFT starting point to keep the model stable. If you see the model forgetting skills, lower the learning rate, shorten the run, or mix in a small SFT loss term.

How it compares with alternatives

How to choose

• If you want the most documented path and have the memory, DPO is a safe default.
• If memory is tight or you want a simpler loop, SimPO is a strong candidate.
• If your labels are single-sided, use KTO.
• If you want one-stage training, look at ORPO.
• If labelling is the bottleneck, combine any of the above with AI-generated preferences, see RLAIF and Constitutional AI.

Role in tabular foundation models

SimPO is a language-model alignment method, so it does not act directly on tables. Its relevance to tabular foundation models is indirect but real, and it helps to be precise about it.

What a tabular foundation model is

A tabular foundation model learns from raw tables. It treats rows as examples and columns as features. It handles mixed types, such as numbers, categories, dates and short text. It copes with missing values. It often cannot rely on column names, so it must infer meaning from values and statistics. Many such models are pre-trained on large numbers of synthetic tables, drawn from random processes, and then use in-context learning: you give the model a labelled table as context and it predicts for new rows with no gradient updates. TabPFN is a well-known example of this style. For the architecture side, see /model/architecture, and for evaluation see /model/benchmarks.

Where preference tuning touches tables

Preference optimization applies when a model's output is free-form text, and a language model sits somewhere around the table. Concrete cases:

• Text-to-SQL and table question answering. A language model writes a query or an answer about a table. Pairs of better and worse queries, judged by execution results, are natural preference data for SimPO.
• Explanations and reports. A model that summarizes a table or explains a prediction can be tuned toward clear, faithful, concise text. The length normalization in SimPO helps resist padded summaries.
• Semantic and sector classification with a language model head. When a text model labels columns or rows with business categories, preference pairs can teach it to prefer the more specific, correct label.
• Data cleaning agents. A model that proposes fixes for missing values or odd records can be tuned on pairs where one fix passed validation and the other failed.
• Imputation and anomaly work with language models in the loop. The numeric core may be a tabular model, see imputation networks and anomaly detection, but a text interface that explains why a value was filled or flagged can be aligned with preference data.

What SimPO does not do

• It does not change how a tabular model reads a table.
• It does not replace pre-training on synthetic tables.
• It is not a way to do continual learning on numeric data.
• It does not give a model column-name awareness.

Parallels that are useful

There is a conceptual link worth noting. In both worlds, the aim is to make a model behave well without hand-written rules. A tabular foundation model learns a general prediction procedure from many synthetic datasets. A preference-tuned language model learns a general behavior from many comparisons. In both, the quality of the training distribution decides the result. If synthetic tables do not cover real-world missingness patterns, the tabular model will struggle with them. If preference pairs do not cover real user tasks, the aligned model will too.

A realistic pipeline

A system that lets people chat with their tables might look like this:

• A tabular foundation model does the numeric prediction, imputation or anomaly scoring.
• A language model reads the user question and calls tools.
• The language model is tuned with SFT on tool-use examples.
• It is then refined with a preference method such as SimPO on pairs where the better answer was correct, grounded in the table and concise.

In that design SimPO shapes the conversational layer only. The numbers still come from the tabular model.

Honest caveats

Preference tuning on table-related text needs reliable judges. Execution-based checks for SQL are strong. Human labels of "good explanation" are noisier. Use checks you can verify, and keep a held-out set of real tables to catch regressions.

Practical guidance and best practices

Data

• Start with a few thousand clean pairs rather than a huge noisy set.
• Make sure the chosen answer is better for a reason you can state.
• Balance lengths. If chosen answers are always longer, the model learns length, not quality.
• De-duplicate prompts and remove pairs where the answers are nearly identical.
• Hold out a validation set of prompts the model never sees in training.

Model setup

• Begin from an SFT checkpoint of reasonable quality.
• Use the same chat template at training and inference time.
• Consider LoRA or QLoRA through PEFT if memory is limited. Adapters work well with reference-free methods because you do not need a second model copy.

Hyperparameters

Evaluation

• Use a held-out preference set and measure reward accuracy.
• Run pairwise comparisons against the SFT baseline with a judge, ideally with position swapping to avoid order bias.
• Track answer length and refusal rate.
• Test general skills such as reasoning and format following to catch forgetting.
• Do not rely on a single leaderboard number.

Checklist before you ship

• Did chosen log-probabilities stay reasonably stable?
• Is the average output length sensible?
• Did general benchmarks hold steady?
• Did you run a red-team style check, see red teaming, to confirm safety did not regress?

A plain-language picture of the margin

Picture a judge scoring two essays by the average quality of each sentence, so that a long essay is not favoured just for being long. SimPO gives each answer a score based on the average per-token log-probability. It then asks that the preferred answer beat the rejected one by at least a fixed gap, the target margin. A gap makes the model separate the two more firmly than a bare "slightly better" would. Without a gap, the training can stop at a tiny difference that is fragile at test time.

Why length normalization matters

• Summed log-probability favours short answers, because each extra token adds a negative number.
• Some preference methods then drift toward longer answers to compensate for other effects.
• Averaging per token puts both answers on the same footing and matches how text is generated, where the per-token probability is what the decoder uses.
• The result is a reward that lines up more closely with the generation process.

Quick comparison table

See also DPO, ORPO, IPO and KTO.

Tuning advice

1. Begin with a small learning rate, because the method has no reference anchor to hold the model near its start.
2. Sweep the reward scale and the target margin over a small grid and compare on a held-out preference set.
3. Check the average length of generated answers during training, to catch length drift early.
4. Keep the number of epochs low, often one to three.
5. Evaluate with a mix of preference win-rate and task metrics, so that style gains are not mistaken for quality gains.

Failure signs

• Answers grow steadily longer or shorter.
• The log-probability of the chosen answers collapses.
• The model repeats stock phrases from the preferred set.
• General ability drops on unrelated prompts.

Each of these suggests lowering the learning rate, reducing the scale, or adding some supervised data to the mix.

Fit with parameter-efficient training

Because no reference model is held, SimPO combines well with low-rank adapters. See LoRA, QLoRA, PEFT and TRL for the usual tooling, and /model/research for how preference ideas relate to table models.

Common mistakes

• Using a learning rate copied from SFT. Preference tuning needs far smaller steps.
• Training too many epochs. The model overfits the pairs and its fluency drops.
• Ignoring the length distribution of chosen and rejected answers.
• Using pairs where the "rejected" answer is trivially bad. The model learns nothing subtle.
• Using a different chat template at inference than during training.
• Skipping the SFT stage. Starting from a raw base model often gives unstable results.
• Treating gamma as a free knob and never checking its effect on length and quality.
• Judging success only with an automatic judge that itself prefers long answers.
• Mixing noisy labels from different annotators without any quality filter.
• Assuming that reference-free means safe by default. Without an anchor, drift is your job to monitor.

Additional pitfalls

• Reusing DPO settings unchanged. The reward scale and margin play different roles here.
• Training on pairs where the rejected answer is simply shorter, which teaches length rather than quality.
• Judging success by a single win-rate score that can itself favour long answers.
• Ending training without checking generated answers by eye.

Frequently asked questions

What does SimPO stand for?

SimPO stands for Simple Preference Optimization. It is a method for training a language model on pairs of preferred and rejected answers using the average log-probability of each answer as the reward, with no reference model and with a target margin between the two rewards.

How is SimPO different from DPO?

DPO measures how much more likely the trained model makes an answer compared with a frozen reference model, while SimPO scores each answer directly by its length-normalized log-probability. SimPO also adds a target margin. The result is that SimPO needs no reference copy, uses less memory, and ties its training reward to the way the model generates text.

Does SimPO need a reference model?

No. The absence of a reference model is the defining feature of SimPO. All it needs is the model being trained and a dataset of prompts with chosen and rejected answers. This is why it can run with less memory than DPO on the same hardware.

Why does SimPO divide by the answer length?

Dividing by length stops long answers from winning only because they are long, or short answers from winning only because they have fewer tokens to pay for. The average log-probability measures quality per token. This makes the learned preference less tied to length, although it does not remove length bias entirely.

What is the target margin gamma for?

Gamma sets how much better the chosen answer's reward must be than the rejected answer's reward before the loss becomes small. A positive margin keeps pushing the model to separate good and bad answers by a clear gap, which tends to improve how well the learned preference generalizes to new prompts.

Can I use SimPO with LoRA?

Yes. Because SimPO needs no reference model, it pairs well with adapter methods such as LoRA and QLoRA through the PEFT library. You train a small set of extra weights while the base model stays frozen, which keeps memory use low. Learning rates for adapters are usually higher than for full fine-tuning.

Is SimPO better than DPO?

It depends on the model, the data and the evaluation. SimPO is simpler and cheaper, and reported results in its paper were strong, but it can be more sensitive to hyperparameters and can drift without a reference anchor. It is best to compare both on your own held-out tasks rather than assume one always wins.

What kind of data does SimPO need?

It needs preference pairs, where each prompt has a better and a worse answer. The pairs can come from human annotators, from an AI judge as in RLAIF, or from automatic checks such as test results or query execution. Quality and diversity matter more than raw volume.

Does SimPO help with tabular data?

Not directly. SimPO is a text alignment method. It can improve language models that write SQL, explain table results or label columns, but the prediction, imputation and anomaly scoring on tables themselves come from tabular models trained on rows and columns.

When should I choose something other than SimPO?

Choose KTO if you only have single thumbs-up or thumbs-down labels, ORPO if you want a single stage that combines SFT and preference learning, and DPO if you value a long track record and a reference anchor. Choose an online RL method if you need the model to explore and learn from fresh samples.

Glossary

• Alignment: shaping a model so its outputs match human goals and values.
• Average log-probability: the sum of token log-probabilities divided by the token count.
• Beta: the scale factor applied to the implicit reward.
• Chosen answer: the preferred response in a pair.
• Gamma: the target margin between chosen and rejected rewards.
• Implicit reward: a reward derived from the model's own probabilities.
• Length normalization: dividing a sequence score by its length.
• Offline preference optimization: learning from a fixed set of pairs.
• Preference pair: a prompt with a chosen and a rejected answer.
• Reference model: a frozen copy of the starting model used as an anchor in some methods.
• Rejected answer: the less preferred response in a pair.
• Reward accuracy: the share of pairs where the chosen reward is higher.
• SFT: supervised fine-tuning on demonstration data.

Summary

SimPO is a reference-free way to tune a language model on preference pairs. It scores each answer by its average token log-probability, requires the chosen answer to beat the rejected one by a margin, and trains with a simple sigmoid loss. That design cuts memory use, matches training to generation, and reduces reward for sheer length.

It is not magic. It needs clean pairs, small learning rates, careful monitoring and a good SFT starting point, because nothing anchors it to the original model. It sits in the same family as DPO, IPO, KTO and ORPO, and it works well with TRL and PEFT.

For tabular foundation models, SimPO matters at the language layer: it can align the text interface that writes queries, explanations and labels, while the table-native model handles prediction, imputation and anomaly scoring. To see how the table-native side is built and measured, read /model/architecture and /model/benchmarks.

Related

It belongs to Alignment in my stack, under LLMs, retrieval, and evaluation. Others in the same group: