KTO

Overview

KTO, short for Kahneman-Tversky optimization, is a method for aligning a language model using simple labels on single answers: this answer is good, or this answer is bad. It does not need paired comparisons of two answers to the same prompt. That makes it a natural fit when your feedback comes as thumbs-up and thumbs-down clicks, pass and fail checks, or accepted and rejected suggestions.

The method borrows an idea from behavioral economics called prospect theory, which describes how people actually value gains and losses. People feel losses more strongly than equal gains, and they judge outcomes relative to a reference point. KTO builds those two ideas into the loss. This article explains the intuition, shows the loss with a worked example and code, compares KTO with DPO, IPO, ORPO and RLHF, and describes how it relates to models that learn from raw tables.

What it is

KTO is an offline alignment objective. It trains on a data set where each example is a prompt, an answer and a binary label that says whether the answer is desirable. It also uses a frozen reference model, normally the supervised fine-tuned starting point, to keep the tuned model from drifting too far.

Think of a restaurant that collects feedback cards. Most guests do not compare two meals side by side. They simply tick "liked it" or "did not like it" for the meal they had. A restaurant can still learn a lot from those cards. KTO is a training method designed for that kind of data.

The core idea in one sentence

KTO raises the probability of answers labeled good and lowers the probability of answers labeled bad, measured relative to a reference model, using a value function that weights losses more than gains.

Why single-answer labels matter

Paired data is expensive. To create a pair, you need two answers to the same prompt, and a person (or a checker) must compare them. Binary feedback is far more common and cheaper:

• user clicks on a thumbs-up or thumbs-down button
• a unit test passes or fails
• a query returns the right result or does not
• a moderator accepts or rejects a suggestion
• a customer marks a response as resolved or unresolved

KTO uses this data directly. It also tolerates imbalance, when there are many more good answers than bad ones or the reverse.

What it keeps from earlier methods

• A frozen reference model and a log-ratio between the tuned and reference models, as in DPO.
• No separate reward model and no reinforcement learning loop.
• Compatibility with adapters such as LoRA and QLoRA.

What is new

• Single-answer, binary labels instead of pairs.
• A loss shaped by prospect theory, with a reference point and different weights for good and bad answers.
• An explicit handling of class imbalance through two weights.

Where it came from

KTO was introduced in 2024 in a paper titled "KTO: Model Alignment as Prospect Theoretic Optimization". The method name refers to the two researchers who developed prospect theory in 1979. That theory describes decision making under risk and was a foundation of behavioral economics.

The paper's argument has two parts. First, it observed that popular alignment losses, including DPO, can be viewed as "human-aware losses" that implicitly model human biases. Second, it proposed a loss that models those biases more directly, using a value function in the style of prospect theory, and showed that this works with only binary signals.

Prospect theory in plain terms

Prospect theory says that people:

1. Judge outcomes relative to a reference point rather than in absolute terms. Finding ten dollars feels like a gain if you expected nothing, and a disappointment if you expected twenty.
2. Feel losses more strongly than gains of the same size. Losing ten dollars hurts more than finding ten dollars pleases. This is called loss aversion.
3. Show diminishing sensitivity. The difference between gaining one dollar and two dollars feels larger than between gaining one hundred and one hundred and one.

KTO uses all three. The reference point is an estimate of the average drift of the tuned model from the reference. The loss weights allow bad answers to count more or less than good ones. A saturating function gives diminishing sensitivity.

Where it sits in the timeline

How it works

Step 1: compute the implicit reward

As in DPO, KTO defines an implicit reward for an answer y given a prompt x as a log ratio between the tuned model and the reference model:

r(x, y) = log( pi(y | x) / pi_ref(y | x) )

If r is positive, the tuned model finds the answer more likely than the reference does. If negative, less likely.

Step 2: estimate a reference point

Prospect theory needs a reference point. KTO estimates it as a batch-level average of the KL divergence between the tuned and reference models. To avoid a costly computation, it uses answers paired with mismatched prompts in the same batch and takes the mean of their log ratios, clipped at zero.

z_ref = max(0, mean over batch of log( pi(y' | x) / pi_ref(y' | x) ))

where y' comes from a different example than x. The reference point is not trained by gradient. It is a running estimate that tells you how far the model has moved on average.

An analogy is a thermostat. The reference point is the current room temperature, and judgments about "warmer" or "cooler" are made relative to it, not against an absolute scale.

Step 3: define the value for each label

For a desirable answer, the value rises when its implicit reward exceeds the reference point. For an undesirable answer, the value rises when its implicit reward falls below the reference point. Both use a sigmoid, which provides diminishing sensitivity:

if label is desirable:
    value = lambda_D * sigmoid( beta * ( r - z_ref ) )
if label is undesirable:
    value = lambda_U * sigmoid( beta * ( z_ref - r ) )

Beta controls the sharpness, as in DPO.

Step 4: define the loss

The loss is the weight times one minus the value, averaged over the batch:

loss = mean( lambda_y - value )

Minimizing it raises the value of each answer. For good answers this pushes r above the reference point. For bad answers it pushes r below the reference point.

Step 5: set the weights for balance and loss aversion

The weights lambda_D and lambda_U serve two purposes.

• Balance. If you have many more good answers than bad ones, weight the minority class more so that both contribute equally.
• Loss aversion. Making lambda_U larger than lambda_D mimics loss aversion, so avoiding bad answers counts more than producing good ones.

A common recommendation is to set the ratio of weighted counts of desirable to undesirable examples in a range near one, roughly between one and one and a half, adjusting for the data balance.

Step 6: train as usual

Training is a normal supervised loop:

1. Load the SFT model as tuned model and frozen reference.
2. For each batch, compute log probabilities under both models.
3. Compute the implicit rewards, the reference point and the loss.
4. Update the tuned model.
5. Evaluate on held-out data and with human or judge model comparisons.

A mental picture

good answer:  push log-ratio UP   past the reference point (gain)
bad answer:   push log-ratio DOWN past the reference point (avoid loss)
loss weights: bad answers can count more than good ones

Key concepts and terms

Binary feedback signals

A binary signal is anything that can be reduced to good or bad. It can come from human clicks, automated tests, rule checks or a judge model. Quality varies: human clicks are noisy and biased toward certain styles, while tests are consistent but narrow. Mixing sources is possible, but be aware of their different biases.

A worked example

Numeric walk-through

Set beta to 0.1 and both lambdas to 1. Suppose the reference point z_ref is 1.0.

For a desirable answer with implicit reward r equal to 6.0:

• r minus z_ref equals 5.0.
• beta times 5.0 equals 0.5, and sigmoid(0.5) is about 0.622.
• The value is 0.622, so the loss for this example is 1 minus 0.622, about 0.378.

For a desirable answer with r equal to 1.0:

• r minus z_ref equals 0.
• sigmoid(0) is 0.5, so the loss is 0.5.

For an undesirable answer with r equal to minus 4.0:

• z_ref minus r equals 5.0.
• sigmoid(0.5) is about 0.622, so the loss is about 0.378.

For an undesirable answer with r equal to plus 3.0 (the model likes it more than before):

• z_ref minus r equals minus 2.0.
• beta times that equals minus 0.2, and sigmoid(minus 0.2) is about 0.450.
• The loss is about 0.550, and the gradient pushes r down.

These numbers show the pattern. Good answers lower the loss when r goes up. Bad answers lower the loss when r goes down. The sigmoid shape means the effect fades as the margin grows, which gives diminishing sensitivity.

Effect of the weights

Suppose your data has 9,000 good answers and 1,000 bad ones. If both lambdas are one, bad answers contribute only a tenth of the total signal. Setting lambda_U to 9 and lambda_D to 1 equalizes the total weight. If you also want to emphasize avoiding bad outputs, raise lambda_U further. The right level depends on how costly bad outputs are.

Code sketch

The sketch below shows a minimal loss in PyTorch. Model helpers are placeholders.

import torch

def kto_loss(policy, reference, batch, beta=0.1, lam_d=1.0, lam_u=1.0):
    # log ratios for the real (prompt, answer) pairs
    pi = sequence_logprob(policy, batch.prompt, batch.answer)
    with torch.no_grad():
        ref = sequence_logprob(reference, batch.prompt, batch.answer)
    r = pi - ref                                   # implicit reward

    # reference point from mismatched pairs (answers shifted by one position)
    shifted = torch.roll(batch.answer, shifts=1, dims=0)
    pi_mis = sequence_logprob(policy, batch.prompt, shifted).detach()
    with torch.no_grad():
        ref_mis = sequence_logprob(reference, batch.prompt, shifted)
    z_ref = torch.clamp((pi_mis - ref_mis).mean(), min=0.0)

    good = batch.label == 1
    value_good = lam_d * torch.sigmoid(beta * (r - z_ref))
    value_bad = lam_u * torch.sigmoid(beta * (z_ref - r))
    lam = torch.where(good, torch.tensor(lam_d), torch.tensor(lam_u))
    value = torch.where(good, value_good, value_bad)
    return (lam - value).mean()

Data format

Each line holds one prompt, one answer and one label:

{"prompt": "Write a query for monthly order totals.",
 "completion": "SELECT date_trunc('month', ordered_at), SUM(total) FROM orders GROUP BY 1;",
 "label": true}
{"prompt": "Write a query for monthly order totals.",
 "completion": "SELECT * FROM orders;",
 "label": false}

Typical configuration

method: kto
base_model: example.internal/sft-checkpoint
reference_model: example.internal/sft-checkpoint
beta: 0.1
desirable_weight: 1.0
undesirable_weight: 1.0
learning_rate: 5.0e-7
epochs: 1
batch_size: 64
max_length: 1024
adapter: lora         # optional

The batch size matters more than in DPO, because the reference point is estimated from the batch. Very small batches give noisy estimates.

Converting paired data to KTO format

If you have DPO-style pairs, you can use them for KTO by splitting each pair into two examples: the chosen answer with a good label and the rejected answer with a bad label. You lose the explicit pairing but gain a simple format. In the other direction, KTO data cannot always be turned into pairs, because good and bad answers may be for different prompts.

Debugging a KTO run

Variants and related ideas

• Paired KTO. Split each preference pair into a good and a bad example and train with KTO. This gives a direct comparison with DPO on the same data.
• KTO with an SFT term. A small cross-entropy term on good answers keeps their probability from falling.
• Iterative KTO. Sample answers from the current model, label them with a checker or reviewers and train again.
• Source-aware weights. Examples from reliable checkers can get higher weight than examples from noisy clicks.
• Reference-free ideas. Related single-stage methods such as ORPO remove the reference model.

Strengths and limitations

Strengths

• Works with binary data. No need to build pairs.
• Matches real feedback. Clicks, tests and acceptances map to labels directly.
• Handles imbalance. Weights let you use skewed data without discarding examples.
• Models human biases. Loss aversion and reference-dependence mirror how people judge outputs.
• Simple stack. No reward model, critic or RL loop.
• Works with adapters. Fits PEFT setups.
• Competitive. In the original report, it matched or beat DPO at several model scales, and it can work even when data is heavily skewed.

Limitations

• Noisy binary labels. A thumbs-down does not say why an answer was bad.
• No direct comparison. Without a paired alternative, the model must infer what better looks like.
• Batch-dependent reference point. Small or unrepresentative batches give a noisy estimate.
• Two extra hyperparameters. The lambda weights add tuning work.
• Offline. It cannot explore beyond stored answers.
• Reference model dependence. Quality is bounded by the starting point.
• Biased feedback. Click data reflects who clicks and what they like, which may not match your goals.
• Length and style effects. Like other methods, it can learn superficial correlates of the label.

How it compares with alternatives

When KTO is a good choice

• Your feedback arrives as binary signals.
• Your data is imbalanced, and discarding examples to build pairs would waste data.
• You want to use production feedback, such as user ratings, with little extra labeling.
• You have objective pass or fail checks, as in code or query tasks.

When another method is better

• You have clean paired comparisons and want to use their relative information: DPO or IPO.
• You need to save memory by skipping the reference model: ORPO.
• You need a reusable reward model or online exploration: RLHF.
• You want to encode explicit principles with AI feedback: see Constitutional AI.

Role in tabular foundation models

KTO is a language-model alignment method. It is not how models that learn from raw tables are pre-trained. Those models, including in-context tabular learners trained on very large numbers of synthetic or real tables, use a prediction loss on masked or held-out cells. The loss is exact, so there is no need for human good or bad labels. For background, see /model/research and /model/architecture.

Where KTO does not apply

A tabular model reads rows and columns of mixed numeric, categorical and date types, handles missing entries and often works without column names. It predicts targets for new rows by looking at labeled context rows. Accuracy and error are the guides. TabPFN shows this style of model, and boosted trees such as XGBoost, LightGBM and CatBoost optimize fixed losses. None of these uses KTO.

Where KTO fits very well

Table-related language assistants often collect exactly the kind of feedback KTO is built for.

• Text-to-SQL. A generated query either returns the expected result or fails. Each generated query becomes one labeled example, with no need to build pairs.
• Column semantic or sector labeling. A language-based labeler proposes a meaning for an unnamed column. A reviewer accepts or rejects it. Accept and reject are binary labels.
• Data-cleaning suggestions. Users apply or dismiss suggested fixes for missing or inconsistent values. Applied means good, dismissed means bad.
• Anomaly alert explanations. Analysts mark explanations as useful or not useful.
• Chart or report suggestions. Users keep or discard generated summaries.

Why imbalance is typical here

In table assistants, most suggestions may be accepted or most may be rejected, depending on the maturity of the system. KTO's weights let you train on all of it without throwing away examples.

Conceptual parallels

• Reference point and anomaly scores. KTO judges an answer relative to a baseline drift. Anomaly detection judges a row relative to what the fitted model expects. Both depend on a good baseline.
• Loss aversion and cost-sensitive learning. Tabular practitioners often weight false negatives more than false positives. KTO's lambda weights are the same idea applied to language outputs.
• Reference model and continual learning. Keeping the tuned model near a reference resembles methods that limit forgetting when learning new tasks.
• Binary labels as weak supervision. Accept or reject signals in table tools resemble weak labels used in tabular pipelines. Their noise must be handled.
• In-context learning contrast. An in-context tabular model adapts to a table without weight updates. KTO changes weights offline and shapes general behavior.

Mapping table tasks to KTO

For how the numeric core is evaluated, see /model/benchmarks.

A recipe for a table assistant

1. Log each assistant answer with the prompt, the context identifier and a placeholder label.
2. Capture binary signals such as execution success and user accept or dismiss.
3. Remove answers with unclear labels, such as ones the user never saw.
4. Count good and bad examples and set the lambda weights to balance them.
5. Train with KTO starting from the SFT model.
6. Compare with the SFT model on a held-out set of fresh tasks.
7. Repeat as new feedback accumulates.

Practical guidance and best practices

Data

• Make sure labels are about the answer, not about the prompt or the user's mood.
• Remove duplicates and contradictory labels for the same prompt and answer.
• Prefer feedback closely tied to quality, such as test results, over indirect proxies.
• Use the same chat template as serving.
• Keep a held-out set of labeled examples and a set of prompts for generation.
• Consider a mixture of human and automatic labels, tracking source.

Hyperparameters

• Start with beta around 0.1.
• Set the weights so that weighted desirable to undesirable counts are near one, then try a modestly higher weight on undesirable answers.
• Use a small learning rate and one or two epochs.
• Use a reasonably large batch size so the reference point is stable.
• With adapters, test slightly higher learning rates while watching stability.

Pipeline

• Run supervised fine-tuning first. KTO is less effective on a model that cannot yet produce the right format.
• Use the SFT model as the reference.
• Cache reference log probabilities where possible.
• Shuffle thoroughly so each batch has a mix of prompts and labels.
• Try iterative rounds: gather feedback on the tuned model's answers and train again.

Evaluation

• Measure win rates against the SFT model with pairwise judge model or human comparison.
• Track execution accuracy or acceptance rate for task-specific work.
• Check capability benchmarks for regressions.
• Test safety behavior in both directions.
• Review sampled outputs by hand.

A short pre-flight checklist

• The SFT model produces the expected format.
• The reference model equals the starting model, with the same tokenizer and chat template.
• Labels are audited: a random sample of good and bad examples has been read by a person.
• The class counts and weights are written down.
• The batch size is large enough for a stable reference point.
• A held-out labeled set and a held-out prompt set exist.
• Logs show loss, implicit rewards for each class, the reference point, length and sample text.
• Rollback checkpoints are stored.
• A baseline comparison, such as DPO on the same data, is planned.

Where binary labels come from in practice

Mixing sources is common. Keep a column in the data for the source so you can analyze each one's effect and apply different weights.

Common mistakes

• Treating missing feedback as negative. A user who did not click is not a thumbs-down.
• Ignoring imbalance. Leaving both weights at one when the data is heavily skewed lets the majority class dominate.
• Tiny batches. The reference point becomes noisy and training unstable.
• Skipping SFT. The model needs to produce sensible answers first.
• Mislabeled data. Binary labels are easy to get wrong. Audit a sample.
• Mixing prompt templates. The log ratios depend on formatting, so use the serving template.
• Over-weighting bad answers. Very large lambda_U can make the model overly cautious.
• Using the wrong reference. The reference must be the starting model.
• Training too long. Repeated passes lead to memorization.
• Reading a falling loss as success. Check samples and held-out results.

Frequently asked questions

What does KTO stand for?

KTO stands for Kahneman-Tversky optimization, named after the two researchers who developed prospect theory. It is a method for aligning language models that uses a loss shaped by how people value gains and losses, and it works with simple good or bad labels on single answers.

How is KTO different from DPO?

DPO needs pairs of chosen and rejected answers for the same prompt and optimizes the margin between them. KTO needs only single answers with a binary label, compares each answer with a reference point estimated from the batch, and uses weights to handle imbalance and loss aversion.

What is prospect theory?

Prospect theory is a model from behavioral economics describing how people judge risky outcomes. It says that people evaluate outcomes relative to a reference point, feel losses more strongly than equal gains and show diminishing sensitivity as amounts grow. KTO builds these ideas into its training loss.

Do I need a reference model for KTO?

Yes. KTO computes log ratios between the tuned model and a frozen reference, usually the supervised fine-tuned starting model. The reference acts as an anchor so the tuned model does not drift far, and its log probabilities can often be cached to save compute.

What if my data is mostly positive or mostly negative?

KTO has two weights, one for desirable and one for undesirable examples, so you can rebalance the contribution of each class. Setting the weights so that the total weighted signal from each class is roughly equal lets you keep all your data instead of discarding the majority class.

Can I use KTO with LoRA?

Yes. Adapter methods work with KTO in the same way as with DPO. They reduce memory use, and the reference log probabilities can be computed with the adapters switched off, which avoids keeping a second full model in memory.

Is KTO better than DPO?

It depends on the data. KTO is attractive when you have unpaired binary feedback, and reports show it can match DPO even with less informative data. When you have high-quality paired comparisons, DPO or IPO may extract more from each example, so testing both on your validation set is wise.

How much data does KTO need?

There is no fixed number. The quantity depends on model size, task breadth and label quality. Because binary labels are cheap to collect, many practitioners gather larger sets than they would with paired data, but cleaner labels and good coverage of prompts usually matter more than raw volume.

Is KTO used for tabular foundation models?

Not for their core pre-training, which uses a direct prediction objective on tables. KTO is useful for language assistants around tabular systems, such as text-to-SQL helpers, semantic labelers and data-cleaning suggestion tools, where accept or reject signals are common and easy to log.

Does KTO remove the need for human feedback?

No. It reduces the format burden, since you only need good or bad labels, but those labels still come from people, tests or other checkers. Automatic checks can supply many labels cheaply, and humans are still needed for subjective qualities such as tone and clarity.

Glossary

• Alignment: making model behavior match human intent.
• Binary feedback: a label with two values such as good or bad.
• Class imbalance: unequal numbers of examples in each class.
• Desirable answer: an answer labeled good.
• Diminishing sensitivity: the idea that each extra unit of gain matters less.
• Implicit reward: the log ratio between the tuned and reference probabilities.
• Lambda weights: weights for desirable and undesirable examples.
• Loss aversion: the tendency for losses to weigh more than equal gains.
• Offline training: learning from stored data.
• Prospect theory: a theory of how people value gains and losses.
• Reference model: a frozen copy of the starting model.
• Reference point: the baseline against which outcomes are judged.
• SFT: supervised fine-tuning on demonstrations.
• Undesirable answer: an answer labeled bad.
• Value function: a curve mapping outcomes to felt value.

Summary

KTO aligns a language model using single answers labeled good or bad, with a loss inspired by prospect theory. It judges each answer against a reference point, weights bad and good examples separately and keeps the model near a frozen reference. This makes it a strong fit for real-world binary feedback such as clicks, test results and accept or reject signals, and for imbalanced data. It still needs a decent SFT model, careful label hygiene and batch sizes large enough for a stable reference estimate. For tabular foundation models, KTO is not part of pre-training; it is a tool for the language layer around them, including text-to-SQL, semantic labeling and data-cleaning assistants. Pair this with DPO, IPO and ORPO when choosing a method, and see /model/benchmarks for how the numeric side is evaluated.

Related

It belongs to Alignment in my stack, under LLMs, retrieval, and evaluation. Others in the same group: