RLHF

Overview

RLHF, short for reinforcement learning from human feedback, is a training method that teaches a language model to produce answers people prefer. Humans compare pairs of answers, a reward model learns to predict those choices, and the language model is then tuned with reinforcement learning to score higher on that reward. The result is a model that follows instructions better, refuses unsafe requests more reliably and writes in a style people find helpful.

The method sits after pre-training and supervised fine-tuning in the usual recipe for assistants. It is powerful but complex, because it needs several models and a delicate optimization loop. Simpler methods such as DPO, IPO, KTO and ORPO try to get most of the benefit without the reinforcement learning stage. This article explains how RLHF works, why it was introduced, where it falls short, and how its ideas relate to models that learn from raw tables.

What it is

RLHF is a three-part pipeline for aligning a language model with human judgment. "Alignment" here means making the model's behavior match what people actually want, which is hard to write down as a formula.

Think of teaching someone to cook. You cannot give them an equation for "tasty". But you can taste two dishes and say which one you prefer. After many such tastings, the cook learns what you like. RLHF follows the same idea: it replaces an impossible-to-write objective with many simple human comparisons.

The three stages in plain terms

• Stage one is supervised fine-tuning, often abbreviated SFT. The base model is trained on example conversations written by people so that it learns the basic format of following instructions.
• Stage two is reward modeling. People rank several model answers to the same prompt. A separate model learns to give higher scores to the answers people ranked higher.
• Stage three is reinforcement learning. The language model generates answers, the reward model scores them, and an algorithm such as PPO nudges the model toward higher scores while keeping it close to its starting point.

What problem it solves

Pre-trained language models are trained to predict the next token. That goal makes them good at continuing text, but not necessarily at being helpful, honest or safe. A model that predicts the next token well may happily continue a harmful request or ramble. RLHF changes the objective from "what text is likely" to "what answer would a person prefer".

Where the feedback comes from

Feedback is usually collected from human annotators following written guidelines. Annotators see a prompt and two or more candidate answers and pick the best one. Some pipelines also collect scalar ratings, edits or free-text critiques. A related approach, RLAIF, replaces some of the human labels with labels from another model, and Constitutional AI uses written principles to guide those labels.

Where it came from

The roots of RLHF lie in reinforcement learning research that tried to learn behavior from human judgments instead of hand-coded reward functions. A well-known 2017 paper, often cited as "Deep reinforcement learning from human preferences", showed that agents could learn complex tasks from a small number of human comparisons between short video clips of behavior.

The idea moved to language around 2019 and 2020. Early work fine-tuned language models with human preferences on tasks such as stylistic text continuation. A 2020 paper on learning to summarize from human feedback showed that a summarizer trained with a learned reward model beat much larger models trained only on supervised data.

In 2022 the InstructGPT paper, "Training language models to follow instructions with human feedback", applied the full three-stage recipe to a general assistant. It reported that people preferred outputs from a much smaller RLHF-tuned model over outputs from a far larger model without RLHF. This result made the recipe standard for chat assistants.

Supporting ideas

• The Bradley-Terry model from 1952 gives the statistical basis for turning pairwise comparisons into scores.
• Proximal policy optimization, introduced in 2017, became the default reinforcement learning algorithm for the final stage.
• KL divergence regularization, a standard tool in probability, keeps the tuned model close to its starting model.

What came after

By 2023 researchers noticed that the reinforcement learning stage was costly and sometimes unstable. Direct preference optimization, published in 2023, showed that the same preference data could be used with a simple classification-style loss. Later work such as IPO, KTO and ORPO explored variations. RLHF remains the reference method against which these simpler methods are measured, and it is still used where an explicit reward model is valuable.

How it works

This section walks through each stage with an everyday analogy.

Step 1: start with a supervised model

You begin with a pre-trained model and fine-tune it on high-quality demonstrations: prompts paired with good answers. This is like giving a new employee a handbook of worked examples. The model learns the basic style and format. This model is called the SFT model, and it becomes both the starting point and the reference for later stages.

Step 2: collect preference data

For each of many prompts, the SFT model generates several answers, typically by sampling with some randomness. Annotators then rank or pick between them. The data set is a collection of triples: a prompt, a chosen answer and a rejected answer.

An analogy is a talent show. The judges do not score each act out of a hundred with perfect consistency. They simply say which of two acts they liked more. Comparisons are easier and more consistent for humans than absolute scores.

Step 3: train a reward model

The reward model is usually a copy of the language model with its output layer replaced by a single number. It reads a prompt and an answer and outputs a score. It is trained so that the chosen answer gets a higher score than the rejected one.

The standard loss comes from the Bradley-Terry model. The probability that answer A beats answer B is the logistic function of the score difference:

P(A preferred over B) = sigmoid( r(prompt, A) - r(prompt, B) )
loss = - log sigmoid( r(prompt, chosen) - r(prompt, rejected) )

Only differences in scores matter, so the absolute scale is arbitrary. Training minimizes the loss over all comparison pairs. The reward model becomes a stand-in for the judges.

Step 4: optimize the policy with reinforcement learning

In reinforcement learning language, the language model is the "policy". Given a prompt, it generates an answer token by token. The reward model scores the finished answer. The policy is updated to make high-scoring answers more likely.

Pure reward maximization would be dangerous. The policy could find strange outputs that fool the reward model but that no person would like. To prevent this, a penalty is added for drifting away from the reference model:

objective = E[ r(prompt, answer) ] - beta * KL( policy || reference )

The coefficient beta controls how tightly the policy is held to the reference. Think of a kite on a string. The kite wants to climb, which is the reward, but the string, which is the KL penalty, stops it from flying off to a place where it would be lost.

Step 5: use PPO machinery

The usual algorithm is PPO. It needs several models in memory at once:

• the policy being trained
• a frozen reference copy of the starting model
• the reward model
• a value model (a critic) that estimates expected future reward and reduces variance

PPO clips each update so that the policy cannot change too much in one step. This keeps training stable. The reward is often assigned at the end of the answer, with the KL penalty applied at every token, and the critic helps spread credit across tokens.

Step 6: iterate and evaluate

Many teams repeat the loop. New answers from the improved policy are collected, humans compare them, the reward model is refreshed and the policy is tuned again. Evaluation uses held-out preference comparisons, win rates against a baseline, safety test suites and sometimes a judge model, as in LLM as judge setups.

A mental picture of the whole loop

prompt -> policy writes answer -> reward model scores it
   ^                                         |
   |        PPO update (with KL penalty)     |
   +-----------------------------------------+

Variants and extensions of the basic recipe

The basic recipe has many practical variations. Knowing them helps when you read papers or design a pipeline.

• Rejection sampling fine-tuning. Generate many answers, keep the ones the reward model likes best, and run ordinary supervised training on them. It is simple and often a strong baseline.
• Best-of-n at inference. Generate n answers at serving time and return the one with the highest reward. No weight update is needed, but each request costs n times more compute.
• Process rewards. Instead of scoring only the final answer, a process reward model scores each reasoning step. This is common for math and code, where steps can be checked.
• Outcome rewards from verifiers. When answers can be checked automatically, such as a unit test passing or a query returning the right rows, the reward can come from the verifier rather than from humans.
• Multi-objective rewards. Helpfulness, harmlessness and honesty can have separate reward heads. Their scores are combined with weights, so a team can tune the trade-off.
• Iterative or online preference collection. New answers from the current policy are labeled each round, which keeps the reward model relevant as the policy changes.
• Reward model ensembles. Several reward models vote or the minimum score is used, which makes exploitation harder.

How feedback is collected in practice

Preference data can take several forms, and the form affects which training method fits.

A ranking of four answers yields six pairs, so rankings are an efficient way to gather more comparisons per prompt. The pairs from one ranking are correlated, so they should stay together when you split data into training and test sets.

Safety and values considerations

RLHF encodes the preferences of the people who labeled the data. Those preferences reflect the guidelines, the annotator pool and the prompts used. This has several consequences.

• Guidelines should be explicit about what counts as harmful, what counts as helpful and how to handle ambiguity.
• A diverse annotator pool lowers the chance that a narrow viewpoint becomes the model's default.
• Refusals should be reserved for real risks. A reward model that always favors refusal produces a timid assistant.
• Red-team prompts, which are written to provoke bad behavior, should be part of both training and evaluation.
• Transparency about how feedback was gathered helps users judge the model's behavior.

Key concepts and terms

Reward hacking in more detail

A reward model is only a proxy for what people want. If the policy is optimized hard, it may find answers that score high for the wrong reasons. Typical symptoms are overly long answers, flattery, repeated confident phrases or formatting tricks. This is a case of a general rule often called Goodhart's law: when a measure becomes a target, it stops being a good measure.

Over-optimization

Studies of reward model scaling show that as the policy drifts further from the reference, the proxy reward keeps rising while the true quality, measured by fresh human judgments, first rises and then falls. The KL penalty, early stopping and larger reward models all help.

A worked example

This example shows a tiny numeric reward model step, then a minimal code sketch of the pipeline using generic placeholder names.

Numeric walk-through of the reward loss

Suppose the reward model gives these scores for one prompt:

• chosen answer: 2.0
• rejected answer: 0.5

The score difference is 1.5. The model's predicted probability that the chosen answer wins is sigmoid(1.5), about 0.82. The loss is negative log of 0.82, about 0.20. This is a small loss, so the model is already doing well on this pair.

Now suppose the model had scored them the other way, chosen 0.5 and rejected 2.0. The difference is minus 1.5, the probability is about 0.18 and the loss is about 1.70. The gradient would push the scores to swap.

Numeric walk-through of the KL-regularized reward

Imagine an answer with a reward model score of 3.0. Over its tokens, the policy assigns much higher log probability than the reference, giving a summed log-ratio of 4.0. With beta equal to 0.1, the penalty is 0.4, so the effective reward is 2.6. If a different answer scores 3.5 but drifts further with a log-ratio sum of 12.0, the penalty is 1.2 and the effective reward is 2.3. The second answer scores higher on the raw reward but loses once the penalty is applied. This is how the KL term discourages odd, over-optimized outputs.

Code sketch with a reward model

The following sketch uses a generic training loop. Names such as load_model and preference_batches are placeholders, not a specific library.

import torch
import torch.nn.functional as F

reward_model = load_model("example.internal/sft-base", head="scalar")
optimizer = torch.optim.AdamW(reward_model.parameters(), lr=1e-5)

for batch in preference_batches("example.internal/prefs.jsonl"):
    r_chosen = reward_model(batch.prompt, batch.chosen)      # shape [B]
    r_rejected = reward_model(batch.prompt, batch.rejected)  # shape [B]
    loss = -F.logsigmoid(r_chosen - r_rejected).mean()
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Code sketch for the policy update

policy = load_model("example.internal/sft-base")
reference = load_model("example.internal/sft-base").eval()
critic = load_model("example.internal/sft-base", head="scalar")
beta = 0.05

for prompts in prompt_batches():
    answers, logp_policy = policy.generate(prompts, return_logprobs=True)
    with torch.no_grad():
        logp_ref = reference.logprobs(prompts, answers)
        score = reward_model(prompts, answers)
    kl = logp_policy - logp_ref                  # per-token log ratio
    reward = score - beta * kl.sum(dim=-1)       # sequence-level shaped reward
    advantage = reward - critic(prompts, answers).detach()
    ppo_step(policy, critic, advantage, clip=0.2)

Reading the signals during training

• Mean reward should rise steadily at first.
• KL to the reference should rise slowly and stay in a bounded range.
• Answer length should be watched. A sudden length increase often signals reward hacking.
• Held-out human or judge win rates are the real test. If they stall while reward climbs, stop.

A second worked example: best-of-n versus PPO

Suppose you want a quick estimate of how much a reward model can help before building a full RL loop. Take 200 prompts and sample 8 answers each from the SFT model. Score all 1,600 answers with the reward model. For every prompt, keep the top answer.

Now ask annotators to compare the kept answers against a single random answer from the same prompt. If annotators prefer the kept answers in a clear majority of cases, the reward model carries real signal. If the preference is near half, the reward model is not tracking what people want, and PPO will not fix that.

This test is cheap and informative. It also gives a ceiling: PPO usually matches or beats best-of-n at the same reward, because it moves probability mass toward good answers instead of only picking among samples.

Compute and scale notes

• The reward model is often smaller than or equal in size to the policy. Larger reward models resist exploitation better.
• Generation dominates the cost of the RL stage, because every update needs fresh samples.
• Memory use can be reduced by offloading the reference and reward models, quantizing them, or using adapters for the policy.
• Distributed training splits generation and optimization across machines. Careful batching of prompts by length avoids idle hardware.
• Reproducibility is hard. Random sampling, clipping and advantage normalization all change results, so log seeds and settings.

Reward model failure modes to test for

Strengths and limitations

Strengths

• It optimizes what people actually prefer, not just likelihood of text.
• It handles goals that are hard to specify, such as tone, helpfulness and harmlessness.
• A learned reward model can be reused to rank samples, filter data or run best-of-n selection.
• It can generalize preferences to new prompts that were never labeled.
• It supports online learning, because fresh samples from the current policy are scored each round.

Limitations

• It is complex. Four models and many hyperparameters must work together.
• It is memory hungry. Policy, reference, reward model and critic all sit in memory during training.
• It can be unstable. Small changes in learning rate, beta or batch size can change results a lot.
• The reward model is a proxy and can be exploited.
• Human labels are costly, noisy and subjective. Annotators disagree, and guidelines shape the outcome.
• It may reduce output diversity, because the policy collapses toward high-reward modes.
• It can encourage sycophancy, where the model agrees with the user because agreement was often rewarded.
• It can raise calibration problems, with the model sounding more confident than its accuracy justifies.

Costs at a glance

How it compares with alternatives

When RLHF is still a good choice

• You need a reusable reward signal for ranking or filtering.
• You have verifiable or programmatic rewards that can be mixed with preference rewards.
• You want online exploration, where the model learns from its own fresh outputs.
• You operate at a scale where the engineering cost is acceptable.

When a simpler method is better

• You have a fixed set of preference pairs and limited compute.
• You want a stable, easy-to-debug training run.
• You are tuning a smaller model with adapters such as LoRA or QLoRA, where memory is tight. See also PEFT.

Role in tabular foundation models

RLHF is a language-model technique. Models that learn directly from raw tables, such as in-context tabular learners pre-trained on many synthetic or real tables, are usually not trained with human preference feedback. Their training signal is a prediction loss on masked or held-out cells, which is already a clear objective. Still, RLHF and its relatives touch the tabular world in several concrete ways.

Where it does not apply

A model that sees a table with rows, columns of mixed types and missing values, and predicts a target for new rows, has a direct score: accuracy, error or likelihood. There is nothing subjective to align. Systems like TabPFN learn to perform in-context learning by being pre-trained on a very large number of generated tables, with no human preference step. Gradient-boosted trees such as XGBoost, LightGBM and CatBoost also optimize a fixed loss. For background on the pre-training approach, see /model/research and /model/architecture.

Where it can apply

• Language interfaces to tables. Assistants that answer questions about a table, write SQL or explain a dataset are language models. Their style, clarity and safety can be tuned with RLHF or a simpler preference method.
• Text-to-SQL and table question answering. Preference pairs can compare a correct query with an incorrect one. A reward can also come from execution: run the query and check the result. This blends human preference with verifiable feedback.
• Explanations of predictions. When a tabular model's output is explained in natural language, people can rank explanations by clarity and faithfulness. A preference-tuned explainer follows those rankings.
• Data cleaning and imputation agents. An agent that decides how to fill missing values or flag anomalies may be rated by experts. Their choices can feed a reward model for agent behavior.
• Semantic and sector classification. When columns lack names, a language layer may propose a semantic type, such as "postal code" or "invoice date". Experts accept or reject proposals, and those decisions are preference-style data.

Conceptual links

• Feedback loops resemble continual learning. RLHF rounds repeat as new human feedback arrives. Continual learning in tabular models also updates as new tables arrive, with the same risk of forgetting. The KL penalty is similar in spirit to methods that keep new weights close to old ones.
• Reward models resemble anomaly scorers. Both turn a complex object into a single score that ranks items. An anomaly detector ranks rows by how unusual they are, while a reward model ranks answers by how preferred they are.
• Mixed signals. Tabular tasks often have exact metrics. A hybrid reward that combines a metric with a human preference term is a practical pattern.

A concrete scenario

Imagine an assistant that sits on top of a data warehouse. A user asks, "Which customer segments are most likely to churn, and why?" The system has two parts. A tabular predictor reads raw rows with mixed numeric and categorical columns, handles missing values and returns scores. A language model then explains the result and may write supporting queries.

Preference tuning applies to the second part. Annotators rank explanations for accuracy, clarity and caution about causal claims. A reward can also check that the explanation matches the numbers from the predictor. The first part needs no human ranking, because its quality is measured with held-out data.

Mapping RLHF ideas to table tasks

Honest summary of the relationship

RLHF is not how tabular foundation models are pre-trained. It is a tool for the language layer around them. If you build a conversational front end to a tabular model, preference tuning can improve how it talks. The numeric core stays governed by its prediction objective. See /model/benchmarks for how tabular models are evaluated by metrics rather than preferences.

Practical guidance and best practices

Data

• Write clear annotation guidelines and test them on a small pilot.
• Measure agreement between annotators. Low agreement means the task or guidelines need work.
• Mix prompts across domains and difficulty so the reward model does not overfit to one style.
• Include safety-relevant and adversarial prompts, not only friendly ones.
• Keep a held-out set of comparisons that never enters training.
• Watch for length bias. Humans often prefer longer answers; normalize or balance lengths.

Reward model

• Start from a strong base, ideally the SFT model or a similar-sized one.
• Train for one or two epochs. Reward models overfit quickly.
• Check accuracy on held-out pairs. Values far above annotator agreement suggest leakage.
• Examine the largest score gaps and the most confusing pairs by hand.
• Consider an ensemble of reward models to reduce exploitation.

Policy optimization

• Begin with a small beta and a conservative learning rate; adjust based on KL behavior.
• Log reward, KL, length and sample outputs every step.
• Use best-of-n sampling as a cheap baseline before committing to PPO.
• Keep a fixed prompt set for qualitative review across checkpoints.
• Stop when held-out judgments stop improving, not when the proxy reward stops rising.

Evaluation

• Use pairwise human evaluation for the final decision.
• Use automatic checks, including a judge model, for fast iteration, but calibrate them against humans.
• Test capabilities that you do not want to lose, such as reasoning or coding, to catch alignment tax.
• Run safety and refusal tests in both directions: harmful requests that should be refused and benign requests that should not.

Efficiency

• Parameter-efficient methods such as LoRA reduce memory for the policy.
• Sharing weights between the policy and the critic can help in small setups.
• Offline methods like DPO are a good first experiment before investing in a full RL loop.

A short pre-flight checklist

Before launching a full RLHF run, confirm each of the following.

• The SFT model already follows the format you want.
• The preference data has been audited for duplicates, leakage and obvious label errors.
• The reward model beats a trivial baseline, such as always choosing the longer answer, on held-out pairs.
• You have a baseline to beat, for example best-of-n or a DPO run on the same data.
• Logging covers reward, KL, length, entropy and sample text.
• A human review of outputs is scheduled at fixed intervals.
• A rollback checkpoint exists.
• Someone owns the decision of when to stop, with written criteria.
• Annotator guidelines are versioned, so later changes can be traced.
• The evaluation set covers the capabilities that must not regress.

Common mistakes

• Skipping the SFT stage. Starting RL from a raw pre-trained model usually produces unstable training.
• Using a weak reward model. A noisy judge gives a noisy teacher. Invest in reward model quality first.
• Ignoring the KL term. Setting beta too low leads to reward hacking; too high prevents learning.
• Training the reward model too long. Overfit rewards generalize poorly to new policy outputs.
• Mismatched distributions. The reward model was trained on outputs from the SFT model, but the policy drifts. Refresh the reward data periodically.
• Trusting the proxy. A rising reward curve is not proof of improvement.
• Ignoring length. Longer is not better. Track length and penalize padding.
• Weak annotation guidelines. Vague instructions give inconsistent labels.
• One-sided safety tuning. Training only to refuse makes a model overcautious.
• No rollback plan. Save checkpoints often and compare against the SFT baseline.
• Evaluating on training prompts. Always use held-out prompts.

Frequently asked questions

What does RLHF stand for and what does it do?

RLHF stands for reinforcement learning from human feedback. It trains a language model to produce answers that people prefer by first learning a reward model from human comparisons and then using reinforcement learning to push the model toward answers that earn higher rewards while staying close to its starting behavior.

Why not just use supervised fine-tuning?

Supervised fine-tuning teaches a model to imitate example answers, which is useful but limited by the quality and coverage of those examples. Preferences let the model learn from comparisons between its own outputs, including cases where no single perfect answer exists. This often improves helpfulness, tone and safety beyond what imitation alone achieves.

What is a reward model?

A reward model is a neural network that reads a prompt and an answer and returns a single score that predicts how much people would like that answer. It is trained on pairs of answers where humans picked a winner, using a loss based on the Bradley-Terry model so that preferred answers receive higher scores.

Why is there a KL penalty?

The KL penalty measures how far the tuned model has moved from the reference model and subtracts a multiple of that distance from the reward. Without it, the model could exploit weaknesses in the reward model and produce odd text that scores well but that nobody likes, so the penalty acts as a safety tether.

Is PPO required for RLHF?

No. PPO is the most common algorithm, but other policy-gradient methods and even simple approaches such as best-of-n sampling or rejection sampling fine-tuning can use a reward model. The core of RLHF is learning from human preference signals, not any one optimizer.

How is RLHF different from DPO?

RLHF trains a separate reward model and then runs a reinforcement learning loop with online sampling, while DPO uses the same kind of preference pairs but optimizes the language model directly with a classification-style loss. DPO is simpler and more stable, whereas RLHF offers a reusable reward model and online exploration.

How much human data does it need?

The amount varies widely with model size, task breadth and label quality. Published systems have used tens of thousands of comparisons or more, but careful guidelines and diverse prompts often matter more than raw volume, and some pipelines add model-generated labels to extend human data.

Can RLHF make a model less accurate?

It can. Optimizing for human approval may reward confident, fluent or agreeable answers over strictly correct ones, and heavy tuning can reduce diversity or slightly harm some capabilities. Good evaluation, a KL penalty and mixed rewards that include factual checks help limit these effects.

Is RLHF used to train tabular foundation models?

Generally no. Tabular foundation models learn from a prediction loss on tables, often using synthetic or real data with masked cells, so they do not need human preferences. RLHF may still help a language interface around such a model, for example in explanations or text-to-SQL assistants.

What is reward hacking?

Reward hacking is when a model finds shortcuts that raise the reward model's score without actually improving quality, such as producing very long answers or flattering phrases. It happens because the reward model is only an imperfect proxy for human judgment, and it is managed with KL penalties, ensembles and fresh human evaluation.

Glossary

• Alignment: making a model's behavior match human intent and values.
• Annotator: a person who labels or ranks model outputs.
• Bradley-Terry model: a statistical model of pairwise comparisons based on latent scores.
• Best-of-n: sample n answers and keep the one with the highest reward.
• Critic: a model that estimates expected reward to reduce variance in policy updates.
• Goodhart's law: when a measure becomes a target, it loses value as a measure.
• KL divergence: a measure of difference between two probability distributions.
• Policy: the model that chooses actions, here the language model generating tokens.
• PPO: proximal policy optimization, an RL algorithm with clipped updates.
• Preference pair: a prompt with a chosen and a rejected answer.
• Reference model: the frozen model used to measure drift.
• Reward hacking: exploiting flaws in a reward model.
• Reward model: a model that scores answers to predict human preference.
• Rollout: a generated answer used during training.
• Sycophancy: telling users what they want to hear instead of what is accurate.
• SFT: supervised fine-tuning on demonstration data.

Summary

RLHF aligns a language model with human preferences in three steps: supervised fine-tuning, reward modeling from human comparisons and reinforcement learning with a penalty that keeps the model near its starting point. It was a key ingredient in making instruction-following assistants practical, and it remains the reference method for preference-based tuning. Its costs are complexity, memory and sensitivity to settings, which is why simpler methods such as DPO, IPO, KTO and ORPO are popular. For tabular foundation models, RLHF is not the pre-training method; it is a tool for the language layer around them, such as explanations, text-to-SQL and agent behavior, while the numeric core is judged by metrics shown in /model/benchmarks.

Related

It belongs to Alignment in my stack, under LLMs, retrieval, and evaluation. Others in the same group: