TRL

Overview

TRL is a Hugging Face library for training language models with supervised fine-tuning and reinforcement learning style methods. Its name comes from Transformer Reinforcement Learning. It provides ready-made trainers for supervised fine-tuning (SFT), reward modelling, proximal policy optimization (PPO), direct preference optimization (DPO) and related alignment methods. You supply a model, a dataset and a config. The library handles batching, loss computation, reference models, logging and saving.

The main value is that it turns research recipes for alignment into short scripts. A team that wants to teach a model to follow preferences does not need to write the loss functions or the generation loops by hand. TRL works with Transformers, Accelerate, Datasets and PEFT, so it fits the usual Hugging Face workflow and runs on one GPU or many. This article explains the main trainers with everyday analogies, shows code, lists strengths and limits, and ends with how the library relates to models that learn from raw tables, as discussed in /model/research.

What it is

TRL is a post-training toolkit. Pretraining teaches a language model to predict text. Post-training shapes how the model behaves: following instructions, answering helpfully, avoiding harmful content, matching a style. TRL covers the post-training stages that come after pretraining.

The stages TRL supports

Names and available trainers change between versions, so check the documentation of the version you install.

An analogy

Think of training a new employee. First, you show worked examples of good replies (SFT). Then you show pairs of replies and say which one is better (preference training). Or you build a scorecard and let the employee practice, getting a score each time (reward model and PPO). TRL is the training office that holds the forms and the procedures for each of these routes.

What it is not

TRL is not a model, not a dataset and not a serving system. It does not replace the evaluation work that tells you whether a tuned model is better. It also does not make a bad dataset good.

Where it came from

The library began as a way to run PPO on transformer language models, following early work on fine-tuning language models from human preferences. As the field moved, the library grew to include supervised fine-tuning helpers, reward model training and, after the direct preference optimization method appeared in 2023, a DPO trainer. Since then it has added trainers for related methods such as KTO, ORPO and SimPO style objectives, and support for online and group-based reinforcement learning methods.

Why it spread

• Open-weight models needed an open alignment toolkit.
• It rides on the Transformers ecosystem, so any supported model can be trained with it.
• It integrates with PEFT, so alignment can run on small hardware with LoRA or QLoRA.
• It supports distributed training through Accelerate, DeepSpeed ZeRO and FSDP.
• Many open alignment recipes and tutorials use it, so examples are easy to find.

Related lineage

The ideas behind the trainers come from reinforcement learning and from RLHF. PPO itself is described in PPO. AI-labeled variants are covered under RLAIF and Constitutional AI.

How it works

Each trainer follows a similar pattern: prepare data, build a loss, run the loop. The details differ by method.

SFT trainer

The SFT trainer takes a dataset of text or of chat messages and trains the model to predict the next token. It can apply a chat template, mask the prompt part so only the answer counts in the loss, and pack short examples together to reduce padding. This is the base step for almost every alignment pipeline. See SFT.

Analogy: a student copying model answers until the style becomes natural.

Reward trainer

The reward trainer takes pairs of answers, one chosen and one rejected, and trains a model to give the chosen one a higher score. The usual loss pushes up the gap between the two scores. The resulting reward model is a judge that can score new answers.

Analogy: a tasting panel learns what a good dish tastes like by comparing pairs of dishes.

PPO trainer

PPO is an online reinforcement learning loop. At each step:

1. Sample prompts.
2. The current policy generates answers.
3. The reward model scores them.
4. A penalty compares the policy with a frozen reference copy, so the policy does not drift too far.
5. A value estimate helps compute how much better than expected each action was.
6. The policy is updated with a clipped objective that limits step size.

Analogy: a cook tries a new recipe, gets a score from a judge, and adjusts a little. A rule says not to change the recipe so much that it no longer resembles the original dish.

PPO needs several models in memory: the policy, the reference, the reward model and a value model (or a value head). This is why it is heavy.

DPO trainer

DPO skips the reward model and the sampling loop. It takes triples of prompt, chosen answer and rejected answer, and directly increases the model's relative probability of the chosen answer over the rejected one, compared with a frozen reference model. A temperature-like parameter, often called beta, controls how strongly the model is pulled away from the reference.

Analogy: instead of building a scorecard, you show the student two answers and say "more like this one, less like that one", while reminding them to stay close to how they wrote before.

See DPO.

Other preference trainers

• KTO uses single examples labeled good or bad, which is easier to collect than pairs. See KTO.
• ORPO merges supervised fine-tuning and preference learning in one stage and does not need a reference model. See ORPO.
• SimPO uses a length-normalized reward and a margin, also without a reference model. See SimPO.

Online and group-based methods

Newer trainers in the library support online methods where the model samples its own answers and a reward function scores them, with group-based advantage estimates in place of a value model. Reward functions can be code: for example, a check that a math answer is correct or that a format is followed. Because the details evolve quickly, read the documentation of your installed version before relying on a specific trainer.

Data formats

Preference data usually has fields for prompt, chosen and rejected. Chat data has a list of messages with roles. The library can apply the tokenizer chat template to convert them to text. Mismatched templates are a common source of silent errors.

Reference models

Many preference methods compare the policy with a frozen reference. If you train with an adapter, the library can use the same base model with the adapter turned off as the reference, so no second copy is needed. This saves memory.

Logging and metrics

Trainers log metrics such as loss, reward margins, accuracy of preferring chosen over rejected, and for online methods the mean reward and the divergence from the reference. These metrics help spot problems early.

Key concepts and terms

KL penalty, in plain words

Imagine you adjust a recipe a little each week. A rule says you must not change it so much that regulars no longer recognize the dish. The KL penalty is that rule. Without it, the model may find odd outputs that please the reward model but read badly to people.

A worked example

The example below walks through a small pipeline: SFT, then DPO with a LoRA adapter. Model and path names are placeholders. Argument names vary by library version, so confirm against your installed version.

Step 1: Supervised fine-tuning

from datasets import load_dataset
from trl import SFTConfig, SFTTrainer
from peft import LoraConfig

data = load_dataset("json", data_files="sft.jsonl")["train"]

cfg = SFTConfig(
    output_dir="./outputs/sft",
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
    max_length=2048,
)

trainer = SFTTrainer(
    model="example-org/example-7b",
    args=cfg,
    train_dataset=data,
    peft_config=LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                           task_type="CAUSAL_LM"),
)
trainer.train()
trainer.save_model("./outputs/sft")

Here each record in sft.jsonl holds a list of messages, such as a user message and an assistant message. The trainer applies the chat template.

Step 2: Prepare preference data

Each record has three fields.

{"prompt": "Explain what a primary key is in a database.",
 "chosen": "A primary key is a column, or a set of columns, whose values uniquely identify each row in a table. ...",
 "rejected": "It is a key. Databases have keys."}

Pairs may come from human labels, from an AI judge as in RLAIF, or from a rule. Clean the data first: remove duplicates, empty answers and pairs that differ only by length.

Step 3: DPO training

from trl import DPOConfig, DPOTrainer
from peft import LoraConfig

prefs = load_dataset("json", data_files="prefs.jsonl")["train"]

cfg = DPOConfig(
    output_dir="./outputs/dpo",
    beta=0.1,
    learning_rate=5e-6,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    num_train_epochs=1,
    bf16=True,
    logging_steps=10,
)

trainer = DPOTrainer(
    model="./outputs/sft",
    args=cfg,
    train_dataset=prefs,
    peft_config=LoraConfig(r=16, lora_alpha=32, task_type="CAUSAL_LM"),
)
trainer.train()
trainer.save_model("./outputs/dpo")

Step 4: Read the metrics

During DPO training, watch:

• Reward accuracy: the share of pairs where the chosen answer has the higher implicit reward. It should rise from about 50 percent toward higher values.
• Reward margin: the average gap. It should grow slowly.
• Log probabilities of chosen answers: a steady fall may mean the model is drifting away from fluent text.

A numeric walk-through of the DPO idea

Suppose for one pair the reference model gives log probability -40 to the chosen answer and -38 to the rejected one. After some training, the policy gives -39 to chosen and -41 to rejected. The relative change for chosen is +1, and for rejected is -3. The implicit reward margin is beta times the difference, here beta times 4, which is 0.4 with beta 0.1. The loss shrinks as this margin grows, so the training nudges the policy to make chosen more likely and rejected less likely, compared with the reference.

A reward model example

from trl import RewardConfig, RewardTrainer
from transformers import AutoModelForSequenceClassification, AutoTokenizer

name = "example-org/example-1b"
tok = AutoTokenizer.from_pretrained(name)
rm = AutoModelForSequenceClassification.from_pretrained(name, num_labels=1)

cfg = RewardConfig(output_dir="./outputs/rm", per_device_train_batch_size=4,
                   num_train_epochs=1, learning_rate=1e-5)
trainer = RewardTrainer(model=rm, args=cfg, train_dataset=prefs,
                        processing_class=tok)
trainer.train()

The reward model outputs one number per answer. Check it on a held-out set of pairs: it should rank the chosen answer higher on most of them.

A reward-function example for online training

For tasks with a checkable answer, a reward can be a function.

def format_and_correct(completions, answers, **kw):
    scores = []
    for c, a in zip(completions, answers):
        ok_format = c.strip().endswith("</answer>")
        ok_value = a in c
        scores.append(1.0 * ok_value + 0.2 * ok_format)
    return scores

Such rule-based rewards are cheap and hard to game when the check is exact. They work for math, code with tests and structured output.

Launching on several GPUs

accelerate launch --config_file accel.yaml train_dpo.py

The config file chooses the strategy: plain data parallel, ZeRO or FSDP. See data parallelism and mixed precision.

A practical pipeline plan

A full alignment project with TRL can follow this plan. Each stage ends with a check before the next begins.

Stage 1: Define the target

Write down what better means. Examples: answers follow a given format, queries run without errors, refusals are polite. Without a target, there is nothing to measure.

Stage 2: Build the evaluation first

Create a held-out prompt set and a scoring method before training. Include automatic checks where possible, and a small human-reviewed sample. Save the baseline scores of the starting model.

Stage 3: Run SFT

Train on clean examples. Inspect the formatted text and the masks. Compare with the baseline. Stop here if SFT alone meets the target, since simpler is better.

Stage 4: Collect preferences

Sample two or more answers per prompt from the SFT model. Label them with people, an AI judge or exact rules. Record the source and version of each label.

Stage 5: Train with a preference method

Start with DPO and a LoRA adapter. Keep the number of epochs low. Log reward accuracy, margin and length.

Stage 6: Compare and decide

Score the new model on the held-out set and compare it with both the SFT model and the base. Check general ability and safety. If gains are small or uneven, adjust the data before adjusting the algorithm.

Stage 7: Iterate or go online

If fixed pairs limit gains, sample from the new model, collect fresh labels or rule-based rewards, and run another round or switch to an online method.

Cost thinking

Most projects spend more on data and evaluation than on the training run itself. Plan for that.

Strengths and limitations

Strengths

• Covers the full post-training toolbox in one library.
• Short scripts. A working DPO run is a few dozen lines.
• Integrates with PEFT, Accelerate, Datasets and Transformers.
• Works from one GPU up to clusters.
• Handles chat templates, masking and packing.
• Open and widely used, with many community recipes.
• Supports both pair-based and single-label preference methods.

Limitations

• Fast-moving API. Argument and class names change across versions, so old scripts can break.
• PPO is memory hungry and finicky. It needs several models and careful tuning.
• Dependence on data quality. No trainer fixes noisy or biased preferences.
• Easy to overfit with many epochs on small preference sets.
• Evaluation is not included. You must build your own tests.
• Reward hacking and length drift are your problem to monitor.
• Large vocabularies and long sequences can cause memory spikes in log probability computation.
• Not every model architecture or modality is supported equally well.

Method trade-offs

How it compares with alternatives

How TRL fits with the others

These tools overlap rather than compete head to head. Axolotl and LLaMA-Factory can call TRL-style trainers underneath for preference stages. Unsloth offers patched kernels that can speed up TRL trainers. PEFT supplies the adapters. If you want full control of the loop, use TRL directly. If you want a file-driven workflow, use a wrapper.

DPO versus PPO

DPO is simpler and cheaper, and works offline on fixed pairs. PPO is online: it can explore, so it can improve beyond the fixed data, but it needs a good reward model and careful tuning. Many teams start with DPO and move to online methods when they need more.

Role in tabular foundation models

TRL targets language models, so its link to tabular foundation models is indirect. The link is still useful, because the same training stages appear around table-focused systems.

Background: how tabular foundation models learn

A tabular foundation model such as TabPFN, TabICL or TabSTAR is pretrained on many tables, mostly synthetic, each with rows and columns, mixed types and missing values, often without useful column names. At inference it reads labeled rows as context and predicts labels for new rows in one forward pass, which is in-context learning. See /model/architecture. Since the generator knows the true labels, pretraining uses direct supervised objectives, not preference learning.

Where TRL-style training applies near table models

1. Language models that read and write about tables. A model that turns a question into a query, labels columns, or explains a row can be tuned with SFT and then preference methods. TRL provides these stages. Examples of preferences: a query that runs and returns the right result beats one that errors; a column label that matches the values beats one that contradicts them.
2. Verifiable rewards from tables. Tables make good reward functions. If a generated query must return a known result on a test table, the reward is exact. If an imputed value must match a held-out true value, the reward is exact. If a column type label must match a schema, the reward is exact. Online methods with rule-based rewards fit these cases and avoid the noise of a judge.
3. Semantic and sector classification. Preference pairs of candidate labels, chosen by data from known schemas or by an AI judge, can tune a classifier-like language model. See semantic sector heads.
4. Imputation. For text and categorical fields, a language model can propose fills, with rewards based on held-out true values. For numeric fields, models built for tables remain the stronger choice. See imputation networks.
5. Anomaly explanations. After a tabular model flags a row, a tuned language model writes the reason. Preferences from reviewers on clarity and accuracy can be fed to DPO. See anomaly detection.
6. Continual learning. Adapters trained through TRL with PEFT let you add new domains over time, while a replay set and regression tests guard against forgetting. See adapters and /model/research.
7. Agents over tables. Agent loops that call table tools can be tuned with reward from task success. See agentic RAG for a related pattern and evals for measuring it.

Honest limits

• Preference training does not make numeric predictions better. For predicting a target column, use a model built for tables and measure it on real labels, see /model/benchmarks.
• Reward functions on tables must be tested for loopholes. A model may learn to return empty results if empty results are never penalized.
• Leakage is a risk. If the reward uses held-out values that also appear in training, scores will be inflated.

Practical guidance and best practices

Build in stages

Do SFT first, check the result, then add preference training. Skipping SFT often gives a policy that is too far from the data for preference methods to help.

Keep the data clean

• Deduplicate prompts and pairs.
• Check that chosen and rejected differ in substance.
• Remove prompts that appear in your evaluation set.
• Look at samples by eye before each run.

Start with safe hyperparameters

Adjust by watching held-out metrics, not only training loss.

Use adapters to save memory

Combine TRL with PEFT. For DPO, the adapter-off model can serve as the reference, saving a copy of the base model in memory. For large models, pair this with a quantized base as in QLoRA.

Control length

Preference methods can reward length. Track average output length on held-out prompts, add a length-aware objective if available, or balance the pairs.

Evaluate independently

Do not grade with the same judge that made the labels. Use held-out human review, task metrics and safety tests. See LLM as judge for judge pitfalls and red teaming for probing.

Pin versions

Record the library versions, the dataset hash and the config. Because APIs move quickly, a pinned environment saves hours.

Monitor training

Watch reward margins, accuracy, log probabilities and output length. Stop when held-out quality stops improving.

Plan for deployment

Save adapters and tokenizers together. Decide whether to merge. Test the merged model before release.

Common mistakes

• Skipping SFT. Preference training on a weak base is unstable.
• Wrong chat template. The model trains on one format and is used with another.
• Learning rate too high for DPO. Quality drops or outputs degenerate.
• Too many epochs on small preference data. The model memorizes.
• Noisy pairs. Ties and near ties add noise.
• Length bias. Chosen answers are longer in the data, so the model learns to ramble.
• No reference anchor. Drift grows without a KL penalty or reference.
• Reward hacking. The model exploits loopholes in a learned or coded reward.
• Judging with the training judge. Scores look good but real quality is flat.
• Ignoring memory for log probabilities. Long sequences with big vocabularies spike memory. Reduce length or batch size.
• Using an old tutorial with a new version. Argument names have changed.
• Not testing after merge. Merged and unmerged models can differ slightly.
• Forgetting safety checks. Helpfulness tuning can erode refusals.

Frequently asked questions

What is TRL used for?

TRL is used to post-train language models with supervised fine-tuning, reward modelling, reinforcement learning and direct preference methods. It gives ready-made trainers so that you can run alignment pipelines without writing the training loops yourself.

What does TRL stand for?

TRL stands for Transformer Reinforcement Learning. The name comes from the library's start as a tool for running reinforcement learning on transformer language models, though it now covers a wider set of post-training methods.

What is the difference between TRL and PEFT?

PEFT provides parameter-efficient methods such as LoRA that change how many weights are trained. TRL provides trainers that define the objective, such as SFT or DPO. They are often used together: TRL runs the objective while PEFT keeps the memory low.

Should I use PPO or DPO?

DPO is simpler, cheaper and works on fixed preference pairs, so it is a good first choice. PPO is online and can explore beyond the data, but it needs a reward model and more memory and tuning. Move to online methods when fixed pairs limit your gains.

Do I need a reward model for DPO?

No. DPO learns directly from chosen and rejected pairs and does not train a separate reward model. It does use a frozen reference model, which can be the same base model with the adapter switched off.

Can TRL run on a single GPU?

Yes, especially with LoRA or QLoRA, a modest sequence length and gradient checkpointing. PPO needs more memory than SFT or DPO, so smaller models or adapters are advised for single-GPU PPO.

What data format does DPO need?

DPO needs a prompt, a chosen answer and a rejected answer for each record. Chat-style data with messages is also supported, and the library applies the chat template. The data should be clean and the pairs should differ in substance.

Can I use AI-generated preference labels with TRL?

Yes. The library does not care where the labels come from. Labels from humans, from an AI judge or from rules all work, as long as they are formatted correctly. Audit AI labels against a human-checked sample, as described in the RLAIF page.

Is TRL useful for tabular data?

It is useful for the language side of table work: query generation, column labeling, imputation of text fields and anomaly explanations. Rewards based on test tables can be exact. For predicting a target column, models built for tables are better, and TRL does not replace them.

How do I avoid reward hacking?

Test rewards for loopholes, keep a KL penalty or reference anchor, monitor output length and format, use more than one signal, and audit samples by hand. Also evaluate with a different measure than the one used for training.

Glossary

• Advantage: How much better an action was than expected.
• Beta: The strength setting for staying near the reference in DPO.
• Chosen answer: The preferred answer in a pair.
• DPO: Direct preference optimization.
• KL divergence: A measure of how much one distribution differs from another.
• KTO: A preference method that uses single good or bad labels.
• ORPO: A method that combines SFT and preference learning.
• Packing: Joining short examples into one sequence.
• Policy: The model being trained.
• PPO: Proximal policy optimization, an online reinforcement learning algorithm.
• Reference model: A frozen model used as an anchor.
• Rejected answer: The less preferred answer in a pair.
• Reward model: A model that scores answers.
• Rollout: Answers generated by the current policy during training.
• SFT: Supervised fine-tuning.
• SimPO: A reference-free preference method with length normalization.
• Verifiable reward: A reward computed by an exact check.

Summary

TRL is the Hugging Face library that packages post-training for language models: supervised fine-tuning, reward modelling, PPO, DPO and related preference methods. It reduces alignment pipelines to short scripts, works with PEFT for low memory and with Accelerate for scale, and exposes metrics that help you monitor training. Its limits are a fast-changing API, the heavy and delicate nature of PPO, the need for clean preference data and the lack of built-in evaluation.

Good practice is to build in stages, clean the data, start with safe hyperparameters, use adapters, control output length, evaluate independently and pin versions. In table work, TRL supports the language-shaped tasks around tabular foundation models: query generation, semantic and sector labeling, text imputation, anomaly explanations and continual updates through adapters, with exact rewards where test tables allow. Numeric prediction stays with models built for tables. Related pages include PEFT, DPO, RLHF, RLAIF, Axolotl, /model/architecture and /model/benchmarks.

Related

It belongs to Training tooling in my stack, under LLMs, retrieval, and evaluation. Others in the same group: