TRPO

Overview

TRPO, short for trust region policy optimization, is a reinforcement learning algorithm that improves a policy while limiting how far each update may move it. The limit is a constraint on the average KL divergence between the old and new policy. Inside that "trust region" the algorithm can be confident that its local estimate of improvement is reliable. The result is steady, conservative learning with a theory that links each step to a guaranteed lower bound on improvement.

TRPO is more complex to implement than its descendant PPO. It uses a conjugate gradient solver, a Fisher-vector product and a line search. For that reason many practitioners now start with PPO. TRPO still matters because it explains why PPO works, and because it remains a good choice when very careful, predictable policy updates are more important than speed. Around models that learn from raw tables, it plays a supporting role: it trains decision agents that act on tabular predictions, and its trust-region idea shows up in many safe-update schemes.

What it is

TRPO is an on-policy policy-gradient method with an actor-critic structure in most implementations. The actor is a policy network. The critic is a value network used to compute advantages. At every iteration, TRPO collects a batch of experience, then solves a constrained optimization problem:

• Maximize a surrogate objective, which estimates how much the new policy would improve over the old one.
• Subject to the average KL divergence between the old and new policy being at most a small number, called delta.

The second line is the trust region. It says: you may change the policy, but only within a neighborhood where we still trust our estimate.

An everyday analogy

Imagine walking down a hillside in thick fog with a flashlight that lights only a few meters ahead. You can see the slope near your feet, but not far away. If you take a giant step based on the nearby slope, you may walk off a cliff. If you take a step no longer than the lit circle, the slope you measured is a good guide. The trust region is the lit circle. The KL limit is its radius.

Why a plain gradient step is risky

In ordinary supervised learning a bad step costs a little accuracy and the next batch repairs it. In reinforcement learning, a bad step changes the behavior of the agent, and therefore changes the data it collects next. A bad policy produces bad data, which produces worse updates. Performance can collapse and not recover. This feedback loop is why policy updates need extra care.

Where it came from

The roots of TRPO lie in several earlier ideas:

• Policy gradient methods such as REINFORCE (early 1990s) showed how to climb the expected reward directly.
• Natural gradient ideas used the geometry of probability distributions, measured by the Fisher information matrix, to make steps that are small in distribution space rather than in parameter space.
• Conservative policy iteration (early 2000s) gave a theoretical bound on how much a mixed policy could improve, and showed that small, careful mixing gives guaranteed progress.
• Natural actor-critic methods applied the natural gradient to actor-critic learning.

TRPO, introduced in a 2015 paper titled "Trust Region Policy Optimization", combined these threads. It proved a bound that relates the true performance of a new policy to a surrogate objective plus a penalty based on the maximum KL divergence. It then turned this theory into a practical algorithm by replacing the maximum KL with the average KL, and by using a hard constraint with a fixed limit instead of a penalty with the theoretical coefficient, which would give tiny steps.

The same research line produced generalized advantage estimation (2015), which is commonly paired with TRPO to reduce the variance of the advantage estimates.

In 2017, PPO simplified the idea by replacing the hard constraint with a clipped objective. Related practical variants of the trust-region idea include ACKTR, which uses a Kronecker-factored approximation to the natural gradient. For the wider field, see reinforcement learning.

How it works

Step 1: collect trajectories

Run the current policy in the environment, typically for several thousand steps in total. Store states, actions, rewards and the action probabilities. Unlike off-policy methods such as DDPG or SAC, TRPO uses this data once and then discards it.

Step 2: estimate advantages

Compute returns and an estimate of the advantage for each step. The advantage says how much better an action was than the policy's average action in that state. Most implementations fit a value network by regression on the returns and use generalized advantage estimation to compute advantages from it.

Step 3: build the surrogate objective

The surrogate objective is the average of

ratio * advantage

where the ratio is the new action probability divided by the old action probability. At the old policy the ratio equals one, and the gradient of the surrogate equals the ordinary policy gradient. As the new policy moves away from the old one, the surrogate becomes a less accurate estimate of the true improvement.

Step 4: define the trust region

Measure the average KL divergence between the old and new action distributions over the states in the batch. The constraint is

average KL(old, new) <= delta

where delta is a small number, often around 0.01.

Step 5: find the step direction

Solving the constrained problem exactly is hard for large networks. TRPO uses a second-order approximation:

• Approximate the surrogate by its first-order (linear) term, with gradient g.
• Approximate the KL constraint by its second-order (quadratic) term, whose matrix is the Fisher information matrix F.

The solution of this simplified problem points in the natural gradient direction, F inverse times g. The step is scaled so that the quadratic KL estimate exactly equals delta.

Step 6: use conjugate gradient to avoid inverting F

A policy network can have hundreds of thousands of parameters, so F is far too large to store or invert. The trick is that F inverse times g can be found by solving F x = g with the conjugate gradient method. This method only needs products of F with a vector, and those products can be computed with automatic differentiation without ever forming F. About ten conjugate gradient iterations are usually enough.

Step 7: scale the step

The proposed step is x. The scale factor is sqrt(2 * delta / (x^T F x)). Multiplying x by this factor gives a step whose predicted KL equals delta.

Step 8: backtracking line search

Because the approximations are only valid locally, the proposed step may violate the true KL constraint or may fail to improve the true surrogate. TRPO therefore tries the full step, then shrinks it by a factor such as 0.5 repeatedly until two conditions hold:

• The true average KL is within delta.
• The surrogate objective improves.

If no acceptable step is found after a fixed number of tries, the update is skipped. This line search is what makes TRPO robust.

Step 9: update the critic and repeat

Fit the value network, often with a few steps of an ordinary optimizer, and then go back to Step 1 with the new policy.

Where the guarantee comes from

The original theory shows that if you maximize the surrogate minus a penalty proportional to the maximum KL, the true performance never decreases. This is called monotonic improvement. The practical algorithm departs from the theory in three ways: it uses the average KL instead of the maximum, uses a fixed limit instead of the theoretical penalty, and uses sampled estimates. The guarantee is therefore approximate in practice, but the behavior is still very stable.

Key concepts and terms

A worked example

A toy numeric picture of the KL constraint

Suppose a policy chooses between two actions. The old policy picks action A with probability 0.5 and action B with probability 0.5. Three candidate new policies are:

The KL values come from the formula sum of p_old * ln(p_old / p_new). With delta = 0.01, only candidate 1 is allowed. Candidate 2 exceeds the limit. The line search would shrink a proposed move toward candidate 3 until its KL fell below 0.01, which gives a policy between candidate 1 and candidate 2.

This shows the trade-off. A larger step might help more if the surrogate were exact. But the surrogate is only accurate nearby, so TRPO trades speed for reliability.

Pseudocode

initialize policy pi_theta and value function V_phi
repeat for each iteration:
    collect trajectories with pi_theta
    compute returns and advantages A (for example with GAE)
    g = gradient of surrogate at theta
    solve F x = g by conjugate gradient (Fisher-vector products only)
    step = sqrt(2 * delta / (x . F x)) * x
    for k in 0..max_backtracks:
        theta_new = theta + 0.5**k * step
        if KL(theta, theta_new) <= delta and surrogate(theta_new) improves:
            theta = theta_new
            break
    fit V_phi to returns by regression

A compact PyTorch sketch

This is a teaching sketch for a discrete policy. It omits batching, normalization and logging.

import torch

def flat_grad(y, params, retain=False, create=False):
    grads = torch.autograd.grad(y, params, retain_graph=retain, create_graph=create)
    return torch.cat([g.reshape(-1) for g in grads])

def set_params(params, flat):
    i = 0
    for p in params:
        n = p.numel()
        p.data.copy_(flat[i:i + n].view_as(p))
        i += n

def conjugate_gradient(fvp, b, iters=10, tol=1e-10):
    x = torch.zeros_like(b)
    r = b.clone()
    p = b.clone()
    rr = r @ r
    for _ in range(iters):
        Ap = fvp(p)
        alpha = rr / (p @ Ap)
        x += alpha * p
        r -= alpha * Ap
        new_rr = r @ r
        if new_rr < tol:
            break
        p = r + (new_rr / rr) * p
        rr = new_rr
    return x

def trpo_step(policy, states, actions, advantages, old_logp, old_dist_logits,
              delta=0.01, damping=0.1, backtracks=10):
    params = list(policy.parameters())

    def surrogate():
        logits = policy(states)
        logp = torch.distributions.Categorical(logits=logits).log_prob(actions)
        return (torch.exp(logp - old_logp) * advantages).mean()

    def mean_kl():
        new = torch.distributions.Categorical(logits=policy(states))
        old = torch.distributions.Categorical(logits=old_dist_logits)
        return torch.distributions.kl_divergence(old, new).mean()

    g = flat_grad(surrogate(), params, retain=True)

    def fvp(v):
        kl = mean_kl()
        grad_kl = flat_grad(kl, params, create=True)
        return flat_grad(grad_kl @ v, params, retain=True) + damping * v

    x = conjugate_gradient(fvp, g)
    step = torch.sqrt(2 * delta / (x @ fvp(x))) * x
    old_params = torch.cat([p.data.reshape(-1) for p in params])
    old_loss = surrogate().item()
    for k in range(backtracks):
        set_params(params, old_params + (0.5 ** k) * step)
        if mean_kl().item() <= delta and surrogate().item() > old_loss:
            return True
    set_params(params, old_params)
    return False

The function returns whether an acceptable step was found. If not, the policy is restored to its previous parameters.

A table-driven scenario

Suppose an agent chooses, every hour, how many machines in a plant to run at high, medium or low speed. The state includes failure-risk scores from a tabular model, current queue lengths and energy prices. The reward balances throughput, energy cost and breakdowns. A bad policy update could cause an expensive wave of breakdowns during training in a realistic simulator, and could waste days of compute. TRPO's small, checked steps suit this kind of setting, where each policy change should be modest and predictable.

Strengths and limitations

Strengths

• Stable updates. The KL constraint and line search make wild policy changes rare.
• Strong theory. The improvement bound explains why small steps work.
• Little step-size tuning. Delta has a clear meaning in terms of behavior change, unlike a learning rate, which has meaning only in parameter space.
• Handles large policies. Conjugate gradient avoids ever building the Fisher matrix.
• Good for sensitive tasks. Where bad intermediate policies are costly, careful steps help.

Limitations

• Implementation complexity. The Fisher-vector product, conjugate gradient solver and line search are fiddly to code and debug.
• Compute per update. Each iteration needs several extra forward and backward passes.
• Poor fit with shared parameters. Architectures that share parameters between the policy and value function, or that use noise layers such as dropout, are awkward under the KL constraint.
• On-policy sample cost. It discards data after every update, so it needs many environment steps. See SAC and TD3 for more efficient off-policy options.
• Not always best in practice. PPO is simpler and often as good, so TRPO is seldom the first pick today.
• Approximate guarantee. The practical algorithm departs from the theory, so improvement is not strictly monotonic.

How it compares with alternatives

TRPO and PPO share the same goal. TRPO enforces the limit explicitly with second-order machinery. PPO enforces it softly with clipping. In many studies the two perform similarly, and PPO wins on speed and simplicity.

Role in tabular foundation models

TRPO is a method for learning decisions, not for reading tables. A tabular foundation model, by contrast, learns from raw tables: it respects the row and column structure, handles numeric and categorical columns together, tolerates missing values, needs no column names, and is often pre-trained on large numbers of synthetic tables. At prediction time it performs in-context learning, using labeled rows as context to predict new rows without weight updates. See TabPFN and /model/architecture for details. The two kinds of tools connect in several concrete ways.

Predictor and decision maker

The foundation model predicts, imputes and scores. A TRPO agent decides. A common pattern is to let the tabular model turn raw records into a small set of useful signals, such as predicted risk, expected demand or an anomaly score, and to feed these to a TRPO policy as state. The policy then chooses actions over time. This division of labor keeps the policy small, which is helpful because TRPO's second-order computations cost more with large networks. See predictive maintenance for a typical domain.

The trust-region idea in continual learning

A tabular model deployed in production must adapt as data shifts. Updating it too aggressively can erase useful knowledge, which is called catastrophic forgetting. The trust-region idea carries over: limit how far the model's output distribution may move from its previous version on a reference set of rows, and take only steps that stay inside that limit. A KL constraint on predictions is a well-known way to do this. TRPO's reasoning about small, measured changes is a useful template, even when the learner is a supervised model rather than a policy.

Learned simulators built from tables

When only logged tables exist, a model trained on them can act as a simulator. TRPO can train against it. The trust region helps here in a second way: a policy that stays close to the logged behavior is less likely to wander into regions where the learned simulator is inaccurate. Keeping the KL to the data-collecting policy small is a standard safeguard in offline and model-based reinforcement learning. Related ideas appear in digital twins.

Imputation and data cleaning as sequential choices

Choosing which imputation method to apply to each column, or in what order to apply cleaning steps, is a sequential decision. A TRPO agent could learn this choice, rewarded by downstream accuracy or by a drop in outliers found by anomaly detection. The tabular model itself can also impute a missing cell directly by predicting it from the rest of the row. See imputation networks.

Synthetic pre-training data

Pre-training on synthetic tables requires choosing the settings of the generator. This choice could be treated as a control problem, with a policy proposing generator settings and a reward based on how well the pre-trained model does on validation tables. In practice teams usually tune such settings by hand or with Bayesian optimization. See also synthetic data generation.

Sector and semantic classification

Foundation models can infer the meaning of a column, or the sector a table belongs to, from values and their statistics even when column names are missing. A downstream agent can use those inferred labels to route a table to the right processing pipeline, and TRPO can learn that routing policy when the outcomes are delayed.

Preference tuning for table-reading language models

Language models that explain tables or generate queries are often tuned from preference data. TRPO is an ancestor of the PPO method used there. See RLHF and DPO. Direct preference methods avoid the RL loop altogether.

An honest boundary

If the task is to predict a column from other columns, use a supervised or foundation model. TRPO only earns its place when actions change future states and rewards arrive later.

Practical guidance and best practices

Choose the KL limit with care

Delta is the main knob. A typical value is around 0.01. Smaller values give very cautious learning. Larger values speed up learning but raise the risk that the line search rejects steps or that performance collapses. Watch the fraction of iterations where the line search fails.

Typical settings

Treat these as starting points.

Keep policy and value networks separate

Sharing parameters between actor and critic complicates the KL constraint, because the value loss would also change the policy's outputs. Using two networks is the usual practice with TRPO.

Avoid sources of noise in the policy

Dropout and other random layers make the KL between old and new policy noisy, since the "old" and "new" outputs are no longer deterministic functions of the state. Prefer deterministic layers.

Normalize inputs and advantages

Scale observations with running statistics. Normalize advantages within each batch. These steps are as helpful here as with any policy-gradient method.

Use a larger batch than you think

Because TRPO makes one update per batch, the quality of the gradient and of the Fisher estimate depends on the batch. Use enough steps to give stable estimates. Parallel environments make this affordable.

Monitor these quantities

• Mean return per iteration.
• Actual KL after the step compared with delta.
• Number of backtracking steps used.
• Policy entropy.
• Value function explained variance.
• Conjugate gradient residual.

Example configuration

algo: trpo
env: example-plant-scheduling
seed: 3
n_envs: 16
steps_per_iteration: 8192
gamma: 0.99
gae_lambda: 0.97
max_kl: 0.01
cg_iters: 10
cg_damping: 0.1
backtrack_coeff: 0.5
backtrack_iters: 10
value_lr: 0.001
value_epochs: 5
normalize_obs: true
normalize_adv: true
log_dir: /var/log/example.internal/trpo-runs

Decide between TRPO and PPO

Start with PPO. Move to TRPO only if you see unstable updates that you cannot fix with a lower learning rate, or if you need the explicit meaning of a KL limit for governance reasons, for example "the policy never changes by more than a fixed amount per release".

Governance for deployed policies

If a policy acts on business data, keep a rule-based fallback, test offline first, limit how much the policy may change between releases, and record which version made which decision. A KL limit is a natural way to express the maximum allowed change.

Related methods worth knowing

• ACKTR applies a Kronecker-factored approximation of the natural gradient, which makes second-order style steps cheaper for larger networks.
• Natural policy gradient drops the line search and uses a fixed step in the natural gradient direction. It is simpler but less safe.
• Mirror descent policy optimization frames the KL term as a Bregman divergence and updates the policy by a proximal step.
• Maximum a posteriori policy optimization also uses a KL constraint, but in an off-policy setting with a learned critic.
• Constrained policy optimization adds cost constraints for safety on top of the trust region, so that expected cost stays under a limit while reward improves.

These show that the trust-region idea is a family, not a single algorithm.

How to read a TRPO training log

A healthy run has a recognisable shape. Early iterations show rising return and a measured KL close to delta, since the line search usually accepts the full step. Later the return curve flattens, and the line search may begin to shrink steps more often, because the surrogate becomes less reliable near a good policy.

Warning signs include:

• Measured KL far below delta for many iterations. The step is being shrunk too much, or the gradient is tiny.
• Measured KL repeatedly above delta before the line search. The quadratic approximation is poor, so raise damping or the batch size.
• Many failed line searches in a row. The advantage estimates are probably noisy, or the value function is poor.
• Return that rises and then falls sharply. Check for a reward loophole, a bug in done flags, or an observation scaling problem.

Cost of a TRPO iteration

Each iteration needs one pass to collect data, one backward pass for the gradient, roughly ten Fisher-vector products for conjugate gradient, a few forward passes for the line search, and a regression for the critic. Each Fisher-vector product costs about as much as two backward passes. For small and medium policy networks this is fine. For very large networks, such as language models, the cost is one reason that PPO and direct preference methods took over.

Testing your implementation

A few simple tests catch most bugs before a long run.

• With delta set to zero, the policy should not change at all.
• On the first step, the ratio should be exactly one and the KL should be zero.
• The Fisher-vector product should be symmetric: u dot F v should equal v dot F u.
• On a tiny bandit problem, the policy should move toward the best arm within a few iterations.
• Compare the conjugate gradient solution on a small network with a direct matrix solve.

A short decision guide

Common mistakes

• Treating delta like a learning rate. It measures behavior change, not step length in parameter space. Tune it by watching actual KL and returns.
• Skipping damping in the Fisher-vector product. Without damping, the conjugate gradient solve can be unstable.
• Forgetting the line search. Without it, the quadratic approximations can lead to steps that violate the constraint or make the policy worse.
• Using too few conjugate gradient iterations. The step direction is then poor.
• Recomputing old probabilities after the update. Old log-probabilities and distributions must be stored from the collection time.
• Sharing layers between actor and critic. This complicates the constraint and can hurt learning.
• Using stochastic layers. Dropout makes the KL measurement noisy.
• Confusing time-limit truncation with true termination. Bootstrap the value in the first case.
• Reporting a single seed. Reinforcement learning results vary widely between runs.
• Using TRPO for a non-sequential task. A supervised model is simpler and better.

Frequently asked questions

What does TRPO stand for?

TRPO stands for trust region policy optimization. The trust region is the neighborhood around the current policy in which the algorithm trusts its local estimate of improvement. The size of the neighborhood is set by a limit on the average KL divergence between the old and new policy.

How is TRPO different from PPO?

Both aim to keep each policy update small. TRPO uses a hard KL constraint solved with natural gradient, conjugate gradient and a line search. PPO replaces this with a clipped objective that can be optimized with ordinary gradient descent. PPO is simpler and faster to run, while TRPO has a more direct theoretical link to guaranteed improvement.

Why does TRPO use the KL divergence?

KL divergence measures how different two probability distributions are, so it measures how much the behavior of the policy changes. A limit on the parameter distance would not capture this, because a small parameter change can sometimes change the output a lot and sometimes barely at all. Limiting the KL controls behavior change directly.

What is the natural gradient?

The natural gradient is the ordinary gradient multiplied by the inverse of the Fisher information matrix. It gives the steepest direction of improvement when distance is measured between distributions rather than between parameter vectors. TRPO's step direction is this natural gradient, scaled to meet the KL limit.

Why is conjugate gradient needed?

The Fisher matrix is too large to store or invert for a neural network policy. Conjugate gradient solves the linear system needed for the step using only matrix-vector products. These products can be computed with automatic differentiation, so the full matrix is never built.

Does TRPO guarantee improvement at every step?

The theory behind TRPO guarantees monotonic improvement for an idealized version of the update. The practical algorithm uses an average KL, a fixed limit and sampled estimates, so the guarantee holds only approximately. The line search adds a real check that the surrogate improves and the constraint is met.

Is TRPO on-policy or off-policy?

TRPO is on-policy. It learns from trajectories collected by the current policy and discards them after each update. This keeps the surrogate objective valid, but it means the method needs many environment steps compared with off-policy methods.

When should I use TRPO instead of PPO?

Use TRPO when you need explicit, interpretable control over how much behavior changes per update, or when PPO shows instability that you cannot tame with ordinary tuning. In most other cases PPO is the more practical default because it is simpler to implement and cheaper per iteration.

Can TRPO work with continuous actions?

Yes. For continuous actions, the policy outputs the mean and spread of a Gaussian distribution, and the KL divergence between two Gaussians has a closed form. TRPO was originally demonstrated on both discrete tasks and simulated locomotion tasks with continuous actions.

How does TRPO relate to tabular foundation models?

A tabular foundation model reads raw tables and predicts or imputes in context, while TRPO learns sequential decisions. The tabular model can supply the state features, the learned simulator or the reward signals, and TRPO can learn the policy that uses them. The trust-region idea also inspires safe, bounded updates when such models are adapted over time.

Glossary

• Actor: the policy network that chooses actions.
• Advantage: how much better an action was than average in its state.
• Backtracking line search: shrinking a step until it meets the requirements.
• Conjugate gradient: an iterative solver for linear systems using matrix-vector products.
• Critic: the value network that estimates returns.
• Damping: a small constant added for numerical stability.
• Delta: the maximum allowed average KL per update.
• Fisher information matrix: a matrix capturing how sensitive a distribution is to its parameters.
• KL divergence: a measure of how one distribution differs from another.
• Monotonic improvement: a guarantee that performance never decreases between updates.
• Natural gradient: the gradient scaled by the inverse Fisher matrix.
• On-policy: using only data from the current policy.
• Surrogate objective: a local estimate of policy improvement.
• Trust region: the neighborhood in which an approximation is trusted.

Summary

TRPO improves a policy by maximizing a surrogate objective inside a trust region defined by a KL-divergence limit. It finds the natural gradient direction with conjugate gradient, scales the step to meet the limit, and verifies the result with a backtracking line search. The outcome is stable, predictable learning with a clear theory behind it.

It is more complex and costlier per update than PPO, which approximates the same idea with clipping, so PPO is usually the first choice. TRPO remains valuable for understanding policy optimization and for tasks that need strictly controlled changes. Off-policy methods such as SAC, TD3 and DDPG are better when samples are expensive.

For tabular foundation models, TRPO is a partner for sequential decisions rather than a table reader. The tabular model supplies predictions, imputations, anomaly scores and learned simulators. TRPO supplies a cautious way to learn what to do with them. See /model/research, /model/architecture and /model/benchmarks for the model side.

Related

It belongs to Reinforcement learning in my stack, under AI and machine learning. Others in the same group: