PPO

Overview

PPO, short for proximal policy optimization, is a reinforcement learning algorithm that improves a policy in small, safe steps. It collects experience with the current policy, then updates the policy using a clipped objective that stops any single update from moving too far from the old behavior. This simple trick makes training far more stable than plain policy gradients, and it needs much less tuning than older trust-region methods.

Because it is stable, easy to implement and works with both discrete and continuous actions, PPO became the default choice for many control, scheduling and game-playing agents. It is also the optimizer behind many RLHF-style pipelines that tune language models from preference feedback. For models that learn from raw tables, PPO is rarely the main learner. It is the tool that trains decision-making agents that sit on top of tabular predictions, or that tunes pipelines which choose what to do with a table.

What it is

PPO is a policy-gradient method. A policy is a function that maps a state to a choice of action. In deep reinforcement learning the policy is a neural network. A policy-gradient method adjusts the network weights to make good actions more likely and bad actions less likely, based on the rewards that followed.

PPO belongs to the actor-critic family. It trains two networks, or one network with two heads:

• The actor is the policy. It outputs action probabilities (or the mean and spread of a continuous action).
• The critic is a value function. It estimates how good a state is, so the actor can tell whether an outcome was better or worse than expected.

PPO is on-policy. It learns from data gathered by the policy that is being improved. After each round of updates, the old data is thrown away and new data is collected. This is simple and stable, but it uses data less efficiently than off-policy methods such as SAC, TD3 and DDPG.

The core idea in one sentence

PPO maximizes expected advantage, but it clips the probability ratio between the new and old policy so that the update cannot become too large.

An everyday analogy

Imagine you are learning to cook a dish by changing the recipe a little each night. If you change salt, heat and timing all at once, and the meal is bad, you will not know why. If you make small changes and keep the ones that help, you improve steadily. PPO builds this habit into the math. The clip is a rule that says: do not change the recipe by more than a small amount per night, no matter how excited a single good meal made you.

Where it came from

Policy-gradient ideas go back to the REINFORCE algorithm from the early 1990s. REINFORCE is simple but noisy, because it relies on whole-episode returns and can swing wildly between updates.

Several steps led to PPO:

• Baselines and advantage estimates reduced the noise of policy gradients by subtracting a value estimate.
• Natural policy gradient methods used the geometry of the policy space to take better steps.
• TRPO (trust-region policy optimization, 2015) added a hard constraint on how far the policy could move, measured by KL divergence. It gave strong stability but needed second-order computations that were awkward to implement.
• Generalized advantage estimation (GAE, 2015) gave a clean way to trade bias against variance in advantage estimates.
• A2C and A3C (2016) made synchronous and asynchronous actor-critic training practical on many parallel environments.
• PPO (2017) kept the spirit of TRPO but replaced the constraint with a simple clipped objective that can be optimized with ordinary first-order methods such as Adam.

The PPO paper, "Proximal Policy Optimization Algorithms", appeared in 2017. Its appeal was practical: it matched or beat earlier methods on many benchmarks while being much easier to code. Later, PPO became the standard optimizer for reinforcement learning from human feedback, described in work on aligning language models with preference data. See RLHF and DPO for how that line of work evolved.

How it works

Step 1: collect experience

The current policy interacts with one or many copies of the environment for a fixed number of steps. At each step the agent stores the state, the action, the log-probability of that action, the reward, and the critic's value estimate. Running many environments in parallel gives a more varied batch and smoother gradients.

Step 2: compute returns and advantages

For each step, PPO needs an advantage. The advantage answers: "Was this action better or worse than what the critic expected?" A positive advantage means the action turned out better than average. A negative one means it turned out worse.

Most implementations use generalized advantage estimation. GAE combines one-step, two-step and longer temporal-difference errors using a decay parameter (often called lambda). A small lambda gives low variance but more bias. A lambda close to one gives low bias but more variance.

Step 3: build the probability ratio

For each stored action, compute the ratio

r = pi_new(action | state) / pi_old(action | state)

When the new policy equals the old one, the ratio is one. If the new policy makes the action twice as likely, the ratio is two. If it halves the likelihood, the ratio is 0.5.

Step 4: apply the clipped objective

The PPO-Clip objective is:

L = mean( min( r * A, clip(r, 1 - eps, 1 + eps) * A ) )

Here A is the advantage and eps is a small number such as 0.1 or 0.2. The clip function forces the ratio to stay inside the interval from 1 - eps to 1 + eps.

The minimum of the two terms makes the objective pessimistic:

• If the advantage is positive, the objective stops rewarding the policy once the ratio exceeds 1 + eps. There is no benefit in pushing a good action's probability up further than that.
• If the advantage is negative, the objective stops rewarding the policy once the ratio falls below 1 - eps. There is no benefit in pushing a bad action's probability down further than that.

So the policy gets no incentive to leave the neighborhood of the old policy. That neighborhood is the "proximal" region in the name.

Step 5: add the value loss and entropy bonus

The full loss usually has three parts:

• The policy loss (the clipped objective above, negated for minimization).
• The value loss, a squared error between the critic and the computed returns.
• An entropy bonus that encourages the policy to stay somewhat random, which helps exploration early in training.

The total is a weighted sum. Typical coefficients are 0.5 for the value loss and a small number, such as 0.01 or less, for entropy.

Step 6: run several epochs of minibatch updates

Unlike vanilla policy gradients, PPO reuses each batch for several passes. It splits the batch into minibatches and runs a few epochs of gradient steps. The clip keeps this reuse safe. Without it, many epochs on the same data would push the policy far from the one that collected the data, and the gradient estimates would become invalid.

Step 7: repeat

After the updates, the new policy becomes the old policy, the batch is discarded, and the loop starts again.

The KL-penalty variant

The paper also describes an adaptive KL-penalty version. Instead of clipping, it subtracts a penalty proportional to the KL divergence between old and new policies, and adjusts the penalty weight based on how large the measured KL was. This variant is used less often, but the idea of watching the KL divergence is still a useful diagnostic, and many RLHF setups combine a KL term against a reference model with PPO.

Key concepts and terms

A worked example

A numeric walk-through of the clip

Suppose eps = 0.2. A state has a sampled action with advantage A = +2.

• If the new policy makes the action 10 percent more likely, r = 1.1. The unclipped term is 1.1 * 2 = 2.2. The clipped term is also 2.2, because 1.1 is inside the interval. The minimum is 2.2. The gradient flows.
• If the new policy makes the action 50 percent more likely, r = 1.5. The unclipped term is 3.0. The clipped term is 1.2 * 2 = 2.4. The minimum is 2.4, and since the clipped branch is constant in r, the gradient is zero. The update stops pushing.

Now take A = -2.

• If r = 0.9, the unclipped term is -1.8 and the clipped term is also -1.8. The gradient flows.
• If r = 0.5, the unclipped term is -1.0, and the clipped term is 0.8 * (-2) = -1.6. The minimum is -1.6, which is clipped, so the gradient is zero.
• If r = 1.5 with A = -2, the unclipped term is -3.0 and the clipped term is -2.4. The minimum is -3.0, so the gradient still flows. This is the case where the policy made a bad action more likely, and PPO does want to correct that.

That last case is the reason for the minimum. It lets PPO undo mistakes even when the ratio is outside the interval.

A small PyTorch sketch

The following shows the core update. It is a teaching sketch, not a full training loop.

import torch
import torch.nn as nn
from torch.distributions import Categorical

class ActorCritic(nn.Module):
    def __init__(self, obs_dim, n_actions, hidden=64):
        super().__init__()
        self.body = nn.Sequential(
            nn.Linear(obs_dim, hidden), nn.Tanh(),
            nn.Linear(hidden, hidden), nn.Tanh(),
        )
        self.pi = nn.Linear(hidden, n_actions)
        self.v = nn.Linear(hidden, 1)

    def forward(self, obs):
        h = self.body(obs)
        return Categorical(logits=self.pi(h)), self.v(h).squeeze(-1)

def ppo_update(model, optimizer, batch, eps=0.2, vf_coef=0.5, ent_coef=0.01,
               epochs=4, minibatch=64):
    obs, actions, old_logp, returns, adv = batch
    adv = (adv - adv.mean()) / (adv.std() + 1e-8)  # normalize advantages
    n = obs.shape[0]
    for _ in range(epochs):
        perm = torch.randperm(n)
        for start in range(0, n, minibatch):
            idx = perm[start:start + minibatch]
            dist, value = model(obs[idx])
            logp = dist.log_prob(actions[idx])
            ratio = torch.exp(logp - old_logp[idx])
            unclipped = ratio * adv[idx]
            clipped = torch.clamp(ratio, 1 - eps, 1 + eps) * adv[idx]
            policy_loss = -torch.min(unclipped, clipped).mean()
            value_loss = (returns[idx] - value).pow(2).mean()
            entropy = dist.entropy().mean()
            loss = policy_loss + vf_coef * value_loss - ent_coef * entropy
            optimizer.zero_grad()
            loss.backward()
            nn.utils.clip_grad_norm_(model.parameters(), 0.5)
            optimizer.step()

Computing GAE

def compute_gae(rewards, values, dones, last_value, gamma=0.99, lam=0.95):
    T = len(rewards)
    adv = [0.0] * T
    gae = 0.0
    for t in reversed(range(T)):
        next_value = last_value if t == T - 1 else values[t + 1]
        nonterminal = 1.0 - dones[t]
        delta = rewards[t] + gamma * next_value * nonterminal - values[t]
        gae = delta + gamma * lam * nonterminal * gae
        adv[t] = gae
    returns = [a + v for a, v in zip(adv, values)]
    return adv, returns

A table-driven scenario

Consider an agent that decides, for each incoming batch of rows in a data pipeline, whether to run a cheap check or an expensive full validation. The state is a small vector built from summary statistics of the batch, such as the share of missing values and the drift score from a tabular model. The reward is the number of real errors caught minus the compute cost. PPO can learn a policy that spends expensive checks only when the summary statistics suggest trouble. The tabular model supplies the features. PPO supplies the decision rule.

Strengths and limitations

Strengths

• Stable. The clipped objective keeps updates small, so training rarely collapses.
• Simple. It needs only first-order optimization and few moving parts.
• General. It handles discrete, continuous and mixed action spaces.
• Parallel friendly. It works well with many environments running at once.
• Well understood. There are many reference implementations and a large body of practical advice.
• Works for language models. It is the classic optimizer for reward-model-based tuning.

Limitations

• Sample inefficiency. Being on-policy, it discards data after use. Off-policy methods such as SAC often need far fewer environment steps.
• Sensitivity to implementation details. Small choices such as advantage normalization, learning-rate schedules, observation scaling and initialization can change results a lot.
• Clipping is a heuristic. It does not strictly guarantee that the policy stays within a trust region, unlike the theory behind TRPO.
• Local optima. Like all policy-gradient methods, it can settle into poor behavior if exploration is weak.
• Reward design. It optimizes whatever reward it is given, including loopholes in the reward.
• Not ideal for expensive simulators. If each step costs a lot, the data appetite of an on-policy method is a problem.

How it compares with alternatives

A short rule of thumb: pick PPO when you want a dependable default and can afford many simulation steps. Pick an off-policy method when samples are costly. Pick DPO when you only need to align a language model with fixed preference pairs and want to avoid an RL loop.

For the general background of the field, see reinforcement learning. For settings with several agents, see multi-agent RL.

Role in tabular foundation models

PPO is not a model that reads tables. It is an optimization method for decision-making. Its role around tabular foundation models is therefore indirect but real. This section explains the connections concretely.

Tabular foundation models in brief

A tabular foundation model is pre-trained on a very large number of tables, often synthetic ones, so that it can handle a new table without task-specific training. It treats rows and columns as structure, copes with mixed numeric and categorical types, tolerates missing values, and does not rely on column names. At prediction time it uses in-context learning: the labeled rows of the new table are given as context, and the model predicts the unlabeled rows in a single forward pass. See TabPFN for a well-known example, and /model/architecture and /model/research for more on this model family.

Such a model is a predictor. It does not decide what to do next. PPO is a decision learner. The two fit together in several ways.

1. Tabular predictions as state features for a PPO agent

Many business processes are sequential decisions based on tabular data: which machine to service, which order to schedule, which batch to inspect. A tabular foundation model can turn each row or batch into compact features, such as predicted failure risk, predicted demand or an anomaly score. A PPO agent then uses these as part of its state. This avoids training a big feature extractor from scratch and lets the agent focus on the sequential trade-offs. See predictive maintenance for a domain where this pattern is common.

2. Tabular models as learned environments

Reinforcement learning needs an environment to interact with. Often only logged tables exist, not a simulator. A tabular model trained on the logs can act as a learned simulator: given a state row and an action, it predicts the next state or reward. PPO can then train against this model. Care is needed, because errors in the simulator can be exploited by the agent. Use short rollouts, add uncertainty checks, and validate the final policy on held-out real data. This idea relates to digital twins.

3. Reward signals from tabular quality checks

A PPO agent can be rewarded for improving data quality. For example, an agent that chooses cleaning steps may be rewarded when an anomaly detection model reports fewer outliers, or when downstream prediction error drops. The tabular model supplies the reward measurement. PPO optimizes the sequence of actions.

4. Imputation and missing values

Missing values are a core feature of real tables. A policy can choose, column by column, which imputation strategy to apply, and be rewarded by downstream accuracy. In-context tabular models can also impute directly by predicting a masked cell from the other cells in a row. PPO is not required for imputation, but it can serve as a meta-controller that picks among imputers. See imputation networks and synthetic data generation.

5. Pre-training on synthetic tables

Tabular foundation models are often pre-trained on tables sampled from a prior over data-generating processes. Choosing the settings of that prior is itself a search problem. A policy could be trained with PPO to select generator settings that maximize the pre-trained model's validation performance. This is an optional research-style use, and most practical systems simply tune the prior by hand or with Bayesian optimization.

6. Continual learning

A deployed tabular model may need to adapt as data drifts. A PPO agent can decide when to retrain, which data to replay and how much to update, with a reward that balances accuracy against compute and forgetting. The agent's state would include drift scores and recent error rates.

7. Sector and semantic classification

When a table has no column names, a foundation model can still infer the semantic type of a column or the sector of a dataset from the values and their statistics. A routing agent trained with PPO could use these inferred labels to pick the right downstream pipeline. The classification is done by the tabular model, and the routing policy is learned by PPO.

8. Preference tuning of language models that explain tables

Language models that describe tables or write queries over them are often tuned with preference data. PPO is a classic way to run that tuning through a reward model, as described in RLHF. The newer DPO and RLAIF approaches offer related alternatives.

What PPO is not

PPO does not replace in-context learning on tables. If your goal is to predict a column from other columns, use a supervised or foundation model. Use PPO only when the problem is truly sequential, with actions that change future states and rewards that arrive later.

Practical guidance and best practices

Start from a known good configuration

A common starting point for many tasks is:

Treat these as starting values only. Tune on your task.

Normalize things

• Normalize advantages within each minibatch or batch.
• Normalize or scale observations, using running mean and variance.
• Scale or clip rewards so their magnitude is moderate.

These steps often matter more than any clever change to the algorithm.

Monitor the right diagnostics

Log these values every update:

• Mean episode return and length.
• Approximate KL divergence between old and new policy.
• The fraction of samples where the ratio was clipped.
• Policy entropy.
• Explained variance of the value function.
• Gradient norm.

If the approximate KL is large, the learning rate or number of epochs is too high. If the clip fraction is near zero, the updates are very small. If entropy collapses early, raise the entropy bonus. If explained variance is poor, the critic is lagging.

Early stopping on KL

Some implementations stop the epoch loop early when the approximate KL exceeds a target. This guards against occasional destructive updates.

Use a configuration file

algo: ppo
env: example-scheduling-env
seed: 7
n_envs: 16
rollout_steps: 256
gamma: 0.99
gae_lambda: 0.95
clip_range: 0.2
epochs: 4
minibatches: 8
learning_rate: 0.0003
lr_schedule: linear
ent_coef: 0.005
vf_coef: 0.5
max_grad_norm: 0.5
normalize_obs: true
normalize_adv: true
log_dir: /var/log/example.internal/ppo-runs

Run several seeds

Reinforcement learning results vary a lot between random seeds. Report the mean and spread over at least a handful of seeds, never a single lucky run.

Handle time limits properly

If an episode ends because of a time limit rather than a real terminal state, bootstrap the value estimate instead of treating the end as a true terminal. Mixing these up biases the critic.

Keep the reward honest

Write the reward function carefully. Test it with simple hand-made policies first. Look for loopholes: if the agent can score by doing something silly, it will.

For language model tuning

• Keep a frozen reference model and penalize the KL divergence from it, to avoid drifting into gibberish that fools the reward model.
• Watch for reward hacking, where the policy finds outputs the reward model likes but humans do not.
• Consider DPO if you want to avoid the cost and fragility of a full RL loop.

Variants and extensions

PPO has many practical variants. Knowing them helps when the plain version struggles.

• Recurrent PPO adds a memory layer such as an LSTM or GRU, so the agent can act when the state is only partly observed.
• PPO with action masking removes invalid actions from the distribution, which is useful in scheduling problems where some choices are not allowed in some states.
• Multi-agent PPO shares or centralizes the critic across agents. See multi-agent RL.
• Value clipping also clips the change in the value estimate. Its benefit is debated, so test it before relying on it.
• Group-based advantage estimates replace the critic with comparisons between several sampled outputs for the same prompt. They are common in language model tuning.
• Penalty-based KL control adds a KL term against a reference policy and adapts its weight over time.

A debugging checklist

When a PPO run does not learn, work through the following list in order.

1. Check that the environment is correct. Run a random policy and a hand-written policy and confirm the rewards make sense.
2. Check shapes and dtypes. A broadcasting error in the advantage or the log-probability can silently ruin learning.
3. Confirm that old log-probabilities are stored at collection time and are not recomputed with the updated network.
4. Confirm that done flags and time-limit flags are handled separately.
5. Plot the approximate KL. If it is huge, lower the learning rate or the number of epochs.
6. Plot the clip fraction. If it is near zero for a long time, the policy is barely moving.
7. Plot entropy. If it falls to zero in the first few updates, the policy has collapsed. Raise the entropy bonus or lower the learning rate.
8. Plot explained variance of the value function. If it is negative, the critic is worse than predicting a constant.
9. Try a smaller, simpler task first. If the code cannot solve a toy problem, it will not solve the real one.
10. Fix the seed and compare against a trusted reference implementation on the same task.

Using PPO with tabular inputs

When the state is a row or a set of columns, a few extra points help:

• Embed categorical columns before feeding them to the policy network, rather than using raw integer codes.
• Scale numeric columns with running statistics, since the distribution may shift as the policy changes the data it sees.
• Mark missing values explicitly with an indicator, so the policy can learn what to do when data is absent.
• Keep the state small. Summaries of a table, such as counts, means and model scores, are usually better inputs than the full table.
• Log the raw state alongside each decision so that every action can be audited later.

Safety and governance

A policy that acts on business data should be treated like any other production decision system.

• Keep a rule-based fallback that takes over when the policy is uncertain or when inputs look out of range.
• Test the policy offline on logged data before it controls anything.
• Roll it out gradually, with limits on how much it may change per day.
• Record the version of the policy, the data it saw and the action it took.
• Review the reward function whenever the business goal changes.

Common mistakes

• Forgetting to normalize advantages. This often causes slow or unstable learning.
• Too many epochs per batch. The policy drifts away from the data-collecting policy, and clipping can no longer protect it.
• Learning rate too high. Symptoms are spiking KL and collapsing returns.
• Mixing up truncation and termination. This biases value targets.
• Sharing a network between actor and critic without care. If the value loss dominates, the shared features can suffer. Either scale the loss terms or use separate networks.
• Ignoring observation scaling. Unscaled inputs make tanh or relu layers saturate.
• Evaluating with the stochastic policy only. Also evaluate the deterministic (mean) action, but know which one you will deploy.
• Reporting one seed. Results will not reproduce.
• Training on a learned simulator without validation. The agent may exploit simulator errors, and the real-world result will disappoint.
• Using PPO when the problem is not sequential. A supervised model is simpler and better.
• Reward leakage. If the state includes information that reveals the future reward, the agent will learn a shortcut that fails at deployment.

Frequently asked questions

What does PPO stand for?

PPO stands for proximal policy optimization. The word proximal means "nearby". The algorithm keeps each new policy close to the previous one by clipping the probability ratio, which makes learning stable and less sensitive to the choice of step size.

Is PPO on-policy or off-policy?

PPO is on-policy. It learns from data collected by the policy that is currently being improved, and it discards the data after a few epochs of updates. Importance ratios and clipping let it reuse each batch a few times, but it cannot use a large replay buffer of old data the way off-policy methods do.

Why is PPO more popular than TRPO?

PPO gets much of the stability of TRPO with a far simpler implementation. TRPO needs a constrained optimization step that uses second-order information, while PPO uses a clipped loss that works with ordinary gradient descent. In practice PPO is easier to code, easier to scale and often performs as well or better.

What is the clip range and how should I set it?

The clip range, written eps, is the allowed distance of the probability ratio from one. Values of 0.1 to 0.2 are common starting points. A smaller value makes updates more conservative and training slower but safer. A larger value allows bigger steps and can speed up learning but risks instability.

Can PPO handle continuous actions?

Yes. For continuous actions the actor outputs the mean and a spread, often a learned log standard deviation, of a Gaussian distribution. The probability ratio is computed from the density of the chosen action. This makes PPO usable for control tasks such as robotics, energy management and process tuning, though off-policy methods like SAC can be more sample efficient there.

How does PPO relate to RLHF?

In RLHF a language model is the policy, a reward model trained on human preferences provides the reward, and PPO updates the language model to increase that reward. A KL penalty against a frozen reference model keeps the tuned model from drifting too far. Newer methods such as DPO skip the reward model and the RL loop by optimizing a loss directly on preference pairs.

Does PPO need a critic?

The standard version does, because the critic provides the baseline used to compute advantages. Some variants for language models estimate advantages from groups of sampled outputs instead of using a learned value network, which saves memory. For classic control tasks, a critic is the normal and best-supported choice.

Why does PPO need so many samples?

Because it is on-policy. Each batch is used for only a few update passes and then discarded. If the environment is cheap, such as a fast simulator, this is fine, and parallel workers produce data quickly. If every step is expensive, an off-policy method that reuses old data will usually be more economical.

How does PPO connect to tabular foundation models?

PPO is a decision learner, while a tabular foundation model is a predictor. The tabular model can supply state features, rewards, imputations or a learned simulator, and PPO can learn the sequential policy that acts on them. The two are complementary, and PPO is not needed for ordinary prediction on a table.

How do I know PPO is training well?

Watch the return curve, the approximate KL divergence, the clip fraction, the entropy and the value function's explained variance. A healthy run shows rising return, small and stable KL, a moderate clip fraction, slowly declining entropy and a critic that explains a good share of the return variance.

Glossary

• Actor: the part of the agent that chooses actions.
• Advantage: the extra value of an action compared with the average action in a state.
• Clip fraction: the share of samples whose ratio fell outside the clip interval.
• Critic: the part of the agent that estimates state values.
• Discount factor: the weight given to future rewards.
• Entropy: a measure of how random the policy is.
• Episode: one run of the environment from start to end.
• GAE: generalized advantage estimation.
• KL divergence: a measure of the gap between two distributions.
• On-policy: learning from data produced by the current policy.
• Policy: the rule that maps states to actions.
• Probability ratio: new action probability over old action probability.
• Return: the discounted sum of future rewards.
• Rollout: a batch of experience collected for one update.
• Reward hacking: exploiting flaws in the reward instead of doing the intended task.
• Value function: the expected return from a state under the policy.

Summary

PPO is a stable, simple and widely used policy-gradient algorithm. It collects experience, estimates advantages, and updates the policy with a clipped objective so that each step stays close to the previous policy. It handles discrete and continuous actions, scales across parallel environments, and is the classic optimizer in RLHF-style language model tuning.

It is sample hungry and sensitive to implementation details, so normalize inputs and advantages, monitor KL and clip fraction, and run several seeds. Compare it with TRPO, SAC, TD3, DDPG and DQN when choosing a method.

Around tabular foundation models, PPO is a partner rather than a rival. The tabular model predicts, imputes and scores rows. PPO learns what to do next with those outputs. Read more on /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Reinforcement learning in my stack, under AI and machine learning. Others in the same group: