DDPG

Overview

DDPG, short for deep deterministic policy gradient, is an off-policy actor-critic algorithm that learns a deterministic policy for continuous action spaces. A neural network, the actor, maps a state directly to an action. A second network, the critic, estimates how good that action is. The actor is improved by following the critic's gradient with respect to the action. Replay buffers and slowly updated target networks, borrowed from DQN, keep the learning stable enough to work with deep networks.

DDPG was one of the first deep reinforcement learning methods to handle continuous control well, and it is the base for later methods such as TD3 and, in spirit, SAC. It is now mostly used as a teaching example and a baseline, because it can be brittle and sensitive to hyperparameters. For work with tabular data, it shows the core pattern of learning continuous decisions from logged experience, with state features supplied by models that read tables.

What it is

DDPG combines two ideas:

• Deterministic policy gradient: for a policy that outputs one action per state, there is a clean formula for the gradient of expected return with respect to the policy parameters. It uses the gradient of the critic with respect to the action, multiplied by the gradient of the actor's output with respect to its parameters.
• Deep Q-learning machinery: a replay buffer, which stores past transitions, and target networks, which provide stable regression targets.

The main components are:

• An actor network, which takes a state and returns an action.
• A critic network, which takes a state and an action and returns a Q-value.
• Target actor and target critic networks, slow-moving copies.
• A replay buffer.
• An exploration noise process added to the actor's actions.

An everyday analogy

Think of a chef (the actor) who always cooks a single dish per situation, and a food critic (the critic) who tastes the dish and says how good it is. The chef cannot ask the critic what to change, but the critic can describe how the score would shift if the amount of salt went up or down. That description is the gradient with respect to the action. The chef then adjusts the recipe in the direction that raises the score. The chef also tries small random variations in practice sessions, which is the exploration noise.

Why a deterministic policy

For continuous actions, a stochastic policy needs a distribution, and computing the best action from a Q-function requires a maximization over a continuous space, which is hard. A deterministic actor sidesteps this: it directly outputs the action, and the critic's gradient tells it how to improve. This is efficient, and the policy is repeatable at test time.

Where it came from

• Q-learning (1989) and deep Q-networks (2013 and 2015) showed that a replay buffer and target network make deep value learning workable for discrete actions. See DQN.
• Deterministic policy gradient (2014) proved that the gradient of a deterministic policy has a simple form, which can be estimated off-policy.
• DDPG (2015), described in the paper "Continuous control with deep reinforcement learning", merged the two and showed learning on a range of simulated physical control tasks, in some cases directly from pixels.
• In the years after, the method was found to be brittle, which motivated improvements such as TD3 (2018), which fixed overestimation, and SAC (2018), which used a stochastic, entropy-regularized policy.

On-policy alternatives, such as PPO and TRPO, took a different route, trading sample efficiency for stability. For the wider field, see reinforcement learning.

How it works

Step 1: act with exploration noise

At each step the actor proposes an action for the current state. Noise is added to encourage exploration, and the result is clipped to the allowed range. The original work used a correlated noise process, the Ornstein-Uhlenbeck process, which produces smooth, drifting noise suited to physical systems with inertia. Simple uncorrelated Gaussian noise often works as well.

Step 2: store the transition

The tuple of state, action, reward, next state and done flag is saved in the replay buffer. Often the first few thousand steps use random actions to fill the buffer with varied data.

Step 3: sample a minibatch

A random minibatch is drawn from the buffer. Sampling at random breaks the correlation between consecutive steps and lets each transition be reused many times.

Step 4: build the critic target

For each sampled transition:

y = reward + gamma * (1 - done) * Q_target( next state, actor_target(next state) )

Here the target actor chooses the next action, and the target critic evaluates it. This is a one-step bootstrapped estimate of the value of taking the stored action.

Step 5: update the critic

The critic minimizes the mean squared difference between its prediction Q(state, action) and the target y.

Step 6: update the actor

The actor is updated to maximize the critic's value of the actor's own action:

maximize the average of Q(state, actor(state))

By the chain rule, the gradient flows from the critic's output, through the action input, into the actor's parameters. This is the deterministic policy gradient.

Step 7: soft-update the targets

After each update, each target network moves a small step toward its main network:

target = tau * main + (1 - tau) * target

with tau small, often 0.001 to 0.005. The slow movement keeps the regression target stable.

Step 8: repeat

The loop continues for as many steps as the budget allows.

Why it is off-policy

The critic target uses the actor's action at the next state, not the action stored in the buffer. Because the update does not depend on which policy produced the stored transitions, old data remains usable. This is what makes DDPG sample efficient compared with PPO or TRPO.

The overestimation problem

The critic is a neural network with approximation errors. The actor learns to find actions where the critic gives high values, which are often exactly the places where the critic's error is positive. The critic is then trained on targets that include these inflated values, which raises its estimates further. This loop causes overestimation bias and can lead to divergence. It is the central weakness addressed by TD3.

Key concepts and terms

A worked example

A numeric walk-through of one update

Suppose gamma = 0.99. A stored transition has reward 1.0, and it is not terminal. At the next state, the target actor proposes action 0.30, and the target critic evaluates that pair at 8.0. The target is

y = 1.0 + 0.99 * 8.0 = 8.92.

The main critic currently predicts 8.2 for the stored state and action. The squared error is (8.92 - 8.2)^2 = 0.5184. The gradient moves the critic toward 8.92.

For the actor, suppose at a state s the actor outputs action 0.30, and the critic's gradient with respect to the action there is +4.0. Increasing the action slightly raises the value, so the actor's output layer is nudged to output a larger action for s. If the gradient were negative, the actor would shift the action down. The step size is set by the learning rate and the actor's own gradient.

After the update, with tau = 0.005 and a target parameter of 1.000 while the main parameter is 1.200, the target becomes 0.005 * 1.200 + 0.995 * 1.000 = 1.001.

A compact PyTorch sketch

The sketch shows the update step. The environment loop and buffer are omitted.

import copy
import torch
import torch.nn as nn
import torch.nn.functional as F

class Actor(nn.Module):
    def __init__(self, obs_dim, act_dim, max_action, hidden=256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(obs_dim, hidden), nn.ReLU(),
            nn.Linear(hidden, hidden), nn.ReLU(),
            nn.Linear(hidden, act_dim), nn.Tanh())
        self.max_action = max_action

    def forward(self, s):
        return self.max_action * self.net(s)

class Critic(nn.Module):
    def __init__(self, obs_dim, act_dim, hidden=256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(obs_dim + act_dim, hidden), nn.ReLU(),
            nn.Linear(hidden, hidden), nn.ReLU(),
            nn.Linear(hidden, 1))

    def forward(self, s, a):
        return self.net(torch.cat([s, a], -1)).squeeze(-1)

class DDPG:
    def __init__(self, obs_dim, act_dim, max_action,
                 gamma=0.99, tau=0.005, lr_actor=1e-4, lr_critic=1e-3):
        self.actor = Actor(obs_dim, act_dim, max_action)
        self.actor_t = copy.deepcopy(self.actor)
        self.critic = Critic(obs_dim, act_dim)
        self.critic_t = copy.deepcopy(self.critic)
        self.a_opt = torch.optim.Adam(self.actor.parameters(), lr=lr_actor)
        self.c_opt = torch.optim.Adam(self.critic.parameters(), lr=lr_critic)
        self.gamma, self.tau = gamma, tau

    def update(self, batch):
        s, a, r, s2, d = batch
        with torch.no_grad():
            y = r + self.gamma * (1 - d) * self.critic_t(s2, self.actor_t(s2))

        c_loss = F.mse_loss(self.critic(s, a), y)
        self.c_opt.zero_grad(); c_loss.backward(); self.c_opt.step()

        a_loss = -self.critic(s, self.actor(s)).mean()
        self.a_opt.zero_grad(); a_loss.backward(); self.a_opt.step()

        with torch.no_grad():
            for net, tgt in ((self.actor, self.actor_t),
                             (self.critic, self.critic_t)):
                for p, pt in zip(net.parameters(), tgt.parameters()):
                    pt.mul_(1 - self.tau).add_(self.tau * p)

Exploration noise

import numpy as np

class OUNoise:
    def __init__(self, dim, mu=0.0, theta=0.15, sigma=0.2, dt=1e-2):
        self.mu, self.theta, self.sigma, self.dt = mu, theta, sigma, dt
        self.x = np.ones(dim) * mu

    def reset(self):
        self.x = np.ones_like(self.x) * self.mu

    def sample(self):
        dx = self.theta * (self.mu - self.x) * self.dt \
             + self.sigma * np.sqrt(self.dt) * np.random.randn(*self.x.shape)
        self.x = self.x + dx
        return self.x

A table-driven scenario

Imagine a pricing agent that sets a continuous discount level each day. The state includes predicted demand and price sensitivity from a tabular model, current stock and the day of the week. The reward is margin. DDPG learns a deterministic mapping from this state to a discount. It reuses each logged day many times, which is helpful because each day of real data is costly. The risk is that the critic overvalues unusual discounts that were rarely seen, so the policy may suggest extreme discounts. That risk is what TD3 was built to reduce.

Strengths and limitations

Strengths

• Handles continuous actions natively. No discretization is needed.
• Sample efficient. Off-policy learning with a replay buffer reuses data.
• Simple structure. An actor, a critic and their targets.
• Deterministic policy. Repeatable behavior and easy testing.
• Historical importance. It is the root of a family of methods, so understanding it helps with TD3 and SAC.

Limitations

• Brittle. It is sensitive to hyperparameters, initialization and random seed.
• Overestimation bias. The critic can inflate values, and the actor exploits them.
• Weak exploration. Added noise is a crude way to explore and can fail in sparse-reward tasks.
• Sensitive to reward scale. Large rewards can destabilize the critic.
• Continuous actions only. For discrete actions, use DQN or PPO.
• Superseded. TD3 and SAC are usually better in practice with little extra complexity.

How it compares with alternatives

In practice, treat DDPG as the starting point. If it is unstable on your task, switch to TD3, which is nearly a drop-in replacement, or to SAC.

Role in tabular foundation models

DDPG is not a model for tables. It is a method for learning continuous decisions. A tabular foundation model, by contrast, is pre-trained on many tables, often synthetic, and handles row and column structure, mixed numeric and categorical types, and missing values, without needing column names. At prediction time it uses in-context learning: labeled rows go in as context and predictions for new rows come out in one pass, with no weight updates. See TabPFN, /model/architecture and /model/research. DDPG and such models connect in these ways.

State features from tables

Many control problems start with tables: machine logs, order histories, price lists. A tabular foundation model can convert the most recent rows into forecasts, risk scores and class probabilities without retraining for each table. These outputs form a compact state for DDPG. A compact state keeps the actor and critic small, which helps because DDPG is sensitive and benefits from simpler learning problems. See predictive maintenance.

Logged tables as the replay buffer

A DDPG replay buffer is, in essence, a table of transitions with columns for state, action, reward, next state and done flag. Existing logs can seed the buffer. The caveat is that DDPG in its basic form cannot safely learn from a fixed table alone. The critic may overvalue actions absent from the data, and without new interaction nothing corrects it. Offline-oriented variants add constraints to keep the policy near the logged actions.

Learned simulators

If new interaction is expensive, a model fitted to logged tables can predict the next state and reward. DDPG can train against it. Deterministic actors are good at finding exact weak points in a simulator, so validate the final policy on held-out real data and use short simulated rollouts. See digital twins.

Missing values and imputation

State vectors from real tables contain gaps. A tabular foundation model can impute a missing cell in context and report uncertainty. The agent should see both the imputed value and a missingness flag. See imputation networks.

Anomaly detection as a guard

DDPG's exploration noise can push the system into unusual states. An anomaly detection model trained on normal operation can score how unusual a state is. The score can be an input, a penalty in the reward, or a trigger for a rule-based fallback.

Synthetic data and pre-training

Pre-training a tabular foundation model on synthetic tables depends on the settings of the table generator. These are normally chosen by hand or with Bayesian optimization. In a sequential setting, a continuous-control method could adjust them, though simpler tools usually suffice. See synthetic data generation.

Continual learning

A deployed policy and its tabular model face drift. A policy could decide how often to refresh the model, which data to replay and how large a step to take, with a reward balancing accuracy against cost. Replay buffer management plays a role in avoiding forgetting by holding a mix of old and new situations.

Sector and semantic classification

A tabular model can infer what a column means, or which sector a table belongs to, from values alone when names are missing. Those inferred labels can pick the action bounds, reward weights or sub-policy for the control agent.

An honest boundary

If the task is to predict a target column, use a tabular foundation model or a supervised learner. DDPG is relevant only when actions change future states and rewards arrive later. Even then, prefer TD3 or SAC for new work.

Practical guidance and best practices

Typical hyperparameters

Treat these as starting points.

Scale everything

Normalize observations with running statistics, keep rewards in a moderate range, and use actions scaled to from minus one to one with a final rescale at the environment boundary.

Handle episode endings correctly

Keep the bootstrap term for time-limit truncations, and drop it only for real terminal states.

Warm up the buffer

Use random actions for a few thousand steps, then start updates. This gives the first critic varied data.

Limit critic overestimation

If you must use DDPG, watch Q-values closely. If they grow far beyond plausible returns, lower the critic learning rate, rescale rewards, add weight decay, or move to TD3.

Monitor these signals

• Episode return and length.
• Critic loss.
• Mean and maximum Q-value.
• Action saturation: the share of actions at the bounds.
• Exploration noise scale and its effect on returns.

Evaluate properly

Evaluate without exploration noise, in a separate environment copy, over several episodes and seeds. Report the mean and the spread.

Example configuration

algo: ddpg
env: example-discount-control
seed: 9
gamma: 0.99
tau: 0.005
lr_actor: 0.0001
lr_critic: 0.001
batch_size: 128
buffer_size: 1000000
start_steps: 5000
noise: gaussian
noise_std: 0.1
hidden_sizes: [256, 256]
normalize_obs: true
eval_every: 5000
log_dir: /var/log/example.internal/ddpg-runs

Variants and extensions

• Prioritized replay samples important transitions more often.
• N-step returns speed up credit assignment, with care about off-policy bias.
• Hindsight experience replay relabels goals in goal-conditioned tasks, helping with sparse rewards.
• Distributed variants run many actors in parallel and feed a shared learner.
• Batch normalization or layer normalization in the networks can help with inputs of very different scales.
• TD3 and SAC are the main successors and fix its key weaknesses.

A debugging checklist

1. Test on a trivial environment where the best action is obvious.
2. Confirm that target networks do not receive gradients.
3. Confirm that done flags and truncation are handled separately.
4. Confirm that noise is added only during data collection, not in the update.
5. Plot the Q-values. Explosive growth signals overestimation.
6. Check for saturated actions at the bounds.
7. Reduce the critic learning rate or rescale rewards.
8. Try another seed. Large differences between seeds suggest brittleness.
9. Compare against a TD3 run on the same task.
10. Confirm that observation scaling is identical in training and evaluation.

Reading a DDPG training log

A healthy run shows a critic loss that is noisy but bounded, Q-values that grow slowly and stay in the range of observed returns, and returns that rise with occasional dips. Because the policy is deterministic, behavior can shift abruptly when the actor finds a new strategy, so step-like improvements are normal.

Warning signs include:

• Q-values that climb steadily past any achievable return. This is the classic sign of overestimation.
• Returns that rise and then collapse repeatedly. The actor may be exploiting critic errors.
• Actions pinned at the bounds. The tanh output may be saturated, or the reward may favor extreme actions.
• A critic loss that grows without bound. Lower the critic learning rate, rescale rewards or clip gradients.

Why DDPG results differ so much between runs

Several features add variance. The initial networks are random. The first critic is trained on a small, random buffer. The actor chases whatever the early critic prefers. Exploration noise is small compared with the space of actions, so two runs can explore different regions and settle on different behaviors. This is part of why repeated seeds matter and why more robust successors were developed.

From DDPG to modern practice

This table is also a quick way to decide what to try if a DDPG run is struggling.

Pre-deployment checks

• Evaluate the policy on a grid of states and review actions with domain experts.
• Stress test with missing or delayed inputs.
• Confirm the saved normalization statistics load with the policy.
• Define a rollback path.

A short learning path

1. Learn Q-learning on a small table of states and actions.
2. Read about DQN to see why replay and target networks help.
3. Implement DDPG on a simple one-dimensional control task.
4. Add the TD3 changes one at a time and see which helps most.
5. Try SAC and compare robustness over several seeds.
6. Explore on-policy methods such as PPO for contrast.

Governance for deployed policies

Because the policy is deterministic, you can test it on a grid of states and review its actions. Enforce action limits outside the policy, keep a rule-based fallback, roll out gradually, and log state, action and policy version.

Common mistakes

• Using DDPG when TD3 or SAC would do. The newer methods are usually more stable.
• Ignoring Q-value growth. Rapidly rising Q-values signal overestimation.
• Updating from an empty or tiny buffer. The first updates overfit.
• Wrong done handling. Treating truncation as termination biases the critic.
• Unscaled rewards or observations. Both harm critic learning.
• Adding noise inside the update. Exploration noise belongs in data collection only.
• Evaluating with exploration noise. This understates the policy.
• Letting targets receive gradients. Targets must change only by soft updates.
• Tiny replay buffers. The agent forgets older situations.
• Applying DDPG to discrete actions. Use DQN or PPO.
• Trusting a learned simulator without validation. Check against held-out real logs.
• Reporting one seed. Report the mean and spread over several.

Frequently asked questions

What does DDPG stand for?

DDPG stands for deep deterministic policy gradient. Deep refers to the use of neural networks. Deterministic means the policy outputs a single action for each state. Policy gradient means the policy is improved by following the gradient of expected return, here computed through a learned critic.

Is DDPG on-policy or off-policy?

DDPG is off-policy. It stores transitions in a replay buffer and learns from them repeatedly. Because the target uses the target actor's action rather than the stored action, data from earlier policies remains valid, which makes the method sample efficient.

Why is DDPG considered brittle?

The critic can overestimate values, and the actor finds and exploits those overestimates. This feedback loop can make learning diverge. DDPG is also sensitive to learning rates, reward scale, noise settings and random seeds, so results can vary widely from one run to another.

How is TD3 related to DDPG?

TD3 is a direct improvement of DDPG. It keeps the actor, critic, target networks and replay buffer, and adds twin critics with a minimum, delayed actor updates and target policy smoothing. These changes tackle the overestimation problem and make training far more stable.

How does DDPG differ from SAC?

DDPG learns a deterministic policy and explores by adding noise. SAC learns a stochastic policy with an entropy bonus, uses twin critics and can tune its temperature automatically. SAC is usually more robust across tasks, while DDPG is simpler but more fragile.

Why does DDPG use target networks?

If the same network produced both the prediction and the target, every update would shift the target too, and learning could oscillate or diverge. Target networks change slowly through soft updates, so the regression target stays nearly fixed over a number of steps, which stabilizes training.

What exploration noise does DDPG use?

The original work used the Ornstein-Uhlenbeck process, which gives smooth, correlated noise suited to physical systems. Many later implementations use simple Gaussian noise, which often works equally well. The scale is typically a fraction of the action range, and evaluation should be done without noise.

Can DDPG handle discrete actions?

Not directly. It needs gradients of the critic with respect to a continuous action. For discrete actions, use a value-based method such as DQN or a policy-gradient method such as PPO. Relaxations exist, but they are not the standard approach.

How does DDPG relate to tabular foundation models?

DDPG learns continuous control decisions, while a tabular foundation model reads tables and predicts or imputes in context. The tabular model can provide state features, anomaly scores, imputed values and a learned simulator. DDPG, or better its successors, can learn the policy that acts on those outputs.

Should I still use DDPG today?

It is useful for learning and as a baseline, but for new work TD3 or SAC are usually better choices. They keep the same overall structure and add features that remove the main failure modes, with little extra code.

Glossary

• Actor: the network that outputs the action.
• Bellman equation: the recursive relation between values at consecutive steps.
• Critic: the network that estimates the value of a state and action.
• Deterministic policy: a policy giving one action per state.
• Exploration noise: noise added to actions while collecting data.
• Off-policy: learning from data produced by earlier policies.
• Ornstein-Uhlenbeck process: a noise process with temporal correlation.
• Overestimation bias: systematic upward error in value estimates.
• Replay buffer: a store of past transitions.
• Soft update: gradual movement of a target network toward the main network.
• Target network: a slowly changing copy used for stable targets.

Summary

DDPG is an off-policy actor-critic algorithm that learns a deterministic policy for continuous actions. It combines the deterministic policy gradient with a replay buffer and target networks, which makes it sample efficient and conceptually simple. Its main weaknesses are brittleness and overestimation bias.

Those weaknesses led to TD3, which adds twin critics, delayed updates and target smoothing, and to SAC, which adds a stochastic policy and an entropy bonus. On-policy methods such as PPO and TRPO trade sample efficiency for stability.

Around tabular foundation models, DDPG illustrates the decision-learning side. The tabular model supplies state features, imputations, anomaly scores and learned simulators, and the control algorithm learns how to act on them. For new work prefer its successors. See /model/research, /model/architecture and /model/benchmarks for the model side.

Related

It belongs to Reinforcement learning in my stack, under AI and machine learning. Others in the same group: