A2C

Overview

A2C, short for advantage actor-critic, is a reinforcement learning algorithm that trains two parts together. The actor is a policy that chooses actions. The critic is a value function that judges how good states are. The critic lets the actor learn from an advantage, which says how much better an action was than expected. This is much less noisy than the raw return used in REINFORCE.

The "synchronous" part means many copies of the environment run in parallel, and one update is made after all of them finish a short rollout. This makes good use of a GPU and gives stable, repeatable training. A2C is simple, a solid baseline for discrete and continuous control, and the direct ancestor of PPO. It is also the synchronous cousin of A3C, which updates asynchronously.

What it is

A2C is an on-policy actor-critic method. It has two outputs, often from a shared network body:

• The actor outputs a distribution over actions, pi(a | s).
• The critic outputs a single number, V(s), the expected return from state s.

The training signal is the advantage:

A(s, a) = (what actually happened) - (what the critic expected)

If the advantage is positive, the action was better than expected and its probability goes up. If negative, its probability goes down. The critic is trained to predict returns more accurately so the advantage stays informative.

An everyday analogy

Imagine a coach and a player. The player (actor) chooses moves. The coach (critic) has a feel for how well the game is going at any moment. After a move, the coach does not say "good" or "bad" in absolute terms. The coach says "that went better than I expected" or "worse than I expected". That relative feedback is the advantage. It is much easier to learn from than raw match scores because it removes the noise of the overall game situation.

What "synchronous" means

Several workers each run their own copy of the environment. Each collects the same number of steps. Then all the data is combined into one batch, and one gradient update is done. Nobody updates while others are still collecting. This is different from A3C, where each worker updates the shared model on its own schedule.

Core components

Where it came from

Actor-critic ideas are old. Early work in the 1980s on adaptive critics and temporal difference learning combined a policy and a value estimate. The policy gradient theorem around 1999 and 2000 gave the formal basis for using a critic to guide a policy.

In 2016, a paper on asynchronous methods for deep reinforcement learning introduced A3C, which ran many workers that updated shared parameters asynchronously on CPUs. Researchers soon noticed that the asynchrony was not essential. Collecting data from workers and updating once in a synchronous batch worked as well or better, and it used a GPU more efficiently. This synchronous version was named A2C. The name came from a public write-up and software release rather than a stand-alone paper.

A2C then became a standard baseline. Trust-region ideas from TRPO and the clipped objective in PPO were built on top of the actor-critic setup. Many software libraries still ship A2C as a core algorithm because it is easy to read and fast to run.

How it works

Step 1: set up parallel environments

Create N copies of the environment, often between 8 and 64 for small tasks. Each copy starts from its own initial state. Stacking their states gives a batch.

Step 2: collect a short rollout

For T steps (often 5 to 20), do this in every environment:

1. Feed the state to the actor and sample an action.
2. Step the environment and get the next state, reward and done flag.
3. Store the state, action, reward, done flag and the critic's value estimate.

After T steps, bootstrap from the critic's estimate of the last state.

Step 3: compute returns

For each step, compute an n-step return from the end of the rollout backwards:

R_t = r_t + gamma * R_(t+1)

with R at the end set to the critic's value of the final state (or zero if the episode ended). This mixes real rewards with a learned estimate of the future, which is the key difference from REINFORCE's full Monte Carlo return.

Step 4: compute advantages

The simplest advantage is:

A_t = R_t - V(s_t)

A more refined version, generalized advantage estimation, blends several n-step estimates with a parameter lambda to trade bias against variance. Many modern implementations use it even though plain A2C is defined with n-step returns.

Step 5: compute the three losses

• Policy loss: the negative of (advantage times log-probability of the taken action). The advantage is treated as a fixed number here.
• Value loss: the squared error between V(s_t) and the return R_t.
• Entropy term: the negative entropy of the policy, which discourages collapse.

The total loss is:

loss = policy_loss + c_v * value_loss - c_e * entropy

Step 6: update once

Back-propagate through the batch of all workers and take one optimizer step. Gradient clipping is common.

Step 7: repeat

Use the updated network to collect the next rollouts. Old data is discarded because A2C is on-policy.

Why a critic helps

REINFORCE must wait until an episode ends and uses a full noisy return. The critic replaces most of that noise with a learned estimate, so the actor can learn from short rollouts. The price is some bias, because the critic is imperfect. Good practice keeps the bias small by training the critic well.

The bias and variance trade

Shared versus separate networks

A shared body saves compute and can help learn common features, but the two losses can fight each other. Separate networks avoid interference at the cost of more parameters. For tabular or low-dimensional states, separate small networks are a safe default.

Why parallel environments help

Samples from one environment are strongly correlated in time. Samples from many environments at once are more diverse, which acts like a decorrelating step. This role is played by the replay buffer in off-policy methods such as DQN, but A2C gets it from parallelism instead, so it needs no buffer.

Key concepts and terms

A worked example

This example builds a compact A2C agent for a discrete-action task using PyTorch. It uses a vectorized environment with N copies.

The network

import torch
import torch.nn as nn
import torch.nn.functional as F

class ActorCritic(nn.Module):
    def __init__(self, state_dim, n_actions, hidden=64):
        super().__init__()
        self.pi = nn.Sequential(
            nn.Linear(state_dim, hidden), nn.Tanh(),
            nn.Linear(hidden, n_actions),
        )
        self.v = nn.Sequential(
            nn.Linear(state_dim, hidden), nn.Tanh(),
            nn.Linear(hidden, 1),
        )

    def forward(self, s):
        logits = self.pi(s)
        value = self.v(s).squeeze(-1)
        return logits, value

The rollout and update

def a2c_update(model, opt, states, actions, rewards, dones, last_value,
               gamma=0.99, c_v=0.5, c_e=0.01):
    # shapes: states (T, N, D), actions (T, N), rewards (T, N), dones (T, N)
    T, N = rewards.shape
    returns = torch.zeros(T, N)
    R = last_value                       # shape (N,)
    for t in reversed(range(T)):
        R = rewards[t] + gamma * R * (1.0 - dones[t])
        returns[t] = R

    logits, values = model(states.reshape(T * N, -1))
    values = values.view(T, N)
    dist = torch.distributions.Categorical(logits=logits.view(T, N, -1))
    logp = dist.log_prob(actions)
    entropy = dist.entropy().mean()

    adv = (returns - values).detach()
    policy_loss = -(adv * logp).mean()
    value_loss = F.mse_loss(values, returns)
    loss = policy_loss + c_v * value_loss - c_e * entropy

    opt.zero_grad()
    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 0.5)
    opt.step()
    return loss.item()

The training loop

model = ActorCritic(state_dim=4, n_actions=2)
opt = torch.optim.RMSprop(model.parameters(), lr=7e-4, alpha=0.99, eps=1e-5)
states = envs.reset()                          # N parallel environments

for update in range(20000):
    S, A, R, D = [], [], [], []
    for _ in range(5):                         # rollout length T = 5
        s = torch.as_tensor(states, dtype=torch.float32)
        with torch.no_grad():
            logits, _ = model(s)
        a = torch.distributions.Categorical(logits=logits).sample()
        states, r, d, _ = envs.step(a.numpy())
        S.append(s); A.append(a)
        R.append(torch.as_tensor(r, dtype=torch.float32))
        D.append(torch.as_tensor(d, dtype=torch.float32))
    with torch.no_grad():
        _, last_v = model(torch.as_tensor(states, dtype=torch.float32))
    a2c_update(model, opt, torch.stack(S), torch.stack(A),
               torch.stack(R), torch.stack(D), last_v)

A numeric walk-through of the advantage

Take one worker with a rollout of three steps, gamma = 0.9. Rewards are 1, 0, 1. The critic's value for the state after step three is 4. The critic's value estimates for the three starting states are 5.0, 4.5 and 4.2.

• R_3 = 1 + 0.9 * 4 = 4.6
• R_2 = 0 + 0.9 * 4.6 = 4.14
• R_1 = 1 + 0.9 * 4.14 = 4.726

Advantages:

The third action did better than the critic expected, so it is reinforced. The first two did slightly worse, so they are slightly discouraged. The critic also moves its estimates toward the returns.

What to expect

On a small control task, A2C usually learns faster per wall-clock second than REINFORCE and with smoother curves. It is more sensitive to the learning rate than value-based methods, and results vary between seeds. For stronger stability with the same setup, switch to PPO.

Generalized advantage estimation in code

def compute_gae(rewards, values, dones, last_value, gamma=0.99, lam=0.95):
    T, N = rewards.shape
    adv = torch.zeros(T, N)
    gae = torch.zeros(N)
    next_v = last_value
    for t in reversed(range(T)):
        mask = 1.0 - dones[t]
        delta = rewards[t] + gamma * next_v * mask - values[t]
        gae = delta + gamma * lam * mask * gae
        adv[t] = gae
        next_v = values[t]
    returns = adv + values
    return adv, returns

Here lambda controls the blend. A value near zero gives the one-step estimate with low variance and more bias. A value near one approaches the full Monte Carlo return with high variance and little bias.

Continuous actions in A2C

For continuous control, replace the categorical head with a Gaussian head.

class GaussianActor(nn.Module):
    def __init__(self, state_dim, action_dim, hidden=64):
        super().__init__()
        self.mu = nn.Sequential(
            nn.Linear(state_dim, hidden), nn.Tanh(),
            nn.Linear(hidden, action_dim),
        )
        self.log_std = nn.Parameter(torch.zeros(action_dim))

    def dist(self, s):
        return torch.distributions.Normal(self.mu(s), self.log_std.exp())

The log-probability of a sampled action is summed over action dimensions. The rest of the update is unchanged.

A mental model of the three losses

If you watch only the total loss you will miss problems. A falling total can hide a critic that has stopped improving, or an entropy that has collapsed.

Rollout length and update frequency

Short rollouts mean more frequent updates and more reliance on the critic. Long rollouts mean fewer updates and more reliance on real rewards. The total number of samples per update is the rollout length times the number of environments, so both knobs set the batch size. Doubling the number of environments halves the noise of the gradient roughly in the same way as doubling the rollout length, but costs more parallel compute rather than more time per update.

Strengths and limitations

Strengths

• Lower variance than REINFORCE. The critic replaces noisy full returns with a learned estimate.
• Online updates. Learning can start after a few steps, not after a full episode.
• Simple and readable. The algorithm has few parts, so it is easy to implement and debug.
• GPU friendly. Batching many environments makes good use of parallel hardware.
• Works for discrete and continuous actions. The actor can output a categorical or Gaussian distribution.
• No replay buffer. Memory use is small.
• Reproducible. Synchronous updates avoid the non-determinism of asynchronous workers.

Limitations

• On-policy and sample inefficient. Each batch is used once, so it needs many environment steps.
• Sensitive to hyperparameters. Learning rate, rollout length and entropy weight matter.
• Critic bias. A poor critic gives misleading advantages.
• No update safeguard. A large step can damage the policy. TRPO and PPO add safeguards.
• Needs many parallel environments. If environments are expensive, this is a bottleneck.
• Can get stuck. Policy collapse or poor local optima happen when entropy drops too fast.

How it compares with alternatives

A2C and A3C differ mainly in how they use workers. A3C updates a shared model asynchronously and was designed for CPUs. A2C waits for all workers and then updates once, which suits GPUs and gives cleaner, more repeatable results. In practice A2C is usually the easier choice.

Role in tabular foundation models

A2C is a reinforcement learning method and does not read tables directly. Still, it connects to models that learn from raw tables in practical ways. A tabular foundation model such as TabPFN is pre-trained on many synthetic or real tables and predicts on a new one using in-context learning, with no per-table training. It copes with row and column structure, mixed numeric and categorical types, and missing values, and it does not need meaningful column names. See /model/research and /model/architecture.

Decisions over tables

Many tabular problems are decisions in sequence: choose the next record to review, the next column to clean, the next treatment to offer, the next model to try. These have delayed rewards and often a discrete action set. An actor-critic is a natural fit: the actor picks the action and the critic estimates how much value remains.

The encoder carries the table

A2C needs a state vector. A tabular encoder, for instance one with column embeddings as in FT-Transformer, or a representation taken from a pre-trained tabular model, converts raw rows into that vector. This encoder takes care of mixed types and missing values. The actor and critic heads can then be small.

Imputation and cleaning pipelines

When imputation or cleaning is a sequence of choices, the reward is the downstream quality of a tabular model. The critic learns how valuable the current state of the table is. The advantage tells which cleaning choice beat expectation. Because tabular foundation models predict without retraining, the reward can be computed quickly, which speeds up A2C training.

Anomaly triage

In anomaly detection operations, the action may be flag, hold or release. Rewards come from analyst confirmation or from the cost of misses and false alarms. A critic estimates the expected cost of the current queue state, and the actor learns to prioritize.

Semantic and sector context

If a tabular model infers a column's semantic type or a table's sector without column names, those inferences can be part of the state. The actor can then learn sector-specific behavior, for example that one kind of table needs conservative outlier handling and another does not.

Continual learning

Because A2C is on-policy, it adapts to the current data stream. It can follow drift in tables but has no memory of old data. A small evaluation set from older periods helps detect forgetting.

Honest limits

A2C is not pre-trained on tables and has no in-context learning. It is an optimizer for policies around a table model. Its fit is strongest when the decisions are sequential and the reward is measurable. For table model results, see /model/benchmarks.

Practical guidance and best practices

Start with sensible defaults

Normalize inputs and rewards

Standardize observations. Scale or clip rewards if they vary widely. A critic that sees huge or tiny targets trains badly.

Watch the entropy

If entropy falls to near zero early, the policy has become deterministic before it has explored. Raise the entropy weight or lower the learning rate.

Check the critic

Track the explained variance of the critic. If it stays near zero, the critic is not learning and the advantages are noise.

Use generalized advantage estimation

It often improves results at no cost. A lambda around 0.95 is a common choice.

Keep rollouts short but not too short

Very short rollouts are noisy and rely on the critic. Very long rollouts bring back the variance of Monte Carlo returns. Five to twenty steps is a typical balance.

Seeds and logging

Use several seeds and log the mean return, entropy, value loss, policy loss and gradient norm.

When to move on

If training is unstable, try PPO. If you need sample efficiency, try SAC or TD3 for continuous actions, or Rainbow for discrete actions.

Tuning guide

Debugging order

1. Verify that the environment returns sensible states and rewards.
2. Train on a trivial task and confirm the agent solves it.
3. Check shapes of every tensor in the return computation.
4. Confirm the done mask is applied in the bootstrap.
5. Compare against a REINFORCE baseline on the same task.
6. Add GAE, then entropy decay, then any other extras, one at a time.

Checklist

• Parallel environments set up
• Critic target uses bootstrapping with done masks
• Advantage detached in the policy loss
• Entropy and clipping in place
• Several seeds compared

Common mistakes

1. Letting gradients flow through the advantage. The advantage must be a constant in the policy loss.
2. Forgetting the done mask. Bootstrapping across an episode end leaks value from the next episode.
3. Mixing up the shapes. Time and environment dimensions get swapped, and the returns are computed across the wrong axis.
4. Weighting the value loss too high. With a shared body, the critic can dominate and harm the policy.
5. No entropy bonus. The policy collapses early.
6. Too few environments. Correlated samples make training unstable.
7. Reusing old rollouts. A2C is on-policy, and stale data breaks its assumptions.
8. Poor reward scale. Large rewards cause huge critic losses and gradients.
9. Comparing to a baseline with different environment steps. Always compare at equal numbers of environment steps.
10. Single-seed conclusions. Policy gradient methods vary widely between runs.

Frequently asked questions

What does A2C stand for?

A2C stands for advantage actor-critic. It trains an actor, which chooses actions, and a critic, which estimates the value of states. The advantage is the difference between what happened and what the critic expected, and it drives the actor's updates.

How is A2C different from A3C?

Both are actor-critic methods with parallel workers. A3C lets each worker update shared parameters asynchronously, while A2C waits for all workers to finish a rollout and makes one combined update. A2C is easier to reproduce and uses GPUs more efficiently, and it often matches or beats A3C.

How is A2C different from REINFORCE?

REINFORCE uses the full sampled return and has high variance. A2C adds a critic that estimates state values, which lets it use short rollouts and lower-variance advantages. The cost is some bias from the critic's errors.

What is the advantage function?

The advantage measures how much better an action was than the average expected outcome in that state. In A2C it is estimated as the return minus the critic's value. Positive values increase the action's probability and negative values decrease it.

Is A2C on-policy?

Yes. It uses data collected by the current policy and discards it after one update. This makes it simple and stable but less sample efficient than off-policy methods that use a replay buffer.

Why use several parallel environments?

Parallel environments provide diverse, less correlated samples and make good use of hardware by batching. They play a role similar to a replay buffer in off-policy methods, giving decorrelated training data without storing it.

Can A2C handle continuous actions?

Yes. The actor outputs the parameters of a continuous distribution, such as a Gaussian mean and spread, and the log-probability of the sampled action is used in the policy loss. The same critic and advantage logic apply.

How does A2C relate to PPO?

PPO is built on the same actor-critic structure but replaces the plain policy loss with a clipped objective that limits how far the policy can move per update, and it reuses each batch for a few epochs. For many tasks PPO is more stable than A2C.

Can A2C be used with tabular data?

Yes, when the task is a sequence of decisions over table states. A tabular encoder produces the state vector, and the actor and critic learn from reward. It does not replace a tabular predictor for standard classification or regression.

What are the main hyperparameters to tune?

The learning rate, the number of parallel environments, the rollout length, the entropy weight and the value loss weight matter most. Gradient clipping and the discount factor also affect stability. Start with common defaults and change one setting at a time.

Glossary

• Actor: The part of the agent that chooses actions.
• Advantage: Return minus the critic's estimate; the relative value of an action.
• Bootstrapping: Using an estimate of future value in a target.
• Critic: The part of the agent that estimates state value.
• Done mask: A flag that stops bootstrapping at the end of an episode.
• Entropy bonus: A reward for randomness that keeps exploration alive.
• Explained variance: A measure of how well the critic predicts returns.
• GAE: Generalized advantage estimation, a blend of n-step advantages.
• N-step return: Real rewards for n steps plus a bootstrapped value.
• On-policy: Learning only from data made by the current policy.
• Rollout: A short sequence of steps collected from an environment.
• Synchronous update: An update after all workers finish collecting.
• Value function: A function that maps a state to expected return.

Summary

A2C is the synchronous advantage actor-critic algorithm. An actor chooses actions, a critic estimates state values, and the advantage tells the actor how much better each action was than expected. Parallel environments and short rollouts give lower variance than REINFORCE and efficient use of hardware.

It is a simple, readable baseline and the base for PPO. Its limits are on-policy sample inefficiency and sensitivity to hyperparameters. In tabular settings, pair it with an encoder such as FT-Transformer or representations from TabPFN, and use it for sequential decisions such as imputation order or anomaly triage. Compare it with A3C, TRPO, SAC and reinforcement learning in general, and read /model/research, /model/architecture and /model/benchmarks for model details.

Related

It belongs to Reinforcement learning in my stack, under AI and machine learning. Others in the same group: