A3C

Overview

A3C, short for asynchronous advantage actor-critic, is a reinforcement learning algorithm that runs many learner threads at once. Each thread has its own copy of the environment and a local copy of the network. Each one collects a short rollout, computes gradients, and pushes them to a shared global network without waiting for the others. Then it pulls fresh weights and continues.

This design made deep reinforcement learning practical on ordinary multi-core CPUs, with no GPU and no replay buffer. The many independent workers see different parts of the environment at the same time, which decorrelates the data and stabilizes training. A3C introduced the advantage actor-critic recipe that later became A2C, the synchronous version that is now more common, and it influenced PPO. This article explains how A3C works, how it differs from A2C, and how it connects to tabular foundation models.

What it is

A3C is an on-policy actor-critic algorithm with asynchronous parallel workers. It has these elements:

• An actor that outputs action probabilities.
• A critic that estimates the value of a state.
• An advantage computed from n-step returns and the critic.
• Many workers, each with its own environment and a local copy of the network.
• A global network that all workers update and read.

An everyday analogy

Picture a team of explorers mapping a large area. Each explorer walks a different path and, every so often, calls the base camp to update the shared map and to download the latest version. Nobody waits for anyone else. The shared map improves quickly because many paths are explored at once, and no single explorer's route dominates. Some explorers will work from a slightly older map than others, and that is accepted as the price of speed.

The three A's

• Asynchronous: workers update the shared model on their own schedules.
• Advantage: the learning signal is return minus the critic's estimate.
• Actor-critic: one network picks actions and another part judges states.

Why it mattered

Before A3C, deep reinforcement learning leaned on experience replay and GPUs to break the correlation in training data. A3C showed that parallel actors could do the same decorrelation job. Replay was not required, so memory use dropped, and the method worked well on a CPU-only machine.

Where it came from

The A3C paper, "Asynchronous Methods for Deep Reinforcement Learning," appeared in 2016. It presented asynchronous versions of several algorithms: one-step Q-learning, n-step Q-learning, SARSA and advantage actor-critic. The actor-critic version, A3C, performed best and became the headline method.

The work built on earlier threads:

• Actor-critic methods from the 1980s and 1990s, and the policy gradient theorem around 1999 and 2000.
• DQN (2013 and 2015), which used a replay buffer to stabilize deep Q-learning.
• Lock-free parallel optimization research, which showed that gradient updates can be applied without locking, with some noise from stale parameters.

After the paper, practitioners found that the asynchrony was not the source of the gain. Parallelism was. A synchronous variant that waits for all workers and updates once, later called A2C, matched or exceeded A3C and used GPUs better. Still, A3C stayed influential, and its ideas of parallel actors, short rollouts and shared parameters continue in distributed systems such as importance-weighted actor-learner architectures. See also PPO and TRPO for later on-policy methods.

How it works

Step 1: create the global network

There is one global set of parameters theta for the actor and the critic, usually in a shared body with two heads. An optimizer with shared statistics, such as RMSprop, holds running averages in shared memory.

Step 2: start the workers

Launch N threads or processes, often equal to the number of CPU cores. Each has:

• its own copy of the environment, started from a different seed,
• a local copy of the network, theta', synchronized from the global network.

Step 3: collect a short rollout

Each worker plays up to t_max steps, often 5 to 20, or until the episode ends. It stores states, actions, rewards and the critic's value estimates, using its local network.

Step 4: compute n-step returns

Starting from the end of the rollout, compute:

R = 0 if the episode ended, otherwise V(s_last)

and then for each step from last to first:

R = r_i + gamma * R

This mixes real rewards with a bootstrapped estimate at the cut point.

Step 5: compute local gradients

For each step i in the rollout, the gradient has two parts:

• Policy part: gradient of log pi(a_i | s_i) times (R_i - V(s_i)). The advantage is a constant here.
• Value part: gradient of (R_i - V(s_i))^2 with respect to the critic.

An entropy bonus is added to the policy part to encourage exploration.

Step 6: apply to the global network, asynchronously

The worker sends its gradients to the global network and applies them immediately with the shared optimizer. No locks guarantee consistency. If another worker updated the global weights in the meantime, those updates are simply added in.

Step 7: synchronize and repeat

The worker copies the new global parameters into its local network and starts another rollout. The cycle continues until a training budget is spent.

Why asynchrony stabilizes learning

At any moment, the workers are in different parts of their episodes and have slightly different policies. Their gradients therefore come from diverse situations. Averaging over them, even asynchronously, has a similar effect to sampling from a replay buffer: it breaks the strong correlation between consecutive samples.

Stale gradients

A worker computes gradients with parameters that may be a few updates old by the time it applies them to the global network. This staleness adds noise. With a modest number of workers and small learning rates, it is usually tolerable and sometimes even helps by adding exploration.

The A3C loop at a glance

How it differs from A2C

Why a shared optimizer

If each worker kept its own optimizer statistics, the running averages would be noisy and disagree. A shared optimizer pools statistics from all workers, which gives a smoother estimate of gradient scale. In many implementations, this shared RMSprop was found to be more robust than separate ones.

Key concepts and terms

A worked example

This example shows a compact, CPU-only A3C setup in Python using processes and shared memory. It keeps the structure clear and leaves out production details.

The shared model

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.multiprocessing as mp

class ActorCritic(nn.Module):
    def __init__(self, state_dim, n_actions, hidden=128):
        super().__init__()
        self.body = nn.Sequential(nn.Linear(state_dim, hidden), nn.ReLU())
        self.pi = nn.Linear(hidden, n_actions)
        self.v = nn.Linear(hidden, 1)

    def forward(self, s):
        h = self.body(s)
        return self.pi(h), self.v(h).squeeze(-1)

global_model = ActorCritic(state_dim=4, n_actions=2)
global_model.share_memory()
opt = torch.optim.RMSprop(global_model.parameters(), lr=1e-4, alpha=0.99, eps=1e-5)

The worker function

def worker(rank, global_model, opt, make_env, total_updates, t_max=5, gamma=0.99):
    torch.manual_seed(rank)
    env = make_env(seed=rank)
    local = ActorCritic(4, 2)
    s = env.reset()

    for _ in range(total_updates):
        local.load_state_dict(global_model.state_dict())
        logps, values, rewards, entropies = [], [], [], []
        done = False

        for _ in range(t_max):
            logits, v = local(torch.as_tensor(s, dtype=torch.float32))
            dist = torch.distributions.Categorical(logits=logits)
            a = dist.sample()
            s, r, done, _ = env.step(a.item())
            logps.append(dist.log_prob(a)); values.append(v)
            rewards.append(r); entropies.append(dist.entropy())
            if done:
                s = env.reset()
                break

        R = torch.tensor(0.0) if done else local(torch.as_tensor(s, dtype=torch.float32))[1].detach()
        policy_loss, value_loss = 0.0, 0.0
        for i in reversed(range(len(rewards))):
            R = rewards[i] + gamma * R
            adv = R - values[i]
            value_loss = value_loss + 0.5 * adv.pow(2)
            policy_loss = policy_loss - logps[i] * adv.detach() - 0.01 * entropies[i]

        opt.zero_grad()
        (policy_loss + 0.5 * value_loss).backward()
        nn.utils.clip_grad_norm_(local.parameters(), 40.0)
        for lp, gp in zip(local.parameters(), global_model.parameters()):
            gp._grad = lp.grad                     # hand local gradients to the global model
        opt.step()

Launching the workers

if __name__ == "__main__":
    procs = []
    for rank in range(8):
        p = mp.Process(target=worker,
                       args=(rank, global_model, opt, make_env, 20000))
        p.start(); procs.append(p)
    for p in procs:
        p.join()

A numeric look at one worker update

Let t_max = 3, gamma = 0.9, and the worker plays three steps with rewards 0, 0, 1 without the episode ending. The critic's estimate for the state after step three is 2.0.

• R after step 3: 1 + 0.9 * 2.0 = 2.8
• R after step 2: 0 + 0.9 * 2.8 = 2.52
• R after step 1: 0 + 0.9 * 2.52 = 2.268

If the critic's estimates for the three starting states are 2.0, 2.2 and 1.5, the advantages are 0.268, 0.32 and 1.3. The last action looks best and is reinforced most. The same gradient also trains the critic toward the returns.

What to expect

On small tasks, eight workers on a multi-core CPU reach good scores quickly in wall-clock time. Results vary between runs because thread timing differs. For a reproducible, GPU-friendly alternative, use A2C.

When to choose it

• You have many CPU cores and no GPU.
• Environment steps are cheap and easy to run in separate processes.
• The network is small, such as a few dense layers for a table state.
• You can tolerate some run-to-run variation.

When to choose something else

• You need exact reproducibility, where A2C is better.
• Your network is large and benefits from GPU batching.
• You need sample efficiency, where off-policy methods such as SAC or Rainbow fit better.
• You want a robust default, where PPO is the common choice.

Design choices inside A3C

Variations on the theme

LSTM version

For partly observable tasks, add an LSTM layer between the body and the heads. Each worker carries its own hidden state through the rollout and resets it at episode end. Truncated backpropagation covers only the current rollout.

Continuous-action version

Replace the categorical head with a Gaussian head. Each worker samples continuous actions, and the policy loss uses the log-probability of the sampled action.

GPU-batched variants

Later systems keep the idea of many parallel actors but gather their experience on a GPU learner, which brings back the efficiency of batching. These designs sit between A3C and A2C in spirit.

Why parallel actors act like exploration

If every worker used the same exploration setting and started from the same state, they would collect near-identical data. Real implementations vary seeds and sometimes the entropy weight, so different workers behave differently. The shared model then learns from a wide range of behavior in each update window.

What happens when a worker is slow

In A3C a slow worker simply contributes fewer updates. Its gradients may be staler when they arrive, but nobody waits for it. In A2C a slow worker holds up the whole batch. This is a practical reason A3C suits heterogeneous hardware, and a practical reason A2C needs evenly sized workers.

A small thought experiment on speed

Suppose each environment step takes a fixed amount of time and one worker produces one step per unit time. Eight workers produce about eight steps per unit time, so the same number of samples arrives in an eighth of the wall-clock time. The cost is that each gradient is slightly more stale and the run is less repeatable. For cheap simulations that trade is often a good one.

Strengths and limitations

Strengths

• Runs on CPUs. No GPU is required, so it suits commodity multi-core machines.
• No replay buffer. Memory use is small, and there is no buffer to tune.
• Decorrelated data. Different workers explore different states at once.
• Fast wall-clock learning. More cores mean more samples per second.
• Simple model of parallelism. Workers share one set of parameters.
• Handles discrete and continuous actions. The actor can use a categorical or Gaussian head.
• Built-in exploration variety. Different workers can use different exploration settings.

Limitations

• On-policy and sample inefficient. Data is used once, so many environment steps are needed.
• Non-deterministic. Thread timing changes results, which makes debugging and comparison harder.
• Stale gradients. Asynchronous updates add noise.
• Poor GPU use. The design favors many small CPU workers, not large batches.
• Superseded in practice. A2C and PPO are usually easier and as good or better.
• Implementation complexity. Shared memory, process management and optimizer sharing need care.
• Hyperparameter sensitivity. Learning rate and entropy weight matter a lot.

How it compares with alternatives

The simplest advice: choose A3C only if you have many CPU cores, cheap environment steps, and a reason to avoid GPU batching. Otherwise choose A2C for simplicity or PPO for robustness.

Role in tabular foundation models

A3C is a reinforcement learning method and does not read tables directly. It still relates to models that learn from raw tables in concrete ways. A tabular foundation model such as TabPFN is pre-trained on many synthetic or real tables and predicts on a new table through in-context learning. It handles row and column structure, mixed numeric and categorical types and missing values, and it does not depend on column names. See /model/research and /model/architecture.

Many cheap simulations

A3C fits settings where environment steps are cheap and plentiful. A table-driven simulation, such as replaying logged records through a decision process, can be run on many CPU cores at once. Each worker plays through a different slice of the log. That is exactly the shape A3C was built for.

Decisions around a table model

The agent chooses actions such as which record to review, which column to clean, or which model to run. A tabular encoder, for example FT-Transformer style embeddings or a pre-trained tabular representation, turns rows into the state vector. The actor and critic heads sit on top.

Imputation and data preparation

When imputation is a sequence of decisions, the reward is the downstream score of a tabular predictor. Because tabular foundation models predict in one forward pass without retraining, reward evaluation is fast, and many workers can evaluate in parallel. This matches the asynchronous design.

Anomaly handling

In anomaly detection, the action may be to flag, hold or pass a row. Workers can each process a different stream or time window. The critic estimates the expected cost of the current queue, and the actor learns a triage policy.

Semantic and sector context

When a tabular model infers column meaning or sector from values alone, those labels can be part of the state. Workers on different sectors give the shared model exposure to varied domains at the same time, in a way that mirrors how foundation models are pre-trained on many different tables.

Continual learning

Because workers continuously feed fresh samples, A3C adapts to a shifting stream of tables. It has no replay memory, so it forgets older regimes unless workers keep sampling them. Assigning some workers to older data is a simple guard.

Honest limits

A3C is not pre-trained on tables, and it has no in-context learning. In modern practice, synchronous methods such as A2C and PPO are more common for the same roles. A3C is best seen as the historical origin of parallel actor-critic training. For model results, see /model/benchmarks.

Practical guidance and best practices

Match workers to cores

Use about one worker per physical core. More workers than cores add scheduling overhead and slow each worker down.

Use small learning rates

Asynchronous updates are noisier. Start around 1e-4 with RMSprop and adjust.

Share the optimizer statistics

Keep optimizer state in shared memory so all workers contribute to the same running averages.

Keep rollouts short

Five to twenty steps is typical. Short rollouts update more often and rely on the critic, while long ones recover Monte Carlo variance.

Add an entropy bonus

An entropy weight near 0.01 is a common start. Decay it if the policy needs to settle later.

Clip gradients

Gradient norm clipping prevents rare huge updates, which are more damaging when many workers push at once.

Seed each worker differently

Different seeds give different trajectories, which is the whole point of parallel exploration.

Log from a separate evaluator

Run one process that periodically copies the global weights and evaluates without exploration noise. Track the mean return, entropy and value loss.

Consider A2C first

If you do not need asynchrony, start with A2C. It is simpler to debug and reproduce. Move to A3C only for a clear reason, such as a CPU-only cluster.

Tuning guide

Debugging order

1. Run a single worker and confirm that it learns a trivial task.
2. Add workers one at a time and watch whether speed scales.
3. Verify that local weights are synced at the start of every rollout.
4. Check that gradients actually reach the global parameters.
5. Compare against a synchronous A2C run on the same task.
6. Only then tune the learning rate and entropy weight.

Checklist

• Global model in shared memory
• Shared optimizer statistics
• Different seeds per worker
• Entropy bonus and gradient clipping
• Separate evaluator process

Common mistakes

1. Using more workers than cores. Context switching slows everyone.
2. Separate optimizers per worker. The statistics become noisy and inconsistent.
3. Forgetting to sync local weights. Workers drift on old parameters.
4. Letting gradients flow through the advantage. The advantage is a constant in the policy loss.
5. Ignoring the done mask. Bootstrapping across an episode end leaks value.
6. Comparing runs without enough seeds. Asynchronous timing adds extra run-to-run noise.
7. Expecting GPU speedups. The design favors CPUs and small networks.
8. No entropy bonus. The policy becomes deterministic too soon.
9. Same seed in every worker. Workers then explore the same trajectory.
10. Reusing the method where a simpler one fits. For many tasks, A2C or PPO is easier and as good.

Frequently asked questions

What does A3C stand for?

A3C stands for asynchronous advantage actor-critic. It is an actor-critic algorithm where many workers run in parallel and update a shared model asynchronously, without waiting for each other. The advantage is the return minus the critic's value estimate.

What is the difference between A3C and A2C?

A3C updates the shared model asynchronously, with each worker applying gradients on its own schedule. A2C collects rollouts from all workers, combines them into one batch and updates once. A2C is easier to reproduce and works well on GPUs, and it often matches A3C in quality.

Why does A3C not need a replay buffer?

Training data from many workers is already diverse, because each worker explores a different part of the environment at the same time. This diversity breaks the correlation between consecutive samples, which is the job a replay buffer does in methods like DQN.

Is A3C on-policy?

Yes. Each worker uses data from its own local copy of the policy, which is a recent version of the global policy. Gradients are applied soon after, so the data is nearly on-policy, with a small amount of staleness.

What are stale gradients?

A stale gradient is one computed with parameters that have since been changed by other workers. When a worker applies it to the global model, the update is slightly out of date. With modest worker counts and small learning rates, this extra noise is usually tolerable.

Does A3C need a GPU?

No. It was designed to run on multi-core CPUs and works best with small networks and cheap environments. Large networks and image inputs often benefit from GPU-based synchronous methods like A2C or PPO instead.

Is A3C still used?

It is mostly a historical and educational reference now. Synchronous methods such as A2C and PPO are more common because they are easier to reproduce and use GPUs well. Its ideas of parallel actors and shared parameters live on in distributed training systems.

How many workers should I use?

A common rule is about one worker per physical CPU core. More workers than cores add overhead without extra samples. Fewer workers mean slower data collection and more correlation between samples.

Can A3C handle continuous actions?

Yes. The actor can output the mean and spread of a Gaussian distribution and sample from it. The log-probability enters the policy loss in the same way, and the critic and advantage logic stay unchanged.

How does A3C relate to tabular data?

A3C is not a table model. It can train decision policies around a table model, such as choosing which rows to review or which columns to clean, using a tabular encoder to form the state. It suits cases where many cheap table-driven simulations can run on CPU cores in parallel.

Glossary

• Actor-critic: A method with a policy (actor) and a value estimator (critic).
• Advantage: Return minus the critic's estimate.
• Asynchronous update: A parameter update applied without waiting for other workers.
• Bootstrapping: Using an estimated value in place of unseen future rewards.
• Entropy bonus: A term that rewards randomness in the policy.
• Global network: The shared parameters that all workers update.
• Hogwild style update: A lock-free parallel update scheme.
• Local network: A worker's copy of the model.
• N-step return: Real rewards for n steps plus a bootstrapped value.
• On-policy: Learning from data produced by the current policy.
• Rollout: A short run of steps collected by one worker.
• Stale gradient: A gradient computed with out-of-date parameters.
• Worker: A thread or process with its own environment.

Summary

A3C trains an actor and a critic with many parallel workers that update shared parameters asynchronously. Each worker plays a short rollout, computes n-step returns and advantages, and pushes gradients to the global model. This decorrelates data without a replay buffer and makes deep reinforcement learning practical on multi-core CPUs.

In modern practice the synchronous A2C and the clipped PPO are more common, but A3C remains the historical origin of parallel actor-critic training. For tabular work, pair it with an encoder such as FT-Transformer or representations from TabPFN, and use it for sequential decisions like imputation order or anomaly triage. Compare it with REINFORCE, TRPO and reinforcement learning in general, and see /model/research, /model/architecture and /model/benchmarks for model details.

Related

It belongs to Reinforcement learning in my stack, under AI and machine learning. Others in the same group: