SAC

Overview

SAC, short for soft actor-critic, is an off-policy reinforcement learning algorithm for continuous action spaces. It trains a stochastic policy to maximize two things at once: the expected reward and the entropy of the policy. Entropy measures randomness. Rewarding it pushes the agent to keep exploring and to avoid locking onto one narrow behavior too early. The combination gives strong sample efficiency, good stability and little sensitivity to the random seed.

SAC reuses old experience from a replay buffer, so it needs far fewer environment interactions than on-policy methods such as PPO. It uses two critics to reduce overestimation, in the spirit of TD3, and it can tune its own temperature, the weight of the entropy bonus. For work with tabular data, SAC is a natural learner for control problems such as energy use, process settings and pricing, where a model that reads tables supplies the state and the reward signals.

What it is

SAC is built on the maximum entropy reinforcement learning framework. In ordinary reinforcement learning the goal is to maximize the sum of future rewards. In maximum entropy reinforcement learning the goal is to maximize the sum of future rewards plus a bonus for how random the policy is in each state:

objective = sum over time of ( reward + alpha * entropy of policy at that state )

The number alpha is called the temperature. A high temperature favors randomness and exploration. A low temperature favors reward and determinism.

SAC is an actor-critic method with these parts:

• A stochastic actor that outputs a distribution over actions, usually a Gaussian squashed through tanh to stay within bounds.
• Two Q-function critics that estimate the soft value of a state and action.
• Slowly updated target critics that give stable learning targets.
• A replay buffer that stores past transitions for reuse.
• Optionally, an automatically tuned temperature.

An everyday analogy

Think about finding a good restaurant in a new city. A strict reward maximizer tries the first decent place and returns every night. An entropy-regularized agent keeps trying different places for a while, even when it already has a favorite, because variety is valued. It still prefers better places, but it avoids committing early. Over time, as it learns more, it settles on a small set of good choices. Entropy plays the role of curiosity that fades as knowledge grows.

Why a stochastic policy helps

A stochastic policy:

• Explores in a smooth, state-dependent way without extra noise schedules.
• Is robust, because it learns several ways to do well rather than one brittle way.
• Can be made deterministic at test time by using its mean action.

Where it came from

SAC grew from the line of off-policy actor-critic methods for continuous control.

• Q-learning and deep Q-networks showed that replay buffers and target networks make deep value learning workable for discrete actions. See DQN.
• Deterministic policy gradients and DDPG (2015) extended this idea to continuous actions, but DDPG proved brittle and hard to tune.
• TD3 (2018) fixed much of DDPG's overestimation with twin critics, delayed policy updates and target smoothing.
• Maximum entropy reinforcement learning had earlier roots in work on inverse reinforcement learning and in "soft" Q-learning (2017), which learned energy-based policies.
• Soft actor-critic was introduced in 2018. A first paper presented the idea, and a follow-up, "Soft Actor-Critic Algorithms and Applications", added automatic temperature tuning and removed the separate value network of the first version.

SAC became a standard baseline for continuous control and a popular choice for real-world robotics and process control, thanks to its stability and efficiency. For the general field, see reinforcement learning.

How it works

Step 1: act and store experience

The agent samples an action from its stochastic policy, applies it to the environment, and stores the transition (state, action, reward, next state, done flag) in the replay buffer. Early on, many implementations take random actions for a few thousand steps to fill the buffer with varied data.

Step 2: sample a minibatch

At each update, a random minibatch of past transitions is drawn from the buffer. Random sampling breaks the correlation between consecutive steps, which stabilizes learning.

Step 3: build the soft target for the critics

For each transition, the next action is sampled from the current policy at the next state. The target is

y = reward + gamma * (1 - done) * ( min(Q1_target, Q2_target)(next state, next action) - alpha * log pi(next action | next state) )

Three features stand out:

• The minimum of two target critics reduces overestimation. Q-value estimates tend to be biased upward when a maximum or a noisy estimate is used, and the minimum counters this.
• The term minus alpha times log probability is the entropy bonus. Since log probability is negative for likely actions, subtracting it adds value to the target when the policy is spread out.
• The target critics are slow-moving copies of the critics, which stops the target from chasing itself.

Step 4: update the critics

Each critic is trained by minimizing the squared difference between its prediction and the target y. The two critics are initialized differently and train on the same targets.

Step 5: update the actor

The actor is trained to maximize

min(Q1, Q2)(state, action) - alpha * log pi(action | state)

where the action is sampled from the policy. The key technique is the reparameterization trick: the action is written as a deterministic function of the state and a noise sample. This makes the sampled action differentiable with respect to the policy parameters, so the gradient can flow from the critic back into the actor with low variance.

Step 6: update the temperature

In the improved version, alpha is learned. The aim is to keep the policy entropy near a target value, usually set to minus the number of action dimensions. If the policy's entropy is below target, alpha rises and the policy is nudged to explore more. If entropy is above target, alpha falls. This removes a sensitive hyperparameter.

Step 7: update the target networks

The target critics are updated by Polyak averaging: after each step, each target parameter moves a small fraction (often 0.005) toward the matching critic parameter.

Step 8: repeat

The loop of acting, storing and updating continues. Many implementations do one gradient update per environment step.

Handling bounded actions

Real actions usually have limits, such as a valve opening between zero and one. SAC samples from a Gaussian and passes the sample through tanh to squash it into the range from minus one to one, then rescales it to the real limits. The log probability must be corrected for this change of variables. Forgetting the correction is a classic bug.

The discrete-action variant

The original algorithm is for continuous actions. A discrete variant exists, in which the policy outputs probabilities over a finite set of actions and expectations are computed exactly over those actions instead of by sampling. It is used less often than the continuous version.

Key concepts and terms

A worked example

A numeric walk-through of one soft target

Suppose gamma = 0.99, alpha = 0.2, the reward is 1.0 and the episode is not done. At the next state, a sampled next action has these values:

• Target critic 1 says 10.0.
• Target critic 2 says 9.0.
• The log probability of the sampled next action is -1.5.

Then:

• The minimum target value is 9.0.
• The entropy term is minus alpha times log probability, which is -0.2 * (-1.5) = +0.3. So the soft value is 9.0 + 0.3 = 9.3.
• The target is y = 1.0 + 0.99 * 9.3 = 10.207.

Both critics are trained toward 10.207. If the policy were more deterministic, the log probability would be less negative (for example -0.2), the bonus would be only +0.04, and the target would be slightly lower. A more random policy earns a larger bonus, which is the maximum entropy idea at work.

Temperature tuning in numbers

Say the action has two dimensions, so the target entropy is -2. If the current policy has an estimated entropy of -3 (more deterministic than desired), the loss for alpha pushes alpha up, so the entropy bonus counts for more and the policy is nudged toward more randomness. If the estimated entropy is -1, alpha is pushed down.

A compact PyTorch sketch

This is a teaching sketch of one update step. It omits the environment loop and buffer.

import torch
import torch.nn as nn
import torch.nn.functional as F

class Actor(nn.Module):
    def __init__(self, obs_dim, act_dim, hidden=256):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(obs_dim, hidden), nn.ReLU(),
                                 nn.Linear(hidden, hidden), nn.ReLU())
        self.mu = nn.Linear(hidden, act_dim)
        self.log_std = nn.Linear(hidden, act_dim)

    def sample(self, obs):
        h = self.net(obs)
        mu, log_std = self.mu(h), self.log_std(h).clamp(-20, 2)
        std = log_std.exp()
        u = mu + std * torch.randn_like(mu)          # reparameterization
        a = torch.tanh(u)
        logp = (-0.5 * ((u - mu) / std) ** 2 - log_std - 0.5 * torch.log(torch.tensor(2 * torch.pi))).sum(-1)
        logp -= torch.log(1 - a.pow(2) + 1e-6).sum(-1)  # tanh correction
        return a, logp

def make_q(obs_dim, act_dim, hidden=256):
    return nn.Sequential(nn.Linear(obs_dim + act_dim, hidden), nn.ReLU(),
                         nn.Linear(hidden, hidden), nn.ReLU(),
                         nn.Linear(hidden, 1))

def sac_update(batch, actor, q1, q2, q1_t, q2_t, log_alpha, opts,
               gamma=0.99, tau=0.005, target_entropy=-2.0):
    s, a, r, s2, d = batch
    alpha = log_alpha.exp().detach()

    with torch.no_grad():
        a2, logp2 = actor.sample(s2)
        sa2 = torch.cat([s2, a2], -1)
        q_min = torch.min(q1_t(sa2), q2_t(sa2)).squeeze(-1)
        y = r + gamma * (1 - d) * (q_min - alpha * logp2)

    sa = torch.cat([s, a], -1)
    q_loss = F.mse_loss(q1(sa).squeeze(-1), y) + F.mse_loss(q2(sa).squeeze(-1), y)
    opts["q"].zero_grad(); q_loss.backward(); opts["q"].step()

    a_new, logp = actor.sample(s)
    sa_new = torch.cat([s, a_new], -1)
    q_pi = torch.min(q1(sa_new), q2(sa_new)).squeeze(-1)
    actor_loss = (alpha * logp - q_pi).mean()
    opts["actor"].zero_grad(); actor_loss.backward(); opts["actor"].step()

    alpha_loss = -(log_alpha * (logp.detach() + target_entropy)).mean()
    opts["alpha"].zero_grad(); alpha_loss.backward(); opts["alpha"].step()

    with torch.no_grad():
        for p, pt in zip(list(q1.parameters()) + list(q2.parameters()),
                         list(q1_t.parameters()) + list(q2_t.parameters())):
            pt.mul_(1 - tau).add_(tau * p)

A table-driven scenario

Consider a building with a cooling system. The state contains temperatures, occupancy, weather forecasts and electricity price. A tabular model reads the history tables and outputs forecasts of load. The action is a set of continuous setpoints. The reward combines comfort and energy cost. SAC can learn setpoints from logged and simulated data, reusing every transition many times, which matters because real interaction is slow and costly. The entropy bonus helps the agent try varied setpoints at first, then settle.

Strengths and limitations

Strengths

• Sample efficient. The replay buffer lets it reuse each transition many times.
• Stable. Twin critics, target networks and entropy regularization all damp divergence.
• Robust to seeds and hyperparameters. Results vary less between runs than with many other methods.
• Built-in exploration. The stochastic policy and entropy bonus explore without hand-made noise.
• Automatic temperature. The tuned alpha removes one sensitive choice.
• Works well for continuous control. It is a strong default for robotics, energy and process tasks.

Limitations

• Mostly for continuous actions. The standard algorithm is not designed for discrete choices.
• More components. An actor, two critics, two targets and a temperature mean more code and more memory than PPO.
• Reward scale matters. The balance between reward and entropy depends on the scale of rewards, so very large or tiny rewards need rescaling.
• Off-policy risks. Reusing old data can cause value estimation errors, and the method can struggle in sparse-reward tasks without extra tricks.
• Heavy compute per step. Many gradient updates per environment step make it slower in wall-clock time when the simulator is very fast and cheap. In that case a parallel on-policy method may finish sooner.
• Entropy target is a heuristic. The default of minus the action dimension works well on many tasks but is not universal.

How it compares with alternatives

SAC and TD3 are the usual two choices for continuous control with limited data. SAC tends to be more robust across tasks, while TD3 is simpler and deterministic. If you have a fast simulator and many parallel workers, PPO may be easier to scale.

Role in tabular foundation models

SAC is a decision algorithm, not a table reader. Tabular foundation models are pre-trained on large numbers of tables, often synthetic, so that they can handle new tables in context. They deal with row and column structure, mixed numeric and categorical types and missing values, and they do not need column names. See TabPFN, /model/architecture and /model/research. SAC and such models complement each other in several concrete ways.

Tabular predictions as state

In a control problem, the raw data is often a set of tables: sensor logs, order histories, price lists. A tabular foundation model can turn each new table or row into forecasts and scores without retraining, because it learns in context from labeled rows. These outputs form a compact, informative state for SAC. This lets the actor and critics stay small, which speeds up training and reduces overfitting. See predictive maintenance.

A learned environment from logged tables

Real systems rarely allow unlimited trial and error. When only logs exist, a model fitted to them can predict the next state and reward for a given state and action, serving as a simulator. SAC is a good learner for this setting because it is sample efficient and handles continuous actions. The risk is model error: the agent may exploit regions where the learned simulator is wrong. Use short rollouts, mix real and simulated data, and check the policy on held-out real logs. This links to digital twins.

Offline settings

When no new interaction is possible, SAC can be adapted to learn from a fixed table of logged transitions. Plain off-policy methods can fail there, because the critic may overestimate the value of actions that never appear in the logs. Offline variants add conservative penalties or constrain the policy to stay near the logged behavior. A tabular model can also supply uncertainty estimates to flag unfamiliar actions.

Missing values and imputation

State vectors built from real tables contain gaps. A tabular foundation model can impute a missing cell by predicting it from the rest of the row in context, and can pass along an uncertainty estimate. The policy can then be fed both the imputed value and a missingness flag. See imputation networks.

Anomaly detection as safety signal

An anomaly detection model trained on normal operation can score whether the current state looks unusual. This score can be used as an input to SAC, as a penalty in the reward, or as a trigger for a rule-based fallback. The entropy bonus encourages exploration, so guarding against unsafe exploration matters in real systems.

Synthetic data and pre-training

Tabular foundation models gain much of their power from pre-training on synthetic tables drawn from a broad prior over data-generating processes. Choosing the generator settings is a design problem that could be treated as a continuous control task. Teams often solve it with Bayesian optimization or hand tuning instead. See synthetic data generation. SAC is a possible alternative when many sequential adjustments are needed.

Continual learning

A deployed policy and its tabular model both face drift. A SAC agent can decide when to refresh the model or which data to replay, with a reward that trades accuracy against cost. The replay buffer can also be managed to hold a balanced mix of old and new situations, which fights forgetting.

Sector and semantic classification

A tabular model can infer what a column means, or what sector a table belongs to, from values alone. Those inferred labels can select which reward function, which action limits or which sub-policy applies, so a general SAC agent can serve many tables.

An honest boundary

If the job is to predict a target column, a tabular foundation model or a supervised learner is the right tool. SAC is only appropriate when decisions change future states and rewards unfold over time.

Practical guidance and best practices

Typical hyperparameters

These are starting points, not rules.

Scale rewards and normalize observations

The reward scale sets the balance between reward and entropy. If rewards are very large, entropy has almost no effect. If they are tiny, the policy stays near random. Rescale rewards to a moderate range, and normalize observations.

Bound actions properly

Normalize the action space to the range from minus one to one, let the actor output within that range, and rescale at the environment boundary. Always include the tanh log-probability correction.

Use the right done handling

Distinguish true terminal states from time-limit truncation. For truncation, do not zero out the bootstrap term, otherwise the critic learns that the end of the time limit is bad.

Monitor the right signals

• Episode return and length.
• Critic losses and Q-value magnitude. Rapidly growing Q-values suggest instability.
• Policy entropy compared with the target.
• The temperature value over time.
• The share of the replay buffer that is recent data.

Use evaluation episodes

Evaluate the deterministic policy (the mean action) separately from training episodes, in a separate environment copy, and average over several episodes and seeds.

Example configuration

algo: sac
env: example-hvac-control
seed: 11
gamma: 0.99
tau: 0.005
lr_actor: 0.0003
lr_critic: 0.0003
lr_alpha: 0.0003
batch_size: 256
buffer_size: 1000000
start_steps: 5000
updates_per_step: 1
auto_alpha: true
target_entropy: -4
hidden_sizes: [256, 256]
normalize_obs: true
eval_every: 5000
log_dir: /var/log/example.internal/sac-runs

Improve sample efficiency further

• Raise the update-to-data ratio, using several gradient steps per environment step, with layer normalization or regularization in the critics to prevent overfitting.
• Use larger or prioritized replay if rare events matter.
• Add n-step returns for faster credit assignment, with care about off-policy bias.

Governance for deployed policies

Keep action limits enforced outside the policy, add a rule-based fallback, test offline first, roll out gradually and log every decision with the state, action, policy version and any anomaly score.

Variants and extensions

• Discrete SAC adapts the algorithm to finite action sets.
• Offline SAC variants add conservative penalties so the critic does not overvalue actions absent from the data.
• Distributional critics learn a distribution of returns rather than a single value, which can help with risk-sensitive control.
• Layer-normalized critics with high update ratios allow many gradient steps per environment step without overfitting.
• Hierarchical and goal-conditioned SAC learn policies conditioned on a goal, which suits tasks with many targets.
• Multi-agent SAC uses shared or centralized critics. See multi-agent RL.
• Safe SAC adds cost critics and constraints so that expected cost stays below a threshold.

A debugging checklist

1. Run a trivial environment where the best action is obvious. SAC should solve it quickly.
2. Check that actions are scaled properly into and out of the tanh range.
3. Check that the log probability includes the tanh correction.
4. Check that target critics are not receiving gradients.
5. Check that done flags and truncation flags are separate.
6. Plot Q-values. Steady, slow growth is normal. Explosive growth means instability.
7. Plot entropy against the target. If it never approaches the target, check the temperature learning rate.
8. Inspect a sample of actions. If they all sit at the bounds, the policy has saturated, and the pre-tanh outputs may be too large.
9. Compare the training return with the evaluation return. A big gap points to a problem with stochastic sampling or overfitting to noise.
10. Re-run with a different seed. If one seed fails and another succeeds, look at the early random-action phase and the reward scale.

How to read a SAC log

Early in training the temperature is relatively high and entropy is high, so actions look random and returns are low. The critics start to fit the returns, and the actor follows the critic's gradient. As the policy improves, the tuned temperature usually falls, entropy declines toward the target, and returns rise. A healthy run shows slowly growing Q-values that match observed returns reasonably well, and a critic loss that is noisy but not exploding.

If the temperature drops toward zero very early, the target entropy may be too low for the task. If it keeps rising, the policy is struggling to reach the target entropy, often because the actions are saturated by the tanh squashing.

Using SAC with tabular state

When the state comes from tables, a few extra points help.

• Encode categorical fields with embeddings or one-hot vectors before they reach the networks.
• Add missingness indicators next to imputed values.
• Keep a small, informative state, built from forecasts and summaries, instead of raw rows.
• Normalize each numeric field with running statistics that are saved with the policy, so that the deployed policy sees the same scaling.
• Log the raw state for audit.

Checks before deployment

• Replay the policy on held-out logged states and confirm that its actions stay inside safe limits.
• Compare its decisions with the existing rule-based controller and review the largest differences by hand.
• Stress test with missing inputs, delayed data and sensor spikes.
• Confirm that the saved normalization statistics load with the policy.
• Define a clear rollback path.

When to prefer SAC

Common mistakes

• Omitting the tanh correction. The entropy estimate becomes wrong and learning degrades.
• Wrong target entropy. A target that is too high keeps the policy noisy. One that is too low makes it greedy too soon.
• Ignoring reward scale. The reward-entropy balance depends on it.
• Treating truncation as termination. This biases the critics.
• Too few initial random steps. Early critics then overfit to a narrow data set.
• Evaluating with the sampling policy. Report the deterministic mean action, and say which one you report.
• Copying the target network incorrectly. Target critics must be separate copies that are updated only by Polyak averaging.
• A tiny replay buffer. The agent forgets old situations and can become unstable.
• Using SAC on discrete actions without the discrete variant. The continuous version does not apply.
• Trusting a learned simulator blindly. Validate on real logged data before deployment.
• Skipping seeds. Report mean and spread over several seeds.

Frequently asked questions

What does SAC stand for?

SAC stands for soft actor-critic. Soft refers to the entropy-augmented objective, in which both the value estimates and the policy objective include a bonus for randomness. Actor-critic refers to the pairing of a policy network, the actor, with value networks, the critics, that guide its learning.

Is SAC on-policy or off-policy?

SAC is off-policy. It stores transitions in a replay buffer and learns from them many times, even after the policy has changed. This is the main reason it needs fewer environment steps than on-policy methods such as PPO or TRPO.

What is the temperature in SAC?

The temperature, usually written alpha, sets how much the agent values randomness compared with reward. A high temperature makes the policy explore more. A low one makes it follow the reward more closely. Modern implementations tune it automatically so the policy entropy stays near a chosen target.

Why does SAC use two critics?

Value estimates learned from noisy data tend to be too high, because the learning process favors errors that look good. Taking the minimum of two independently trained critics gives a more cautious estimate and reduces this overestimation. The same idea appears in TD3.

What is the reparameterization trick?

It is a way to write a random sample as a deterministic function of the policy parameters and an independent noise value. Because the function is differentiable, the gradient of the critic's value can flow back through the sampled action into the actor. This gives lower-variance gradients than methods that rely only on score-function estimates.

How does SAC differ from TD3?

TD3 learns a deterministic policy and explores by adding noise, while SAC learns a stochastic policy and explores through an entropy bonus. Both use twin critics and target networks. TD3 adds delayed policy updates and target smoothing, whereas SAC adds automatic temperature tuning. SAC is often more robust across tasks.

Can SAC handle discrete actions?

The standard algorithm is for continuous actions. A discrete variant exists, which computes expectations exactly over the finite set of actions. For many discrete problems, a value-based method such as DQN or a policy-gradient method such as PPO is a more common choice.

Is SAC good for real-world systems?

It is a popular choice because it is sample efficient and stable, which matters when real interaction is slow or costly. Real deployments still need safety measures, such as action limits, a rule-based fallback, offline testing and monitoring, since exploration can produce unusual actions.

How does SAC relate to tabular foundation models?

SAC learns control decisions, while a tabular foundation model reads tables and predicts or imputes in context. The tabular model can supply state features, uncertainty estimates, anomaly scores or a learned simulator, and SAC can learn the policy that uses them. They are complementary rather than competing.

What is the biggest tuning concern with SAC?

Reward scale is the most common issue, because it sets the balance between reward and entropy. After that come the target entropy, the replay buffer size and the number of updates per environment step. Normalizing rewards and observations and keeping the defaults for the rest is a solid way to start.

Glossary

• Actor: the policy network.
• Alpha: the temperature that weights the entropy bonus.
• Critic: a network estimating the value of a state and action.
• Entropy: a measure of randomness in the policy.
• Maximum entropy reinforcement learning: maximizing reward plus entropy.
• Off-policy: learning from data collected by earlier policies.
• Polyak averaging: smooth updating of a target network.
• Replay buffer: a store of past transitions.
• Reparameterization trick: a differentiable way to sample actions.
• Soft Q-function: a Q-function including entropy bonuses.
• Target entropy: the entropy the temperature tuning aims for.
• Target network: a slowly changing copy used for stable targets.
• Twin critics: two Q-functions whose minimum is used.

Summary

SAC is an off-policy, entropy-regularized actor-critic algorithm for continuous control. It trains a stochastic actor and two critics from a replay buffer, adds an entropy bonus to encourage exploration, and tunes its temperature automatically. This gives high sample efficiency, stable learning and low sensitivity to seeds.

Compared with PPO and TRPO, it uses far fewer interactions. Compared with TD3 and DDPG, it is usually more robust, at the price of a stochastic policy and a few more components. It is mainly for continuous actions.

Around tabular foundation models, SAC acts as the decision learner. The tabular model supplies state, forecasts, imputations, anomaly scores and learned simulators, and SAC learns how to act on them. See /model/research, /model/architecture and /model/benchmarks for the model side.

Related

It belongs to Reinforcement learning in my stack, under AI and machine learning. Others in the same group: