Must SAC use a separate value network?

The commonly used later formulation does not.

Is the policy deterministic?

Training uses a stochastic actor; evaluation conventions can select a deterministic representative action.

Does automatic entropy adjustment choose the whole learning setup?

No. It adapts temperature relative to a specified entropy target while other choices remain.

AI Glossary / Reinforcement learning and robotics / Advanced

Soft actor-critic

Definition

Soft actor-critic, or SAC, is an off-policy actor-critic reinforcement-learning algorithm that optimizes expected reward together with a policy-entropy term. A stochastic actor selects actions, while learned critics estimate their long-term value. Experience replay allows earlier transitions to be reused. This entry describes the widely used twin-critic formulation with optional automatic temperature adjustment, rather than the earlier version requiring a separate value network.

What it means

The actor learns a distribution over actions instead of a single deterministic choice. The critics judge those choices using an entropy-regularized return. A temperature parameter, commonly called alpha, sets the relative weight of the entropy term.

Entropy encourages the policy to retain multiple plausible actions when their values are similar. It is part of the learning objective, not simply independent noise added to a deterministic controller after training.

Off-policy learning means stored experience can support updates even when it was collected by an older policy. It does not mean an arbitrary fixed dataset is sufficient for reliable learning. Coverage, critic errors and the mismatch between recorded and proposed actions still affect performance.

How it works

Collect transitions in a replay buffer. For a sampled nonterminal transition, draw a next action from the current actor. Evaluate it using two slowly updated target critics, take the smaller value, and subtract alpha times its log policy density. Multiply that continuation by gamma and add the observed reward to form the critic target.

Fit both critics to this detached target. Update the actor using sampled actions to minimize alpha times log policy density minus the smaller current critic value. With a reparameterized stochastic actor, gradients can flow through sampled actions into policy parameters.

Update target critics by a moving average. If automatic temperature tuning is enabled, adjust alpha using an entropy target. The entropy target and other training settings still require appropriate choices; automatic tuning does not remove all configuration decisions.

Continuous actions are often generated through a squashed Gaussian. Its log density must include the transformation correction. Genuine termination removes the continuation term, whereas an artificial time limit may need different treatment. Exact soft-policy-iteration results do not establish unconditional convergence for every practical neural-network implementation.

Three practical examples

1. A continuous-control simulator trains a stochastic actuator policy while reusing earlier transitions for critic learning.

2. A researcher compares fixed and learned temperatures, measuring both task return and the policy's entropy during training.

3. An implementation review checks the minimum over target critics, terminal handling and the log-density correction after action squashing.

Think of it this way

Think of a decision-maker who values both strong outcomes and keeping several good options available. The critic estimates the benefit of each option, while the actor avoids collapsing prematurely onto one choice. The temperature determines how much that diversity matters relative to the task's rewards. Diversity alone does not make a poor action desirable enough to overcome every reward difference.

Worked example

Consider an original single-sample critic-target calculation. The observed reward is 1, gamma is 0.9 and temperature alpha is 0.2. The transition is nonterminal. A next action sampled from the current actor has log density -0.7, and the two target critics estimate 4 and 5 for that action.

The smaller critic value is 4. The entropy-adjusted continuation is 4 minus 0.2 times -0.7, or 4.14. The target is therefore 1 plus 0.9 times 4.14, or 4.726.

If the first current critic predicts 3.5 for the recorded state-action pair, its target residual is 1.226. Its half-squared error for this sample is 0.751538. Using the average of the two target values instead would produce 5.176, which is a different target and not the stated twin-minimum calculation.

For an independent actor-loss sample at a replayed state, suppose the sampled action's log density is -0.6 and the smaller current critic value is 3.8. The sampled actor objective is 0.2 times -0.6 minus 3.8, or -3.92. This is a loss contribution to minimize, not a directly observed environmental reward.

If the original transition had genuinely terminated, its target would be 1, with no next-state value or entropy continuation. These examples check arithmetic only; they do not specify network gradients, an entire batch or a successful learning run.

Illustrative case study

In a fictional project, Harbor Robotics evaluates SAC for simulated gripper control. The team compares it with a deterministic actor-critic baseline using equal environment-interaction budgets and several random seeds.

During review, a critic-target check finds that the implementation was averaging target critics. The researchers correct it to the intended minimum and rerun the affected experiment. They also separate stochastic training behavior from the action-selection convention used during evaluation.

Reported simulator return, failed grasps and action-limit violations are kept distinct. Entropy regularization does not itself enforce a physical safety constraint. This is an illustrative development scenario, not a report of measured gains or a real-world deployment.

Common misconceptions and mistakes

Do not call alpha the learning rate without checking the notation; in SAC it commonly denotes temperature. Do not confuse action density with a discrete action probability: continuous densities can exceed one, and differential entropy can be negative.

Do not assume the smaller of two critics eliminates all estimation bias. Also avoid treating off-policy capability as a general offline-learning guarantee. Actor optimization can exploit inaccurate values for poorly represented actions in a fixed dataset.

Business use

SAC is a useful candidate for continuous sequential control when interactions are expensive enough that replay reuse matters. Evaluate data requirements, reward design, computational cost, variability across runs and operational constraints. Record the algorithm version and temperature settings so comparisons are meaningful. A successful benchmark result does not establish suitability for a particular machine or business process.

Check your understanding

What target does the nonterminal example produce?

Answer: 4.726.

Why is 5 not used as the next-action critic value?

Answer: The stated formulation takes the smaller of the two target estimates.

What is the target after genuine termination?

Answer: The immediate reward, 1.

Related terms

Maximum entropy reinforcement learning
Experience replay
Target network
Reparameterization trick
Twin delayed deep deterministic policy gradient
Actor-critic
Entropy regularization

Learn next

Study actor-critic learning and soft value functions first. Then examine replay-based critic targets, stochastic actor gradients and temperature optimization. Check action transformations and termination conventions before comparing implementations.

Sources

Haarnoja and colleagues, Soft Actor-Critic Algorithms and Applications, 2018, revised 2019, sections 4–6 and appendices: https://arxiv.org/pdf/1812.05905. This primary paper explains the later twin-critic formulation, removal of the separate value network and temperature adjustment. The sample calculations and fictional project above are original teaching examples.

Previous: Socio-technical system Next: Soft clustering