Are the actor and critic always separate networks?

No. They may share learned representations while producing different outputs.

Is the advantage simply the reward?

No. It compares an estimated action return with a state-value baseline.

Does a positive advantage guarantee that an action is best?

No. It is an estimate from limited experience relative to a baseline, not proof that every alternative is worse.

AI Glossary / Probabilistic machine learning / Advanced

Advantage actor-critic

Definition

Advantage actor-critic is a reinforcement-learning approach in which an actor learns a policy and a critic helps estimate whether an action performed better or worse than expected in its state. The estimated advantage guides policy updates. A2C commonly refers to the synchronous variant that combines experience from parallel environments before updating shared parameters.

What it means

The actor answers, probabilistically, which action to take. The critic estimates expected future return and provides a reference against which an observed or bootstrapped outcome can be compared. A positive advantage suggests the selected action exceeded that reference; a negative advantage suggests it fell short.
Advantage does not mean an action has an absolute benefit in every situation. It is relative to a state and policy, and the estimate can be inaccurate. A reward of five might represent a disappointing result when ten was expected, while a reward of one could exceed an expectation of zero.

How it works

The policy generates short trajectories. Returns or bootstrapped targets are used to train the value function and estimate advantages. A basic one-step estimate is r + gamma V(s_next) − V(s), with appropriate handling of genuine terminal states. Longer returns and generalized advantage estimation are also possible.
The actor update weights the gradient of the selected action's log probability by its estimated advantage. The critic learns to reduce prediction error, and implementations may include an entropy term to encourage exploration. A2C synchronizes collected experience before the update; A3C instead permits asynchronous workers to update shared parameters. Synchronous updating does not imply that the policy or environment is deterministic.

Three practical examples

A game-development team trains a simulated character to complete a navigation task. The actor chooses movements, while the critic estimates expected remaining reward from each observed state.
A scheduling researcher experiments in a simulator where actions assign jobs to available machines. Evaluation includes waiting times and constraint violations, since a reward score alone may conceal undesirable schedules.
A robotics team studies a simulated reaching task. Multiple environment copies supply experience for synchronous updates, and the eventual policy is evaluated separately under changed initial conditions and disturbances.

Think of it this way

Think of a player and a coach sharing information. The player chooses a move, and the coach compares the outcome with what was reasonably expected from that position. Praise for an unexpectedly good result can help guide future choices. If the coach's expectations are wrong, however, that feedback can mislead the player.

Worked example

Consider an original one-step teaching calculation. Let the current value estimate be V(s) = 4, the observed reward r = 1, the next-state estimate V(s_next) = 5, and the discount factor gamma = 0.9. Assume the transition is not terminal.
The bootstrapped target is 1 + 0.9 × 5 = 5.5. The estimated advantage is 5.5 − 4 = 1.5. Although the immediate reward is only one, the target exceeds the current prediction.
Suppose a simplified two-action actor uses a single logit z for the chosen action, with probability p = sigmoid(z). At p = 0.4, the derivative of log(p) with respect to z is 1 − p = 0.6. Holding the advantage estimate fixed, the policy-gradient contribution is 1.5 × 0.6 = 0.9. A gradient-ascent step of size 0.1 increases z by 0.09. Starting from log(0.4/0.6), the updated probability is approximately 0.4218.
For a genuinely terminal transition with the same reward, the continuation value is zero, so the target would be one and the advantage −3. This reverses the update signal. A time-limit truncation is not automatically the same as a true terminal state. The example omits batching, entropy, shared-network effects, and optimizer details; it illustrates one update rather than a complete training run.

Illustrative case study

Fictional simulator developer QueueCraft tests A2C for allocating repair jobs to workstations. Training reward rises, but a review finds that the reward penalizes average delay while barely reflecting rare, very long waits. The learned policy exploits that imbalance by repeatedly postponing a difficult job type.
The team revises the evaluation criteria and investigates the reward design before conducting fresh training. It compares against a simple scheduling rule on held-out scenarios and reports variability across runs. This fictional example illustrates why an improving learning curve does not by itself establish a useful operational policy; it reports no documented deployment or measured savings.

Common misconceptions and mistakes

The critic is not an external human evaluator and does not necessarily know the true value function. Its errors can influence the actor. Advantage estimates also need not be unbiased in every implementation, especially when they rely on learned bootstrapped values.
Another mistake is interpreting synchronous A2C as deterministic action selection. Synchronization describes coordination of updates. Random exploration, stochastic environments, and numerical details can still affect results. Finally, ordinary A2C is an on-policy method: arbitrarily reusing old experience without accounting for the changed policy is not equivalent to its usual training procedure.

Business use

A2C can serve as a baseline for sequential decision experiments when interaction can be simulated or otherwise conducted within acceptable constraints. Define the observation, action, reward, and evaluation criteria before training. Compare performance with simpler methods and measure sensitivity to environment assumptions. Account for the cost of generating experience, since faster parallel collection does not necessarily mean fewer interactions are required. Deployment decisions need evidence beyond the training reward.

Check your understanding

Why is the example's advantage 1.5?

Answer: The bootstrapped target 5.5 exceeds the current value four by that amount.

Why does the chosen action's probability increase in the simplified update?

Answer: Its fixed positive advantage multiplies a positive log-probability gradient.

Why treat terminal status carefully?

Answer: It determines whether a continuation value belongs in the target.

Related terms

Policy gradient
Advantage function
Generalized advantage estimation
Actor-critic
value function
A3C
temporal-difference learning
on-policy learning

Learn next

Study the distinction between state value, action value, and advantage. Then compare one-step and multi-step targets, and examine how synchronous experience collection changes implementation without removing estimation uncertainty.

Sources

Stable Baselines3, A2C documentation: https://stable-baselines3.readthedocs.io/en/master/modules/a2c.html Mnih and colleagues, Asynchronous Methods for Deep Reinforcement Learning, 2016, foundational actor-critic paper: https://arxiv.org/pdf/1602.01783

Previous: Admissible heuristic Next: Advantage function