Does the critic choose the executed action?

The actor normally defines that choice; the critic guides training or evaluation, although specific algorithms may use its estimates in additional ways.

Must actor and critic have separate neural networks?

No. Some designs share representations, while others separate them.

Are these methods always on-policy?

No. Actor-critic families include on-policy and off-policy methods, with different requirements for using collected experience correctly.

AI Glossary / Reinforcement learning and robotics / Intermediate

Actor-Critic Methods

Definition

Actor-critic methods are reinforcement-learning methods that learn both a policy, called the actor, and a value estimate, called the critic. The actor determines how to choose actions. The critic evaluates expected returns and supplies a learning signal that helps improve the actor's decisions.

What it means

Learning what to do and learning how promising a situation looks are related but different tasks. The actor represents the decision rule, while the critic estimates consequences under a policy. These estimates help the actor learn from experience without relying only on complete, finished episodes.

The two roles need not be two separate computers or even entirely separate neural networks. They may share features while using different output heads and losses. Actor-critic names a family of approaches rather than one universal update equation.

How it works

The actor interacts with an environment, producing experience that includes observations, actions and rewards. The critic learns a state-value function or an action-value function, depending on the method. Many actor-critic designs bootstrap, meaning they use an estimated future value when constructing a training target.

An actor update uses the critic to favor decisions with better estimated outcomes. In a simple stochastic version, a temporal-difference error can act as an advantage signal for a policy-gradient update. Other versions use different critics, multi-step targets or deterministic actors. Because the critic is learned rather than infallible, its errors can distort policy improvement.

Three practical examples

1. A simulated robot's actor outputs continuous control commands. A critic estimates expected return so that training can adjust the command policy.

2. A game-playing actor assigns probabilities to available moves. A state-value critic helps identify whether an observed outcome was better or worse than expected.

3. A research scheduling system learns an allocation policy and a value model together. Reviewers evaluate finished schedules independently instead of treating the critic's estimates as verified savings.

Think of it this way

Think of a player practicing with a coach who estimates the promise of each position. The player chooses moves, while the coach's assessments guide practice. Both improve through experience. An inaccurate coach can encourage poor habits, so the quality of the assessment matters as much as the willingness to act. Unlike a human coach, the critic is a numerical model.

Worked example

Consider a one-step toy update at a nonterminal state. The observed reward is 2, the discount factor is 0.9, the current state's value estimate is 5, and the next state's value estimate is 4. The temporal-difference error is:

Delta = 2 + 0.9 × 4 - 5 = 0.6.

For a tabular critic with learning rate 0.2, the state's value changes to 5 + 0.2 × 0.6 = 5.12. The next-state estimate is treated as fixed for this update.

Now suppose a two-action actor uses a single logistic preference z for action A, initially z = 0, giving probability 0.5. A was selected. The gradient of log probability for A is 1 - 0.5 = 0.5. With actor learning rate 0.1, a simplified update gives z = 0 + 0.1 × 0.6 × 0.5 = 0.03, so A's probability becomes approximately 0.5075.

This local example omits additional weighting and regularization. It illustrates an update, not a guarantee that A is globally optimal.

Illustrative case study

Fictional case: Willow Controls studies a simulated conveyor-routing problem. Its actor chooses routing settings, and its critic estimates the expected discounted reward from current operating conditions. Training produces a rising value estimate, but independent evaluation reveals occasional congestion.

The team investigates reward design, critic errors and evaluation conditions rather than assuming the actor has improved simply because its internal scores rose. It compares completed throughput, delays and failures across separate runs. This invented case shows why both learning components and external outcomes need review; it reports no actual deployment or measured financial benefit.

Common misconceptions and mistakes

The critic is not an independent auditor merely because it evaluates actions. It learns from data and can be wrong, particularly where experience is scarce. An actor-critic model also does not necessarily use human feedback; environment rewards can provide its training signal.

Another mistake is assuming every method with a learned baseline has identical actor-critic properties. Terminology varies, and bootstrapping is an important distinction in standard treatments. Finally, an improving critic loss does not automatically imply an improving policy. Changes in data, reward scaling or value targets can alter the loss without improving decisions.

Business use

Actor-critic methods are relevant to sequential decision problems where policies must be learned and evaluated together, especially when enumerating every possible continuous action is impractical. Their usefulness depends on the environment, data and reward specification, not simply on having two named components.

Plan for independent policy evaluation, reproducible training runs and analysis of value-estimation errors. Domain requirements should be reflected in the task and assessment, while constraints need explicit implementation. A critic trained to predict the chosen reward does not automatically recognize business objectives that the reward omitted.

Check your understanding

Why does the critic increase the state's estimate in the worked example even though the immediate reward is only 2 and the old value is 5?

Answer: The target includes estimated future reward: 2 + 0.9 × 4 = 5.6. That exceeds the old estimate by 0.6. Comparing only the immediate reward with the value would ignore the return the critic is trying to estimate.

Related terms

Policy gradient
value function
temporal-difference learning
advantage
bootstrapping
state value
action value
on-policy learning
off-policy learning

Learn next

Study returns and temporal-difference updates before actor-critic methods. Trace a small update for both actor and critic, then examine how imperfect value estimates influence decisions. Compare methods using the same evaluation setting and report actual returns separately from training losses.

Temporal-difference

Sources

Kochenderfer, Wheeler and Wray, Algorithms for Decision Making, Chapter 13, Actor-Critic Methods: https://algorithmsbook.com/files/chapter-13.pdf

Sutton and Barto, Reinforcement Learning: An Introduction, second-edition draft, Chapter 13: https://www.incompleteideas.net/book/bookdraft2018mar21.pdf

The update calculations and Willow Controls case are original hypothetical illustrations.

Previous: Actor Next: Acyclic directed mixed graph