The policy part that chooses actions, trained using the critic's judgments.

Is the critic a separate model?

Usually a separate learned component, trained alongside the actor from observed outcomes.

Where does the reward come from?

From the environment or a defined business metric, such as throughput, retention or margin.

Is the critic used after training?

Often it is dropped at deployment, since the trained actor can act alone, though some systems keep it for monitoring.

AI Glossary / AI and machine learning / Intermediate

Critic

Definition

In reinforcement learning, the critic is the part of the system that judges how good the current situation or action is, while another part, the actor, decides what to do.

What it means

Actor-critic systems split the work of learning from rewards. The actor proposes actions; the critic scores them, estimating how much future reward to expect from here.

The critic's judgment is what the actor learns from. When the critic says a move went better than expected, the actor is nudged to repeat it; worse than expected, and it is nudged away.

This split helps because judging is easier than choosing. Learning to value a situation is a prediction problem, while choosing among all possible actions is a much larger search.

The critic is trained from actual outcomes: its estimates are compared against the rewards that really arrived, and the gap drives improvement in both parts.

For a business, you will meet the critic-actor idea inside robotics, recommendation and optimization systems, and it offers a useful mental split: the part that acts and the part that grades, each needing its own data and care.

The critic also reduces noise. Raw rewards are lumpy, a win or a loss at the end, while the critic's estimates give a steady signal at every step, so learning is faster and less erratic.

How it works

The actor takes the current state and picks an action, according to its current policy.

The environment responds with a reward and a new state.

The critic compares what it predicted for that action against what actually followed, including the value of the new state, and computes the error in its own judgment.

That error trains both: the critic updates its estimates to be more accurate, and the actor updates its policy toward actions the critic underrated.

Over many rounds, the critic becomes a fair judge and the actor a good chooser, each improving the other's training signal.

A common variant has the critic estimate how much better an action was than average. This advantage signal tells the actor not just that something worked, but how much more than usual.

Three practical examples

Robotics: a critic scoring joint positions so the actor can learn smooth, efficient movements.
Recommendation: a critic estimating long-term engagement from a suggested video, beyond the immediate click.
Game-playing systems, where the critic judges board positions and the actor picks moves.

Think of it this way

Think of a junior chef cooking for a restaurant critic. The chef tries dishes; the critic scores them against how the room actually responds. Over time the critic's scores sharpen, and the chef's menu improves with them.

The limit: if the critic only ever sees safe dishes, their judgment of bold ones is worthless. The actor must explore, or the critic never learns the full menu.

Worked example

the approval that learned to say no

Suppose a warehouse system learns to route parcels, with an actor choosing conveyor lanes and a critic scoring each choice against actual throughput.

Early on, the actor floods the fastest lane. The critic, comparing predictions against real congestion outcomes, learns the fast lane's value collapses when crowded. The example is illustrative.

The actor's policy shifts toward balanced routing, and average parcel time drops.

The lesson: the critic turned raw outcomes into graded experience, which is far denser training signal than win-or-lose at the end of the day.

Illustrative case study

A fictional delivery platform trains a dispatch system with a critic that only sees the first hour after dispatch.

It learns to love decisions that look great for an hour and terrible by evening, and customer complaints rise before anyone traces the cause.

The invented lesson: a critic grades what it is shown. Show it short-term outcomes and it will happily teach short-term greed.

Common misconceptions and mistakes

The critic makes the decisions. It scores; the actor chooses, and the two are trained together but play different roles.
The critic knows the true value. It learns estimates from observed outcomes, and it can be biased by what it is shown.
This only matters for games. The actor-critic split sits inside industrial optimization, robotics and recommendation systems.

Business use

When vendors describe reinforcement learning systems, ask what the reward and the critic's time horizon are; short horizons teach short-term behavior.

If you run such a system, log both the actions and the critic's scores, because disagreements between them are early warning signs.

If you commission such a system, agree the reward in plain words first. The critic will faithfully learn whatever the reward measures, including loopholes.

Check your understanding

What does the critic contribute that the actor cannot learn alone?

Answer: A graded judgment of each action's value from observed outcomes, giving the actor a dense learning signal instead of rare final results.

Related terms

Learn next

AI for Business Owners explains the moving parts inside adaptive AI systems in plain terms: https://aiforbusinessandlife.com/academy/ai-for-business-owners. [C2]

Sources

[S1] Russell and Norvig, AIMA glossary; [S2] Google, Machine Learning Glossary. Full URLs are in the source ledger. Verified 11 October 2026.

Previous: Criterion variable Next: Critical path