Can a positive-reward action have negative advantage?

Yes, if the policy's average expected return is higher.

Must advantages average to zero equally across actions?

No. The identity uses the policy's action probabilities.

Is an advantage estimate always accurate?

No. Sampling and value-estimation errors can affect it.

AI Glossary / Reinforcement learning and robotics / Advanced

Advantage function

Definition

The advantage function measures how much better or worse an action's expected return is than the expected return from following a particular policy at the same state. It is defined as A_pi(s, a) = Q_pi(s, a) - V_pi(s). Positive advantage means better than that policy's average choice, under the same reward and discounting conventions.

What it means

An action can have a high expected return but still have negative advantage if the policy's alternatives are even better. Conversely, an action with a negative expected return can have positive advantage when the other choices are worse. Advantage is a relative comparison, not an absolute score of desirability.
The policy reference matters. Q_pi evaluates taking the specified action now and following policy pi afterward. V_pi evaluates following pi from the current state, including its initial action choice. Changing the policy can change both values and therefore change the advantage. A number reported without its state, policy and reward definition is difficult to interpret.

How it works

For discrete actions, the state value equals the policy-weighted average of action values: V_pi(s) = sum_a pi(a|s) Q_pi(s, a). Subtracting this average from each action value centers the advantages around zero under that policy. Thus sum_a pi(a|s) A_pi(s, a) = 0 when the values are exact and internally consistent.
Learning systems usually estimate these quantities from data rather than knowing them exactly. Actor-critic methods can use advantage estimates to guide policy updates, while a critic supplies value information. One-step temporal-difference residuals and combinations of multiple residuals provide possible estimators. An observed residual is a noisy sample, not automatically the exact advantage.
With the true policy value function, the conditional expectation of the one-step residual R + gamma V_pi(S_next) - V_pi(s), given the current state and action, equals the advantage. Approximate values can introduce estimation error. A useful implementation keeps the mathematical quantity, its estimator and the update rule distinct rather than giving all three the same meaning.

Three practical examples

Simulated scheduling: an action can be compared with the current scheduling policy's average behavior at a particular decision point. The comparison concerns the modeled reward, not every business outcome.
Game learning: an agent estimates whether a move is better than its usual choice in that position, considering expected future play as well as immediate reward.
Policy diagnostics: a team checks whether sampled advantage estimates show systematic errors for certain states. A positive estimate can motivate investigation without proving that a policy update will improve performance.

Think of it this way

Imagine choosing among three routes from the same starting point. You want to compare one route with what your usual route-selection habit achieves on average. That comparison is different from asking whether the route is good in absolute terms. The analogy also depends on what counts as success: travel time, comfort and cost need not rank routes identically.

Worked example

Consider a one-step decision ending after the chosen action. Three actions have expected rewards 10, 6 and 2. The current policy chooses them with probabilities 0.2, 0.5 and 0.3. Because the decision is terminal, these expected rewards are also the action values; no future-value calculation is required.
The policy value is 0.2 times 10 + 0.5 times 6 + 0.3 times 2 = 5.6. The three advantages are therefore 10 - 5.6 = 4.4, 6 - 5.6 = 0.4 and 2 - 5.6 = -3.6.
Their policy-weighted average is 0.2 times 4.4 + 0.5 times 0.4 + 0.3 times (-3.6) = 0.88 + 0.20 - 1.08 = 0. This zero average uses the policy probabilities. The ordinary unweighted average is 0.4, so simply averaging action advantages equally would not reproduce the relevant identity.
Now change the policy probabilities to 0.7, 0.2 and 0.1. The new value is 7 + 1.2 + 0.2 = 8.4. Action values remain unchanged in this one-step example, but advantages become 1.6, -2.4 and -6.4. The middle action changes from positive to negative advantage even though its expected reward remains 6. The stronger policy provides a higher comparison baseline.
Finally, suppose the middle action's reward is 0 or 12 with equal probability, preserving its expectation of 6. Under the original policy baseline of 5.6, individual terminal residuals are -5.6 or 6.4. Their average is 0.4, matching the middle action's advantage. One negative observation therefore does not establish negative expected advantage. These values are stipulated teaching assumptions, not results from a trained agent or deployed system.

Illustrative case study

Fictional company Birch Simulation tests a sequential decision model. Its analysts record the policy version and reward specification alongside advantage estimates. They compare estimated improvements with outcomes from separate evaluation runs rather than judging success solely from training signals.
When an action's advantage changes sign after an update, the team checks whether the policy baseline changed and whether estimates remain reliable. It does not assume that the action's immediate reward deteriorated. This fictional workflow reports no commercial result.

Common misconceptions and mistakes

Advantage is not immediate reward minus an arbitrary global average. Its baseline is the state value for the policy being evaluated. Positive advantage also does not mean an action is globally optimal; it only compares with the referenced policy's average behavior at that state.
The zero-mean identity is policy-weighted and exact only for consistent exact values. A finite sample or approximate critic need not satisfy it perfectly. Normalizing a batch of estimated advantages is a separate training operation, not the definition. Finally, a favorable learning signal does not guarantee a better deployed policy: estimation, optimization and environment differences can affect the result.

Business use

Advantage functions help explain how some reinforcement-learning methods compare decisions with current behavior. For business evaluation, the reward definition and policy reference should remain visible. Teams should distinguish estimated training improvements from measured outcomes, and avoid interpreting advantage units as money or customer value unless the reward was explicitly defined that way.

Check your understanding

Why did the middle action change sign?

Answer: The new policy raised its baseline from 5.6 to 8.4.

Why did the noisy residual average to 0.4?

Answer: Its expected reward was 6 and the baseline was 5.6.

Why specify the policy?

Answer: Advantage is defined relative to its expected behavior.

Related terms

Learn next

Compute policy-weighted action values, then compare exact advantages with estimates from several sampled outcomes.

Sources

Schulman and colleagues, High-Dimensional Continuous Control Using Generalized Advantage Estimation, ICLR 2016, definitions and residual estimators: https://arxiv.org/pdf/1506.02438 . The one-step calculations and company scenario are original illustrations.

Previous: Advantage actor-critic Next: Adverb