AI Glossary / Reinforcement learning and robotics / Intermediate

Policy value

Definition

Policy value is the expected return obtained by following a specified policy under a specified decision problem. It may refer to the value from one starting state, represented by a state-value function, or to an overall value averaged over a starting-state distribution. The policy, reward definition, horizon, and discounting must be stated for the number to have a clear meaning.

What it means

A policy describes what an agent does; its value describes the consequences expected from doing it. Two policies can make different choices in the same environment and therefore have different values. The same policy can also have different values when it starts in different situations.
Value is not simply the reward from the next action. It usually includes later rewards as well. Nor is it necessarily the best return achievable: evaluating a policy asks how that particular behavior performs, while finding an optimal policy asks which behavior performs best.

How it works

For a discounted return, future rewards are multiplied by successive powers of a factor γ. The state value Vπ(s) averages that return over the action choices of policy π and the environment's possible responses, starting at s.
The action value Qπ(s, a) is related but different. It evaluates taking action a first and then following π. Averaging these action values using the policy's action probabilities gives Vπ(s). The average must follow the policy being evaluated, rather than automatically selecting the largest action value.
When a transition and reward model is available, policy evaluation can solve Bellman expectation equations. With observed episodes, suitable averages or other learning methods can estimate value. Estimation error, coverage, and differences between the data-collection policy and target policy affect how reliable those estimates are.
For an overall performance measure, specify a starting-state distribution and average state values accordingly. A policy that looks good under one distribution may perform poorly under another. Continuing tasks can also use an average-reward criterion, which is different from a discounted sum and should not be mixed with it.

Three practical examples

• Robot task evaluation: estimate expected total task reward for a controller starting from specified configurations.
• Service management: compare fixed routing policies using the same reward definition and incoming-case distribution.
• Simulation analysis: evaluate an existing strategy before deciding whether a policy-improvement step is worthwhile.

Think of it this way

Think of assessing a travel plan rather than one road segment. Its value depends on where the journey begins, which route choices the plan makes, and what outcomes can occur along the way. A single unusually quick journey does not establish the plan's expected travel quality.

Worked example

Consider a fictional discounted task with one nonterminal state S and one terminal state with zero continuation value. At S, Cashout gives reward 4 and terminates. Retry gives reward 1, then returns to S with probability 0.75 or terminates with probability 0.25. Let γ = 0.80.
The policy being evaluated chooses Retry with probability 0.60 and Cashout with probability 0.40 every time it visits S. Its expected immediate reward is 0.60 × 1 + 0.40 × 4 = 2.2. The probability of returning to S under this policy is 0.60 × 0.75 = 0.45.
Its Bellman expectation equation is therefore Vπ(S) = 2.2 + 0.80 × 0.45Vπ(S). Rearranging gives 0.64Vπ(S) = 2.2, so Vπ(S) = 3.4375.
Check the result through action values. Cashout has Qπ(S, Cashout) = 4. Retry has Qπ(S, Retry) = 1 + 0.80 × 0.75 × 3.4375 = 3.0625. The policy-weighted average is 0.60 × 3.0625 + 0.40 × 4 = 3.4375, agreeing with the equation.
The maximum of those action values is 4, but substituting that maximum would change the question from evaluating this mixed policy to considering an improvement. A policy that always cashes out is worth 4. A policy that always retries instead satisfies V = 1 + 0.80 × 0.75V, giving V = 2.5.
Finally, if an evaluation begins at S with probability 0.80 and already terminal with probability 0.20, this mixed policy's overall expected value is 0.80 × 3.4375 = 2.75. That scalar differs from its state value because the initial-state weighting differs.

Illustrative case study

Fictional case: WillowServices compares two fixed routing rules. An initial report averages performance over a test set dominated by easy cases, while the intended operation contains more difficult cases. The team keeps the rules unchanged but recalculates the evaluation using a clearly specified case distribution.
It reports expected performance and uncertainty separately, rather than describing one observed run as the policy's true value. This fictional case illustrates why the starting conditions and evaluation method belong beside a policy-value claim. It does not report an actual service improvement or suggest that reweighting fixes missing coverage.

Common misconceptions and mistakes

Policy value is not the policy itself, a probability of taking an action, or the immediate reward alone. It is not automatically an optimal value. A sampled return can be above or below the expectation. The value of a state under one policy cannot be transferred to another policy without justification, and values calculated with different reward scales or discount factors are not directly comparable.

Business use

Use policy value to compare candidate decision rules under consistent conditions. Define what rewards include, how episodes end, and which starts are relevant. Record whether the figure comes from an exact model, simulation, or historical-data estimation. Include uncertainty when estimates depend on finite samples. If the objective changes, reevaluate policies instead of carrying forward an old value label.

People also ask

Can a policy value be negative?

Yes, when the reward definition includes costs or negative outcomes.

Does a higher discounted value always mean better immediate performance?

No. Future outcomes may drive the difference.

Can two policies have the same value?

Yes. They may tie from some or all starting states.

Check your understanding

Why is the continuation probability 0.45 in the example?

Answer: Retry is chosen with probability 0.60 and returns to S with probability 0.75.

Why not use the maximum action value when evaluating the mixed policy?

Answer: Its actual choices are probabilistic, so evaluation averages them.

What does 2.75 represent?

Answer: Overall expected value under the stated initial-state distribution.

Related terms

• State-value function
• Policy evaluation
• Discount factor
• Average reward
• action-value function
• expected return
• optimal value
• Bellman expectation equation
• starting-state distribution

Learn next

Next, solve a small policy-evaluation problem with two nonterminal states. Then compare exact values with estimates from sampled episodes and examine how uncertainty changes with the available experience.

• Policy evaluation

More in Reinforcement learning and robotics

• Playout policy - A playout policy is the rule used to choose actions during a simulated continuation of a decision process, especially beyond the explicitly explored part of...
• Policy - A policy is a rule that determines how an agent selects actions from the information available to it.
• Policy gradient - A policy gradient is the derivative of a policy's expected performance with respect to the parameters that control its action choices.
• Policy-gradient baseline - A policy-gradient baseline is a reference quantity subtracted from a return or action-value estimate when forming a policy-gradient update.
• Policy improvement - Policy improvement is the process of changing a decision policy using information about the value of its actions.
• Policy iteration - Policy iteration is a method for finding an improved decision policy by repeatedly alternating policy evaluation and policy improvement.

Sources

Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, second edition, Sections 3.3–3.5 and 4.1: https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf AIMA authors' code documentation, value determination for a fixed policy: https://aima.cs.berkeley.edu/lisp/doc/overview-UNCERTAINTY.html

Previous: Policy search Next: Poly-Weibull distribution