Is policy evaluation the same as policy iteration?

No. Policy iteration alternates evaluation with steps intended to improve the policy.

Does evaluation require a known transition model?

No. Some methods use a model, while others learn value estimates from experience.

Can two starting states have different values under one policy?

Yes. The same action rule can lead to different expected returns depending on the starting situation.

AI Glossary / Causal inference and experimentation / Intermediate

Policy evaluation

Definition

Policy evaluation is the task of estimating how much future reward an agent can expect when it follows a specified policy. In reinforcement learning, a policy determines how actions are chosen in each state. Evaluation estimates that policy's value, rather than searching directly for the best policy. The term here concerns sequential decision-making; it is distinct from reviewing an organization's written rules or evaluating public policy.

What it means

A policy might direct a simulated warehouse robot's movements. Its value includes later consequences: an apparently convenient action can lead to delays or rewards later.

Policy evaluation asks what cumulative outcome is expected if the agent keeps following that policy. The answer can be a state-value function, assigning an expected return to each starting state, or an action-value function, assessing an initial action followed by the policy.

Evaluation depends on the reward definition, environment, and horizon or discounting convention. A policy can score well under one reward scheme and poorly under another. Value is a mathematical quantity, not an unrestricted judgment that a policy is good.

How it works

Specify the policy and the return to be evaluated. A discounted return adds successive rewards while multiplying later rewards by powers of a discount factor, often written gamma. An episodic task can instead end at a terminal state under an appropriate return definition.

With a known environment model, compute expected immediate rewards and successor-state values under the policy. Bellman expectation equations connect these quantities. Iterative policy evaluation repeatedly updates value estimates using that relationship until a chosen accuracy criterion is met under suitable conditions.

When the environment model is unavailable, experience-based methods can estimate values from observed trajectories. Monte Carlo methods use sampled returns, while temporal-difference methods update estimates using rewards and estimated successor values. Whatever the method, distinguish uncertainty due to limited experience from errors in the environment model or function approximation.

Three practical examples

1. Simulated navigation: engineers estimate expected accumulated travel penalties for a fixed robot movement policy before comparing it with alternatives.

2. Queue management: analysts evaluate a specified scheduling rule in a model of incoming jobs, using a reward that reflects the intended waiting-time objective.

3. Inventory simulation: a retailer estimates the return from a defined reorder policy under an explicit demand model, then examines sensitivity to different demand conditions.

Think of it this way

Think of estimating the total travel time for a route plan that tells you what to do at every junction. Evaluation follows the plan's consequences; choosing a different turn belongs to improving the plan. The analogy is limited because reinforcement-learning environments can be stochastic, actions may be randomized, and the measured return can combine rewards that are not physical travel time.

Worked example

Consider a fictional environment with states A, B, and a terminal state T. A fixed policy moves from A to B and receives reward 2. From B it moves to T and receives reward 5. T has no further reward. Let gamma = 0.9.

The value of B is 5 because the next reward ends the episode. The value of A is 2 + 0.9 × 5 = 6.5. This evaluates the stated policy, not every possible action available at A or B.

Starting iterative evaluation with both nonterminal values at zero, a synchronous first update gives V(A) = 2 and V(B) = 5. The second gives V(A) = 6.5 and V(B) = 5. Values have propagated backward along the policy's transitions. In a stochastic environment, updates would average over actions chosen by the policy and possible outcomes. The small deterministic example reaches its answer quickly because the episode has only two transitions.

Illustrative case study

Fictional case: Oakline Dispatch tests a scheduling policy in a simulator. Its first report gives the average immediate reward per decision. A reviewer points out that some choices postpone difficult jobs, making the next reward look good while creating later delays.

The team defines an episode and evaluates cumulative return under the fixed policy. It also reports waiting-time outcomes directly, because the chosen reward is only a representation of operational priorities. Before drawing deployment conclusions, it checks whether the simulator captures the relevant arrival patterns. This illustrates careful evaluation scope, not an actual company's measured results.

Common misconceptions and mistakes

Policy evaluation does not necessarily improve the policy. It can provide information for policy improvement, but predicting the consequences of a fixed rule and changing that rule are different steps.

A single trajectory is not usually the expected value. Random transitions and rewards can produce different outcomes, so sampled estimates need enough relevant experience and appropriate uncertainty assessment. Likewise, a precise solution to an inaccurate simulator evaluates that simulator, not automatically the real operation.

Do not replace the policy's action probabilities with whichever action currently looks best and still call it evaluation of the original policy. That changes the update's purpose. Off-policy evaluation also requires care when experience was generated by a different policy; the observed data may not adequately cover the actions being assessed.

Business use

Policy evaluation supports comparisons of sequential operating rules when choices affect later situations. It helps businesses examine delayed consequences that a one-step metric might miss.

Reports should identify the policy version, reward definition, initial-state distribution, horizon, and environment assumptions. Compare interpretable operational outcomes alongside the return. If the evidence comes from a simulator or historical data, explain what it supports and where additional validation is needed before relying on the policy in practice.

Check your understanding

Under a fixed policy, a state gives immediate reward 3 and then reaches a state valued at 8. With gamma = 0.5 and a deterministic transition, what is the first state's value?

Answer: It is 3 + 0.5 × 8 = 7, assuming the successor value already represents the remaining return under that policy.

Related terms

Policy
Return
Discount factor
Bellman equation
Policy improvement
value function
Monte Carlo prediction
temporal-difference learning

Learn next

Draw a three-state episodic environment and fix one policy. Calculate its values backward from the terminal state. Then change one action in the policy and explain why this creates a new evaluation problem.

Policy

Sources

Sutton and Barto, Reinforcement Learning: An Introduction, second-edition draft, policy evaluation and prediction chapters, author text hosted by Stanford: https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf

The accessible draft grounds the reinforcement-learning meaning and evaluation updates. The current author-hosted PDF retrieval failed; this source is explicitly a historical draft. Calculations and case are original illustrations.

Previous: Policy capturing Next: Policy gradient