AI Glossary / Reinforcement learning and robotics / Intermediate
Optimal policy
Definition
An optimal policy is a decision rule that achieves the best expected return allowed by a specified decision problem. It selects actions with their immediate and future consequences in view. Optimality is always relative to the reward definition, available information, environment assumptions, constraints, and time horizon; it does not mean universally best behavior in every possible situation.
What it means
A policy tells an agent what to do in the situations it may encounter. Calling it optimal means that no admissible alternative has a better value under the stated comparison. In a standard Markov decision process, an optimal policy can achieve the optimal value from every state under suitable assumptions.
That is stronger than saying a policy won a particular test. A favorable trial may be luck, and a learned policy may be only approximately optimal. Even exact optimization can produce an unsuitable real-world result when the reward or model omits something important.
How it works
For a discounted reward objective, a policy's value is the expected sum of rewards with later rewards multiplied by powers of a discount factor. The optimal value function takes the best achievable value. An optimal action balances immediate reward against the optimal expected continuation from the next state.
This relationship is expressed by the Bellman optimality equation. If the model is known, methods such as value iteration or policy iteration can solve suitable problems. Reinforcement learning attempts to estimate useful decisions from experience when the model is not fully known.
For a finite, fully observed, stationary MDP with bounded rewards and an infinite-horizon discounted objective, an optimal stationary deterministic policy exists. Stationary means the same state leads to the same decision rule over time. This statement should not be generalized without qualification to constrained, partially observed, or differently defined problems.
Finite-horizon decisions may depend on how much time remains. Several policies can also be equally optimal when their returns tie. The optimal value may be unique even though the policy achieving it is not.
Three practical examples
- Resource planning: select actions that balance current output against the future condition of limited resources.
- Simulated navigation: choose a route policy that accounts for expected delays and later movement costs under the model.
- Game strategy: select moves for expected eventual outcome rather than only the immediate score increment.
Think of it this way
Think of choosing whether to finish a small task now or prepare for a larger task tomorrow. The better choice depends on tomorrow's value, the preparation cost, and whether tomorrow is within the planning horizon. The word optimal describes the answer to that specified decision problem, not an eternal ranking of the two actions.
Worked example
Consider a fictional deterministic system with states A and B and an unlimited horizon. At A, action Stay gives reward 2 and returns to A. Action Upgrade gives reward −3 and moves to B. At B there is only one action, Operate, giving reward 4 and returning to B. Let the discount factor be γ = 0.80.
At B, the value is V(B) = 4 + 0.80V(B), so V(B) = 20. Upgrading immediately at A is therefore worth −3 + 0.80 × 20 = 13. Staying forever is worth 2/(1 − 0.80) = 10.
To verify optimality rather than merely compare two selected plans, test the candidate V*(A) = 13. The value of staying once and then following the optimal continuation is 2 + 0.80 × 13 = 12.4. This is below the upgrade value 13. Together with V*(B) = 20, the candidate satisfies both Bellman optimality comparisons. The optimal policy upgrades at A and operates at B.
Now change only the discount factor to 0.50. Then V(B) = 8, upgrading is worth −3 + 0.50 × 8 = 1, and staying forever is worth 4. The candidate V*(A) = 4 satisfies Stay = 2 + 0.50 × 4 = 4 and Upgrade = 1. Staying is now optimal.
The immediate rewards have not changed, but the optimal action has. With just one decision remaining and no terminal reward, Stay would also win because 2 exceeds −3. These calculations show why discounting and horizon are part of the meaning of optimality.
Illustrative case study
Fictional case: ElmOperations compares two policies for a simulated machine-management task. One produces the highest average modeled output but frequently uses an expensive maintenance action that the initial reward does not price. The team adds the relevant cost and resolves the decision problem.
It reports the resulting policy as optimal for the revised small model only after checking the model's decision equations. Separate trials assess whether that model describes actual operation. This fictional case illustrates how an omitted objective component can change the right decision; it does not claim real operational savings.
Common misconceptions and mistakes
Optimal does not mean greedy for immediate reward, best in every realized episode, or free of uncertainty. A policy optimized for one starting-state distribution need not have demonstrated optimality in every state. A large neural network does not automatically find a global optimum. Selecting the best checkpoint from a benchmark run also does not provide a proof of optimality for the underlying task.
Business use
Use the term precisely when comparing proposed automation strategies. State the objective, constraints, horizon, and evidence behind the claim. For small exact models, mathematical verification may be possible; for large learned systems, measured performance and approximation limits are often the appropriate report. Revisit the policy when costs, dynamics, or operating conditions change, because a formerly optimal decision can become inferior.
People also ask
Must an optimal policy be unique?
No. Different actions can tie in value.
Does optimal mean risk-free?
No. Maximizing expected return can still permit undesirable individual outcomes unless the objective or constraints address them.
Can the best action change with time remaining?
Yes, especially in finite-horizon problems.
Check your understanding
Why does Upgrade win when γ = 0.80?
Answer: Its long-term benefit outweighs the immediate cost.
Why compare Stay followed by optimal continuation?
Answer: That checks the Bellman decision, not just a single fixed alternative plan.
Does the same policy remain optimal at γ = 0.50?
Answer: No. Stay becomes optimal at A.
Related terms
- Policy
- Discount factor
- Markov decision process
- Policy iteration
- Value iteration
- Finite horizon
- value function
- action-value function
- Bellman optimality equation
Learn next
Next, distinguish evaluating a fixed policy from finding an optimal one. Then study how reward design, constraints, and uncertainty change the decision problem being solved.
More in Reinforcement learning and robotics
- On-policy - On-policy describes reinforcement learning in which the policy being evaluated or improved is also the policy used to generate the relevant experience.
- One-armed bandit - A one-armed bandit is a sequential decision problem involving one uncertain reward-producing option and a known alternative, such as a fixed reward or the...
- Open-source robotics - Open-source robotics develops or uses robot software, hardware designs, or both under licenses that permit access, modification, and redistribution under...
- Optimistic initial action values - Optimistic initial action values are deliberately high starting estimates of the rewards or returns available from actions.
- Path consistency learning - Path consistency learning, abbreviated PCL, is a reinforcement-learning approach that jointly trains policy and value estimates to satisfy...
- Perceptual robotics - Perceptual robotics studies how principles of perception can inform the design of robots and their interaction with the world.
Sources
Prashanth L. A., Lectures on Reinforcement Learning Theory, Discounted Cost MDPs: https://cse.iitm.ac.in/~prashla/notes/rlbook/discounted-cost-mdps.html Stanford CS234, Lecture 2 notes, value functions and optimal policies: https://web.stanford.edu/class/archive/cs/cs234/cs234.1224/CS234Win2019/slides/lnotes2.pdf
Independent review checked discounted-MDP Bellman optimality conditions.
Previous: Optimal matching Next: Optimal solution