AI Glossary / Reinforcement learning and robotics / Beginner
Greedy policy
Definition
A greedy policy chooses an action with the highest current estimated value at each decision point. In reinforcement learning, it is normally greedy with respect to an action-value function or another specified evaluation rule. It exploits the available estimates without deliberately selecting lower-valued actions simply to gather information.
What it means
Greedy describes how a policy uses its current values, not whether those values are correct. If the estimates favor action A, the policy selects A even when an unexplored action might actually be better.
The relevant value can include future rewards. A greedy policy based on long-term action values is therefore not necessarily shortsighted about time. It is shortsighted about improving knowledge when it never departs from the current ranking for exploration. Always state what quantity the policy is maximizing.
How it works
At the current state, evaluate or look up the value assigned to each available action. Identify the maximum and choose an action that attains it. If several actions tie, use a defined tie-breaking rule, which may itself be random.
As learning updates the estimates, the greedy choice can change. The selection rule does not specify how estimates were obtained. They might come from a known model, observed experience or a learned critic. Greediness relative to an imperfect estimate does not establish optimality, whereas selecting maximum actions from a correct optimal action-value function yields an optimal policy under the relevant decision-model assumptions.
Three practical examples
1. A bandit agent always chooses the option with the highest estimated average reward. Without additional exploration, early estimation errors can persist.
2. A game agent selects the action with the highest learned discounted-return estimate, even if that action gives a smaller immediate reward than another move.
3. A planning system chooses a maximum-value action from an accurately solved finite decision model. Its justification depends on the model and objective, not on the word greedy alone.
Think of it this way
Think of choosing the route that your current estimates say will get you there fastest. You always follow the best-rated route, but the rating might include the entire journey rather than just the first street. If another route has never been tried, always following the favorite may leave its potential undiscovered.
Worked example
Consider a two-step toy decision. At the starting state, action A gives an immediate reward of 5 and ends the episode. Action B gives reward 1 and moves to a state where the next and final reward is 10. Let the discount factor be 0.9.
A's total discounted value is 5. B's is 1 + 0.9 × 10 = 10. A policy greedy with respect to these correct long-term values chooses B, despite B's smaller immediate reward.
Now suppose a learner's current estimates are Q(A) = 6 and Q(B) = 4. A greedy policy based on those estimates chooses A. The selection rule is functioning as specified, but the estimates lead it away from the truly better action in this toy model.
If both estimates were 6, either action would be greedy, and a random tie-break would remain greedy because it selects only maximizers. This example separates the value being estimated, the accuracy of that estimate and the action-selection rule.
Illustrative case study
Fictional case: Pine Scheduling evaluates a simulated resource-allocation agent. The team notices that its greedy evaluation policy repeatedly chooses a familiar allocation. A colleague calls the behavior short-term thinking, but inspection shows that its values already include discounted future rewards.
The actual problem is poor evidence for an alternative allocation. The team compares value estimates with independently evaluated outcomes and reviews the training exploration process. It reports the difference between greedy evaluation and exploratory training. This invented case illustrates diagnosis; it makes no claim that one particular policy or exploration method improved a real scheduling operation.
Common misconceptions and mistakes
Greedy does not always mean choosing the largest immediate reward. The objective may be estimated cumulative return. It also does not mean selfish, unethical or aggressive; this is a technical selection rule.
Random tie-breaking is compatible with greediness when all selected actions share the maximum value. A policy that sometimes chooses a genuinely lower-valued action for exploration is not purely greedy at those decisions. Finally, a greedy policy can be optimal when its values are appropriate and accurate, but a greedy selection rule alone cannot correct bad estimates or an unsuitable reward definition.
Business use
A greedy policy offers a clear baseline and a useful way to evaluate what a learned value function currently recommends. It can also be appropriate when a reliable model already supports the decision and deliberate exploration is unnecessary for the task.
For business assessment, examine the quality and coverage of the estimates, the reward objective and how conditions change. Separate learning-time exploration from deployment or evaluation choices. Report performance on independently assessed outcomes rather than assuming that selecting the largest model score necessarily produces the best operational result.
People also ask
Can a greedy policy be stochastic?
Yes, if it randomly chooses among tied maximum-value actions. It still avoids lower-valued actions.
Is greediness always a bad strategy?
No. It depends on the accuracy of the values and the need to learn about alternatives.
How is epsilon-greedy different?
Epsilon-greedy adds an exploratory branch with a specified probability. That branch can select actions outside the set of current maximizers.
Check your understanding
Why does the worked example's policy choose B when it uses correct long-term values, even though A gives an immediate reward of 5?
Answer: B's discounted total is 10, which exceeds A's total of 5. Greediness is relative to the specified value estimate, so choosing the largest expected return can include future consequences.
Related terms
- Policy
- Epsilon greedy policy
- Optimal policy
- Critic
- Action value
- expected return
- discounted reward
- exploration-exploitation
- tie-breaking
Learn next
Distinguish immediate reward from expected cumulative return. Calculate a small action-value table and identify all maximizers, including ties. Then examine how estimation error and limited exploration change the behavior of a greedy policy without changing its selection rule.
More in Reinforcement learning and robotics
- Global costmap - A global costmap is a spatial cost representation used to support planning a route across an operating area.
- Gradient bandit algorithm - A gradient bandit algorithm learns action preferences and converts them into action-selection probabilities, typically with a softmax function.
- Gradient temporal-difference learning - Gradient temporal-difference learning refers to methods that estimate value functions by following gradients of carefully defined temporal-difference...
- GTD2 - GTD2 is a gradient temporal-difference algorithm for learning a linear value-function approximation using a primary parameter vector and an auxiliary vector.
- Hierarchical reinforcement learning - Hierarchical reinforcement learning is an approach to learning sequential decisions at more than one level of abstraction.
- Human - robot collaboration - Human - robot collaboration is coordinated activity in which people and robots contribute to a shared task or goal.
Sources
Google, Machine Learning Glossary, greedy policy and expected return: https://developers.google.com/machine-learning/glossary
Poole and Mackworth, Artificial Intelligence: Foundations of Computational Agents, Decision Processes, policies and state values: https://www.artint.info/3e/html/ArtInt3e.Ch12.S5.html
The two-step calculation and Pine Scheduling case are original hypothetical illustrations.
Previous: Greedy embedding Next: Greedy search