What happens when epsilon is zero?

The rule always uses its greedy branch, with whatever tie-breaking convention is defined.

What happens when epsilon is one?

It always uses the exploration distribution, which need not exclude the action with the highest estimate.

Must epsilon decrease?

No. Fixed and changing schedules serve different purposes. A decreasing schedule is a design choice, not part of every epsilon-greedy definition.

AI Glossary / Reinforcement learning and robotics / Beginner

Epsilon greedy policy

Definition

An epsilon greedy policy is a decision rule that usually chooses an action with the highest current estimated value but sometimes explores through random choice. Epsilon, written as the Greek letter epsilon or a number between zero and one, controls the probability of taking the exploratory branch.

What it means

The rule balances using what an agent currently believes with collecting information about alternatives. If epsilon is 0.2, the exploratory branch is used with probability 20%, and the greedy branch with probability 80%.

That does not necessarily mean a non-greedy action is chosen 20% of the time. In the common convention where exploration samples uniformly from all valid actions, random selection can also choose a greedy action. Implementations must state their exploration and tie-breaking conventions to make the probabilities clear.

How it works

Maintain an estimated value for each available action at the current state. Draw a random number to choose between exploration and exploitation. On the greedy branch, select an action with maximum estimated value. On the exploratory branch, sample according to the defined exploration distribution, often uniformly across available actions.

After observing experience, a learning algorithm may update the value estimates. Epsilon-greedy itself specifies action selection, not the complete learning update. Epsilon can remain fixed or follow a schedule. Reducing it is common, but the schedule must suit the problem; decreasing exploration too quickly can leave alternatives poorly understood.

Three practical examples

1. A simple game agent occasionally tries a move that does not currently have the highest estimated return, allowing its value estimates to improve with experience.

2. A simulated machine chooses among several permitted operating settings. Random exploration is restricted to valid settings rather than including impossible commands.

3. A bandit demonstration compares fixed epsilon with a decreasing schedule. The class examines both total reward and how often each action was sampled.

Think of it this way

Think of ordering from a familiar menu. Most of the time you select the option you currently expect to enjoy most, but occasionally you choose randomly to learn about alternatives. A random choice may still land on your favorite. The rule is simple, although it does not distinguish a promising unfamiliar dish from one already known to be disappointing.

Worked example

Suppose four valid actions have estimated values A = 8, B = 5, C = 3 and D = 1. A is the unique greedy action. Set epsilon to 0.2 and let exploration choose uniformly from all four actions.

A is selected through the greedy branch with probability 0.8 and through the exploratory branch with probability 0.2/4 = 0.05. Its total probability is 0.85. Each other action has probability 0.05. The probability of a non-greedy action is therefore 0.15, not 0.2.

Across 200 independent choices with these values held fixed, expected counts are 170 for A and 10 each for B, C and D. Actual counts need not match exactly.

If B also had value 8 and the greedy branch broke ties uniformly, A and B would each have probability 0.8/2 + 0.2/4 = 0.45, while C and D would remain at 0.05. This example explicitly fixes the convention; exploring only non-greedy actions would produce different probabilities.

Illustrative case study

Fictional case: Willow Testing compares routing rules in a simulator. A developer sets epsilon to 0.1 and expects exactly one in ten actions to differ from the greedy choice. Review reveals that the random branch includes that same greedy action, so the expectation was incorrectly stated.

The team documents the sampling convention, logs branch decisions separately from selected actions, and checks valid-action handling. It then compares reward and failure outcomes across repeated runs. This invented case illustrates verification of exploration logic, not a claim that random exploration is suitable for unrestricted use in a real operation.

Common misconceptions and mistakes

Epsilon is a probability, not a promise of an exact number of exploratory choices in a short run. It also does not directly equal the probability of choosing a non-greedy action when exploration includes greedy choices.

The highest estimated value may be wrong, so exploitation is not necessarily optimal behavior. Uniform exploration can waste trials on clearly poor actions and may be unsuitable when trials have serious consequences. Finally, epsilon-greedy does not by itself guarantee convergence: the learning rule, visitation, learning rates, environment assumptions and exploration schedule all matter.

Business use

The policy is a useful baseline for simple repeated-decision experiments because its behavior is easy to explain and test. It can help teams understand the cost of exploration before adopting more specialized uncertainty-aware methods.

Define the permitted action set and measure exploration outcomes rather than treating epsilon as a harmless technical setting. Compare configurations under the same interaction budget and separate training behavior from final policy evaluation. If operating conditions change, a schedule that removes exploration may make it harder to detect that previously unattractive actions have improved.

Check your understanding

With four actions, one unique greedy action and epsilon 0.2, why is the greedy action chosen with probability 0.85 in the example?

Answer: It receives 0.8 probability from exploitation and an additional 0.05 from uniform exploration. The branches are distinct, but they can lead to the same action.

Related terms

Greedy policy
Random policy
Q-learning
Policy evaluation
exploration-exploitation
action value
bandit
action masking
epsilon decay

Learn next

Calculate selection probabilities for a few actions before implementing the rule. Include ties and changing valid-action sets. Then compare a fixed epsilon and a defined schedule in a small simulator, reporting both rewards and action coverage without assuming that more exploration is always better.

Sources

Google, Machine Learning Glossary, epsilon greedy policy and greedy policy: https://developers.google.com/machine-learning/glossary

Lattimore and Szepesvari, Bandit Algorithms, author-hosted book for repeated-decision and exploration context: https://tor-lattimore.com/downloads/book/book.pdf

The action probabilities and Willow Testing case are original hypothetical illustrations. The worked example uses uniform exploration over all valid actions.

Previous: Epsilon-ball Next: Equal impact