Can it work with epsilon equal to zero?

Yes; the worked example uses fully greedy selection.

Must every action have the same starting value?

No, although equal initialization makes the mechanism easier to inspect.

Does constant alpha preserve the initial value forever at full strength?

No. Its direct contribution decays geometrically with updates of that action.

AI Glossary / Reinforcement learning and robotics / Intermediate

Optimistic initial action values

Definition

Optimistic initial action values are deliberately high starting estimates of the rewards or returns available from actions. They can encourage an action-value learner to explore because untried actions continue to look promising after tried actions receive less favorable evidence. This can produce exploration even when the action-selection rule always chooses the currently highest estimate.

What it means

A greedy learner usually exploits what it currently believes is best. If all untried actions begin with generous estimates, trying one and receiving an ordinary outcome can lower its estimate below those of the alternatives. The learner then tries another action without needing a separate random-exploration decision at that moment.

It is a way to influence early information gathering. Its effect depends on the update rule, reward scale, tie-breaking and environment. Initial optimism alone is not a general solution to exploration throughout a changing task.

How it works

Choose starting Q-values above the plausible expected rewards or returns, with the scale matched to what Q represents. In a bandit, Q estimates an immediate expected reward. In a sequential problem, it estimates a discounted or otherwise specified return, which can have a different scale.

Select an action with the highest current estimate. Observe the reward or learning target, then update that action's estimate. A common bandit update is Q_new equal to Q_old plus alpha times reward minus Q_old. If reward is below Q_old, the estimate falls.

Unselected actions retain their optimistic values in a tabular bandit. Repeating this process can make the learner inspect several options. With constant alpha, the initial estimate's direct weight after k updates of that action is (1 minus alpha) to the power k. With the usual sample-average update and zero initial observation count, the first observation replaces the initial value completely.

The induced exploration is temporary. Once estimates settle, the greedy policy may stop revisiting inferior-looking actions. A new opportunity or changed reward does not automatically restore the original optimism. Function approximation can also couple action estimates, so changing one prediction may change others.

Three practical examples

1. A stationary bandit simulation uses optimistic estimates to test whether greedy selection discovers rewarding alternatives early.

2. An educational comparison holds the update rule fixed and changes only initialization to reveal its effect on action selection.

3. A sequential-learning prototype checks whether its initial values are optimistic relative to total discounted returns rather than merely immediate rewards.

Think of it this way

Think of giving every unfamiliar option a provisional high rating. Trying one replaces some of that favorable assumption with evidence, allowing untouched options to move to the front. Eventually the provisional ratings fade. If the world later changes, those original ratings do not return on their own.

Worked example

Consider an original deterministic three-armed bandit. Action A always gives reward 1, B gives 2 and C gives 3. Initialize all estimates to 6. Use learning rate alpha equal to 0.5, always select a maximum estimate, and break ties alphabetically.

On the first step, choose A. Its estimate becomes 6 plus 0.5 times (1 minus 6), or 3.5. B and C remain at 6, so the second step selects B and changes its estimate to 4. The third step selects C and changes its estimate to 4.5.

C remains highest, so the fourth step selects it again and reduces its estimate to 3.75. B is now highest at 4, so the fifth step selects B and reduces it to 3. The sixth step selects C, reducing its estimate from 3.75 to 3.375.

A, still at 3.5, is highest on the seventh step. Selecting it reduces its estimate to 2.25. C is then selected on the eighth step and moves from 3.375 to 3.1875. The first eight choices are A, B, C, C, B, C, A, C.

After C has been selected three times, its estimate is 3.375. Its remaining excess above the true reward 3 equals (6 minus 3) times 0.5 cubed, or 0.375. The calculation confirms how constant-step updates gradually remove initial optimism.

For contrast within this same deterministic example, initialize every estimate to zero. Alphabetical tie-breaking selects A first, and its estimate becomes 0.5. Since B and C remain at zero, pure greedy selection keeps choosing A and never discovers C. This illustrates a mechanism, not a guarantee for noisy or changing environments.

Illustrative case study

In a fictional project, Elm Experiments compares simple exploration strategies in a simulator. Optimistic initialization helps a greedy learner sample several alternatives at the beginning of a stationary test.

The team then changes one previously unattractive action's reward midway through the run. The learner does not promptly revisit it, revealing the temporary nature of the exploration. The report separates early discovery from adaptation after change and compares approaches that maintain an ongoing reason to explore. It does not present a strong initial result as proof that the same setup will track future changes.

Common misconceptions and mistakes

Optimism is not an upper-confidence bound. A large initial number does not automatically represent a calibrated probability statement or a count-dependent uncertainty bonus.

Greedy selection does not imply that only one action will ever be tried; optimistic estimates can make the greedy choice move between actions. Conversely, high initialization does not guarantee sufficient long-run exploration or optimal performance.

Avoid arbitrarily enormous starting values. They can prolong misleading estimates and waste interactions. Also specify the update convention: a pseudo-count prior and a zero-count sample average retain initialization differently.

Business use

The idea is useful as a simple baseline for controlled learning experiments. Its practical appeal is ease of implementation, but the cost is sensitivity to reward scale and limited ability to renew exploration. Compare cumulative reward, early coverage and adaptation after changes.

Check your understanding

Why is B selected on the fifth step?

Answer: C's updated estimate has fallen below B's remaining optimistic estimate.

What is missing when rewards change much later?

Answer: An ongoing mechanism that encourages renewed exploration.

Related terms

Greedy policy
Upper confidence bound
action-value estimate
epsilon-greedy exploration
multi-armed bandit
learning rate and nonstationarity

Learn next

Next, repeat the example with sample-average updates. Then introduce noisy rewards and compare how often initialization leads the learner to revisit each action.

Sources

Primary source: Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, author draft, section on optimistic initial values: https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf

The deterministic bandit sequence and fictional project are original teaching material. The worked example specifies both its constant learning rate and tie-breaking rule.

Previous: Optimism under uncertainty Next: Optimistic knowledge gradient