Why explore an action with a lower average?

Its uncertainty may leave substantial potential.

Does UCB1 automatically handle changing reward distributions?

Its standard guarantee assumes a stationary setting, so changing environments require additional consideration.

AI Glossary / Statistics and probability / Intermediate

Upper confidence bound

Definition

An upper confidence bound is an upper limit intended to cover an unknown quantity with a specified confidence under stated assumptions. In machine learning, the term also names a family of action-selection methods that combine an estimated reward with an uncertainty bonus. Such methods favor actions that either look rewarding or remain insufficiently explored. The confidence interpretation depends on how the bound is constructed; an arbitrary estimate-plus-bonus formula is not automatically a calibrated statistical guarantee.

What it means

A purely greedy system chooses the action with the highest observed average reward. It may neglect an action that performed poorly in a few early trials but is actually better. An upper-confidence-bound strategy gives uncertain actions an optimistic score, creating a reason to collect more evidence. As an action is sampled, its uncertainty bonus typically decreases. The approach balances learning and earning, but the relevant guarantees require assumptions about rewards, dependence and whether the environment changes.

How it works

For the classical UCB1 rule with rewards bounded between zero and one, first sample every action. Then score action i by its empirical mean plus sqrt(2 log t / ni), using a consistent convention for the total trial count t and the number ni of observations for that action. Choose an action with the largest score and update its statistics after observing the reward. The bonus reflects a concentration-bound argument. Other UCB algorithms use different uncertainty models, contexts or time-varying rules, so their formulas and guarantees should not be mixed indiscriminately.

Three practical examples

Example 1: A recommendation system explores less-tested options while favoring those with strong observed rewards. Example 2: An AI service routes requests among alternatives to learn which performs well under a suitable reward definition. Example 3: A Bayesian optimization procedure selects an input using predicted value plus an uncertainty term. These share an optimistic principle but are different algorithms. A fixed-sample upper confidence limit used in a statistical report is also related terminology, not necessarily a bandit policy.

Think of it this way

Think of comparing restaurants using both their average ratings and how little you know about them. A place with fewer visits may deserve another trial because its true quality is still uncertain. An optimistic score recognizes that possibility. It does not promise that the next meal will be good or that every uncertainty bonus has a valid probability interpretation.

Worked example

Suppose a two-action bandit has completed 100 trials. Action A has 80 observations and mean reward 0.60; action B has 20 observations and mean reward 0.50. Using t = 100 in the stated UCB1 score, A receives 0.60 + sqrt(2 log(100)/80), approximately 0.9393. B receives 0.50 + sqrt(2 log(100)/20), approximately 1.1786. The rule therefore chooses B despite its lower empirical mean, because it has a larger uncertainty bonus. A score above one is possible even though rewards lie between zero and one; the untruncated score is an optimistic selection index, not a predicted reward probability. This calculation illustrates one decision, not a guarantee about which action is truly best or what reward arrives next.

Illustrative case study

In this fictional case study, Sequoia Help tests two eligible response templates with binary task-success rewards. Its team uses an upper-confidence-bound policy to avoid abandoning a lightly tested template after a few outcomes. Analysts log assignments and reward timing, define how delayed or missing outcomes are handled and monitor whether customer mix changes. They evaluate cumulative performance while recognizing that adaptively collected observations need suitable methods if later used for formal treatment comparisons.

Common misconceptions and mistakes

Common mistakes include treating the upper score as the expected reward, assuming it is always below one, and interpreting it as a universal 95% confidence limit. Another mistake is applying a stationary bounded-reward guarantee to a changing or heavy-tailed environment without adaptation. Unobserved rewards for unchosen actions remain missing. The policy can also optimize an incomplete objective, so a reward that ignores quality or undesirable consequences may produce poor business behavior even if the bandit algorithm works as designed.

Business use

Businesses can use UCB methods when repeated choices provide feedback and exploration has an acceptable cost. Define reward, eligible actions, constraints and the evaluation horizon clearly. Distinguish the goal of maximizing accumulated reward from identifying the best action with a fixed confidence or estimating causal effects precisely. Account for context and changing conditions when the basic stationary model is inadequate. Keep monitoring interpretable so that a high selection score is understood as a combination of performance and uncertainty, not proof of superiority.

Check your understanding

Which action is selected in the example?

Answer: B.

Why?

Answer: Its larger uncertainty bonus outweighs its lower observed mean.

What is B's approximate score?

Answer: 1.1786.

Is that a 117.86% probability of success?

Answer: No; it is a selection index produced by the specified formula.

Related terms

Exploration
Exploitation
Concentration inequality
Regret
Related terms include multi-armed bandit
Thompson sampling and Bayesian optimization

Learn next

Learn next how concentration inequalities motivate exploration bonuses and how reward assumptions enter their guarantees. Compare UCB with randomized exploration and Thompson sampling. Then study contextual and nonstationary bandits when action quality depends on user features or changes over time.

Sources

Sources: Auer, Cesa-Bianchi and Fischer, Finite-time Analysis of the Multiarmed Bandit Problem, https://doi.org/10.1023/A:1013689704352; Stanford CS246, Bandits lecture, https://web.stanford.edu/class/cs246/slides/19-bandits.pdf. These primary research and instructional sources explain UCB action selection. The two-action calculation is original and independently checked; the response-template case is fictional.

Previous: Uplift modeling Next: Upper ontology