Why is it sometimes modeled with two choices?

One is uncertain; the other is a known baseline.

Must rewards be monetary?

No. They can represent any consistently defined objective.

Is a stopping time a predetermined number of trials?

Not necessarily. It can depend on outcomes observed so far.

AI Glossary / Reinforcement learning and robotics / Advanced

One-armed bandit

Definition

A one-armed bandit is a sequential decision problem involving one uncertain reward-producing option and a known alternative, such as a fixed reward or the option to stop. The learner decides whether further use of the uncertain option is worthwhile. In ordinary language the phrase also means a slot machine; in decision theory it highlights the choice between continued experimentation and a known baseline.

What it means

There is only one option whose payoff needs to be learned, but there is still a meaningful decision. Trying it can produce both an immediate reward and information that affects later choices. Taking the known alternative avoids that experiment and provides its specified payoff.
The problem can also be expressed as repeatedly trying one option at a cost, with stopping as the alternative. A known reward forgone by experimenting acts as an opportunity cost. The precise horizon, discounting, reward model, and stopping rules must be stated before deciding what strategy is best.

How it works

In a Bayesian formulation, the learner begins with a probability distribution describing uncertainty about the arm's reward parameters. Each observed outcome updates that distribution. The decision depends on both the current estimate and the value of information for the opportunities that remain.
A finite-horizon dynamic program compares the expected total reward from experimenting now with that from choosing the known alternative. An immediate comparison of current means can miss the value of learning. Information has little decision value if there will be no later opportunity to act on it.
For the standard stationary Bayesian setting with a known constant alternative and a known finite horizon, optimal behavior can be represented by a retirement policy: experiment until a suitable stopping time, then use the known alternative for the remaining opportunities. This structure is assumption-dependent, not a rule for every changing environment.
If the uncertain arm changes while unused, observations arrive from elsewhere, or the baseline changes, the decision problem can be different. Even within the simple setting, a strategy that is optimal under one prior or horizon may be unsuitable under another.

Three practical examples

Process trials: compare a new operating procedure of uncertain quality with an established procedure having a specified baseline outcome.
Equipment evaluation: decide whether another test of one candidate configuration is worth its cost before settling on the known alternative.
Simulated learning experiments: study how the number of remaining opportunities changes the incentive to gather information about one uncertain action.

Think of it this way

Think of evaluating one unfamiliar lunch counter while having a dependable packed lunch available. Trying the counter teaches you something useful for future days. If today is your last day in the neighborhood, that information has less practical value. The analogy assumes your packed lunch's value is already known and your preferences remain stable.

Worked example

Consider a fictional two-round problem with no discounting. The uncertain arm gives reward 1 on success and 0 on failure. Its fixed but unknown success probability p has a uniform prior on [0, 1], equivalently Beta(1, 1). The known alternative gives reward 0.55 every round and provides no information about p.
With one round remaining under the initial prior, the uncertain arm's expected reward is 0.50, so choosing the known alternative yields the higher expectation, 0.55.
With two rounds remaining, consider trying the uncertain arm first. Its expected first reward is 0.50. A success occurs with predictive probability 0.50 and updates the posterior to Beta(2, 1), whose mean is 2/3. A failure also has probability 0.50 and gives Beta(1, 2), whose mean is 1/3.
After success, choose the uncertain arm again because 2/3 exceeds 0.55. After failure, choose the known alternative because 0.55 exceeds 1/3. The expected two-round total is therefore 0.50 + 0.50 × (2/3) + 0.50 × 0.55 = 133/120, approximately 1.10833.
Choosing the known alternative first leaves the prior unchanged. With only one round left, the best choice is again the known alternative, for total expected reward 1.10 = 132/120. Experimenting first is therefore better by 1/120, about 0.00833 reward units, under these exact assumptions.
The result does not say the uncertain arm has a higher current mean. It shows that a small information benefit can outweigh its immediate disadvantage when another decision remains. Realized rewards can still be lower than those from the baseline.

Illustrative case study

Fictional case: AspenTesting considers one candidate machine setting against a documented baseline. The team initially plans to continue trials whenever the candidate's average result exceeds the baseline. It then recognizes that trial cost, remaining production opportunities, and uncertainty all affect the decision.
The analysts build a small model with explicit assumptions, examine how conclusions change under different priors, and keep model-based expectations separate from measured outcomes. This fictional case illustrates a structured question for decision analysis. It does not claim that a specific stopping rule improves actual manufacturing performance.

Common misconceptions and mistakes

One arm does not mean there is no choice: stopping or taking a known alternative is part of the formulation. A low current mean need not make experimentation worthless when learning can improve later decisions. Conversely, uncertainty alone does not justify endless testing. The number of remaining opportunities and the cost of obtaining information matter. The term is not a claim that a real slot machine can be profitably learned.

Business use

This model can clarify whether to continue evaluating one uncertain option against a meaningful baseline. Define reward units consistently, include the consequences of failed trials, and distinguish known assumptions from estimated quantities. Use sensitivity analysis to see whether the preferred decision depends on a fragile prior or horizon assumption. A simple one-armed model may be inadequate when there are several uncertain alternatives or interactions between projects.

Check your understanding

Why is experimentation preferred in the two-round example?

Answer: Its information can improve the second decision.

What happens after the first failure?

Answer: The known alternative is selected for the final round.

Would the initial uncertain arm be preferred with only one round left?

Answer: No. Its mean 0.50 is below 0.55 and there is no later decision to benefit from learning.

Related terms

Multiarmed bandit
Opportunity cost
Finite horizon
optimal stopping
Bayesian updating
Beta distribution
exploration and exploitation
value of information

Learn next

Next, vary the baseline reward and remaining horizon in a small decision tree. Then compare this single-uncertain-option problem with competition among several uncertain arms.

Decision tree

Sources

Tor Lattimore and Csaba Szepesvari, Bandit Algorithms, Section 35.3, one-armed Bayesian bandits: https://tor-lattimore.com/downloads/book/book.pdf Alexander Goldenshluger and Assaf Zeevi, Woodroofe's One-Armed Bandit Problem Revisited, author abstract: https://arxiv.org/abs/0909.0119

Previous: On-policy Next: One-factor-at-a-time method