Is a Bernoulli bandit the same as a general reinforcement-learning environment?

It is a simpler repeated-choice setting without the full state-transition structure of a general sequential control problem.

Does reward zero mean an action is permanently bad?

No. A good action can fail on an individual trial when its success probability is below one.

Can rewards be monetary amounts?

The standard Bernoulli reward is binary. Variable monetary amounts require a different reward model unless explicitly reduced to a binary indicator.

AI Glossary / Reinforcement learning and robotics / Intermediate

Bernoulli bandit

Definition

A Bernoulli bandit is a multi-armed bandit problem in which each chosen action produces a binary reward, commonly 1 for success and 0 for failure. In the standard stationary version, each action has an unknown fixed success probability. The learner repeatedly chooses an action, observes its reward, and tries to earn high cumulative reward.

What it means

Bandit is a mathematical term for a repeated choice problem with limited feedback. The learner normally sees the outcome of the action it chose, not the outcomes that would have followed every alternative. Bernoulli specifies the reward distribution: there are two possible reward values.

The main challenge is balancing exploration of uncertain alternatives with exploitation of actions that currently look promising. It is not necessary for the application to involve gambling. Binary outcomes such as a successful simulated operation can produce the same mathematical structure.

How it works

Assign each action an unknown success probability p. Selecting that action produces reward 1 with probability p and 0 otherwise. Under the usual stationary assumptions, repeated observations help estimate those probabilities. An algorithm uses the available history to select later actions.

Different algorithms handle uncertainty differently. Epsilon-greedy occasionally explores randomly, upper-confidence methods favor plausible high values, and Thompson sampling draws from posterior beliefs. Their behavior and guarantees depend on assumptions. If success probabilities change over time or depend on observed user context, the basic stationary Bernoulli model may need a nonstationary or contextual extension.

Three practical examples

1. A teaching simulator offers two buttons with different unknown chances of awarding one point. The learner must discover which button is more rewarding while collecting points.

2. A controlled interface experiment treats a completed optional action as a binary outcome. A bandit formulation may allocate trials adaptively, provided the objective and feedback assumptions are appropriate.

3. A simulated machine chooses between alternative handling procedures and observes success or failure. Engineers check whether repeated trials truly have stable probabilities before interpreting stationary-bandit results.

Think of it this way

Think of choosing between several unfamiliar methods, each of which either works or does not on a particular trial. You learn only about the method you actually try. Always repeating an early winner can miss a better method; testing alternatives forever can waste opportunities. The bandit problem makes that learning-versus-earning tension precise.

Worked example

Suppose a fictional two-action bandit has true success probabilities p(A) = 0.7 and p(B) = 0.5. These values are known to the person evaluating this example but hidden from the learner. A fixed evaluation schedule chooses A 80 times and B 20 times.

Expected reward is 80 × 0.7 + 20 × 0.5 = 56 + 10 = 66. An oracle that knew the probabilities and always chose A would have expected reward 100 × 0.7 = 70. The expected shortfall for this schedule is therefore 4, also calculated as 20 × (0.7 - 0.5).

Actual observed totals can differ because rewards are random. Suppose early samples produce 6 successes from 10 A trials and 7 from 10 B trials. Their empirical rates are 0.6 and 0.7, temporarily reversing the true ranking. Those observations do not change the assumed true probabilities or prove B is better. This explains why an algorithm must account for uncertainty rather than equating small-sample rates with ground truth.

Illustrative case study

Fictional case: Rowan Learning tests two optional tutorial layouts in a controlled simulator. A binary reward records whether the simulated learner completes one clearly defined task. The team compares adaptive allocation with a fixed allocation while keeping evaluation conditions explicit.

It discovers that one simulated learner group behaves differently, so a single success probability per layout hides useful context. Reviewers reconsider the model and report completion rates by group before selecting an approach. This invented case demonstrates modeling choices; it reports no real user experiment, business uplift or statistical conclusion about an actual product.

Common misconceptions and mistakes

A Bernoulli bandit does not require all actions to have the same probability. Nor does binary reward mean the outcome is deterministic. The mean reward of a Bernoulli action equals its success probability, but a sample proportion is only an estimate of that mean.

Another mistake is confusing realized reward with expected reward or regret. Random outcomes can make a weaker action look good in a short run. A binary proxy can also omit important outcomes, such as how much effort a success required. Optimizing the proxy does not automatically optimize the broader objective.

Business use

The model is relevant when repeated decisions have simple binary feedback and the organization can define a meaningful reward. Before applying it, examine whether outcomes are delayed, correlated, changing or influenced by context. Those features affect the algorithm and analysis needed.

Adaptive allocation also changes how data are collected. Do not automatically apply statistical conclusions designed for a different experiment. Track exposure, missing feedback and the consequences of exploration. A successful deployment requires an appropriate objective and evaluation design, not just an algorithm that maximizes observed clicks or completions.

Check your understanding

Why is the expected shortfall four in the worked example rather than twenty?

Answer: Each B choice loses 0.2 expected reward compared with A, not one whole reward. Twenty such choices therefore produce 20 × 0.2 = 4 expected shortfall. Individual outcomes remain random.

Related terms

Regret
Thompson sampling
Upper confidence bound
Multi-armed bandit
Bernoulli distribution
exploration-exploitation
expected reward
epsilon-greedy
contextual bandit

Learn next

Understand Bernoulli probabilities and sample proportions first. Simulate a small bandit and compare actual reward with expected reward across repeated runs. Then study how exploration rules use uncertainty and identify which assumptions would fail in your intended application.

Sources

Lattimore and Szepesvari, Bandit Algorithms, author-hosted book, stochastic and Bernoulli bandit formulations: https://tor-lattimore.com/downloads/book/book.pdf

Russo and Van Roy, Learning to Optimize via Posterior Sampling, 2013, sequential decision context: https://arxiv.org/abs/1301.2609

The expected-reward calculation and Rowan Learning case are original hypothetical illustrations.

Previous: Bernoulli Next: Bernoulli mixture model