Is it the same as A/B testing?

No. A bandit usually adapts allocation while learning; an experiment may primarily seek a reliable comparison under a chosen allocation design.

Does every bandit need a neural network?

No. Simple estimates can suffice.

Yes, but a method designed for stable rewards may need modification when they do.

AI Glossary / Reinforcement learning and robotics / Intermediate

Multiarmed bandit

Definition

A multiarmed bandit is a sequential decision problem in which a learner repeatedly chooses among alternatives, called arms, and observes the reward of the chosen alternative. The learner must use incomplete feedback to decide what to try next. A central challenge is balancing exploration of uncertain options with exploitation of options that currently look promising.

What it means

Imagine choosing which of several useful actions to offer each time an opportunity arrives. You learn something about the option you select, but usually do not see what would have happened under the other choices. An early success can therefore be misleading, while refusing to try alternatives can leave a better option undiscovered.
The term describes a problem family, not one algorithm. Different methods make different assumptions about rewards, changes over time, and available information. A practical design must specify the actual outcome being optimized and when that outcome becomes observable.

How it works

A basic interaction has three steps: choose an arm using the history available so far, observe its reward, and update the information used for subsequent choices. The policy determines how choices depend on that history.
Exploration can be random or directed toward options with greater uncertainty. Exploitation favors an option estimated to perform well. In an epsilon-greedy policy, a specified fraction of decisions explore randomly and the others choose an estimated best option.
Stochastic formulations model rewards as random variables, often with a stable distribution for each arm. Other formulations permit changing or adversarial rewards. Contextual bandits additionally use information about the current opportunity before choosing.
Cumulative regret compares performance over time with a defined benchmark. Identifying a good final option is a different objective from earning as much reward as possible during learning. Neither objective can be assessed properly without specifying the comparison and assumptions.

Three practical examples

Interface design: choose among approved help prompts and observe whether the displayed prompt resolves a user's request.
Computing: choose among interchangeable processing configurations and observe a reward that accounts for successful completion and resource cost.
Education: choose among suitable practice formats and measure an agreed immediate learning outcome, while recognizing that longer educational effects may require a richer model.

Think of it this way

Think of trying several unfamiliar lunch counters while working in a new neighborhood. Returning to today's favorite is sensible, but trying another counter may reveal a better choice. One unusually good meal does not prove that counter is consistently best. The analogy leaves out important complications such as changing preferences and different meal prices.

Worked example

Consider three fictional arms, A, B, and C. Their current estimated rewards are 0.60, 0.50, and 0.40. These estimates make A the unique greedy choice. Set epsilon to 0.15 and, during exploration, choose uniformly among all three arms, including A.
A is therefore selected with probability 0.85 + 0.15/3 = 0.90. B and C each receive probability 0.05. The probabilities sum to one. It would be incorrect to say that A is selected only 85% of the time: exploration can also select A.
For a separate evaluation calculation, suppose the actual reward means are A = 0.40, B = 0.70, and C = 0.50. The learner does not know these values; they are supplied here to assess the decision rule. Expected reward for the next choice is 0.90 × 0.40 + 0.05 × 0.70 + 0.05 × 0.50 = 0.42.
Always choosing the truly best arm B would yield expected reward 0.70. The next-choice expected regret is therefore 0.28. Across 100 independent opportunities with these same fixed selection probabilities, expected regret would be 28 reward units. A learning policy would normally update its estimates, so that last calculation is not a forecast for an updating algorithm.

Illustrative case study

Fictional case: CedarHelp evaluates three approved versions of an in-product help prompt. The team initially counts clicks as reward, then discovers that one version encourages clicks without resolving the underlying issue. It changes the measured outcome to confirmed resolution and includes an appropriate observation window.
The team records the prompt shown, selection probability, outcome, and relevant timing. It also checks whether the mix of incoming requests changes during the evaluation. This fictional example illustrates why improving an algorithm's exploration rule cannot repair an unsuitable reward definition. It makes no claim about any real organization's results.

Common misconceptions and mistakes

A bandit is not simply an ordinary classifier trained on complete labels for every option. Feedback is selective. Random exploration is not necessarily a mistake, and the highest observed average is not necessarily the highest true mean. Basic bandit models also do not automatically represent actions that alter future system states. Treating a sequence of consequential robot movements as unrelated immediate choices can discard essential planning information.

Business use

Use this framework when repeated choices provide useful feedback and the selected outcome reasonably represents the decision's objective. Specify costs, eligible actions, delays, and constraints before selecting an algorithm. Compare with a suitable baseline and retain decision records that support evaluation under adaptive sampling. A theoretical regret guarantee applies under its stated assumptions; it is not a promised business improvement.

Check your understanding

Why does A receive 90% rather than 85% probability in the example?

Answer: Uniform exploration sometimes chooses A too.

Why is expected regret positive even though A looks best?

Answer: Current estimates differ from the supplied true means.

What feedback is missing after choosing B?

Answer: The outcomes that the same opportunity would have produced under the unchosen arms.

Related terms

Learn next

Next, compare cumulative regret with best-arm identification. Then study how uncertainty-based exploration differs from fixed random exploration and why delayed outcomes complicate evaluation.

Regret

Sources

Aalto Dictionary of Machine Learning, multiarmed bandit and stochastic multiarmed bandit entries: https://aaltodictionaryofml.github.io/ADictML_Main.pdf Tor Lattimore and Csaba Szepesvari, Bandit Algorithms, foundations and exploration methods: https://tor-lattimore.com/downloads/book/book.pdf

Independent review checked bandit feedback, regret and epsilon-greedy allocation.

Previous: Multiagent systems Next: Multiattribute utility theory