Does the learner need absolute reward scores?

Not in the basic dueling setting. It learns from relative feedback.

Must a single best option exist?

No. It depends on the preference structure and the chosen definition of best; a Condorcet winner can be absent.

Are dueling bandits only for human judgments?

No. Any suitable noisy pairwise feedback can fit the framework, provided the observation assumptions and objective are appropriate.

AI Glossary / Causal inference and experimentation / Intermediate

Dueling bandit

Definition

A dueling bandit is a sequential learning problem in which the learner chooses pairs of options and receives relative feedback, such as which option was preferred. It uses these comparisons to learn useful choices while managing the cost of continued exploration.

What it means

Many experiences are easier to compare than to score numerically. A person may prefer one answer to another without being able to assign either a meaningful quality score out of one hundred.

The options are often called arms, borrowing language from multi-armed bandit research. A duel means a comparison, not a physical contest. The learner decides which pair to present next, observes the result and revises its knowledge.

Different formulations pursue different objectives. Some aim to identify a best option using few comparisons. Others minimize cumulative regret while comparisons continue. The meaning of best must be specified: pairwise preferences can form cycles, so a universally preferred option need not exist.

How it works

In a simple stationary stochastic model, each pair has an unknown probability of one option defeating the other. The learner collects noisy outcomes and estimates these probabilities or a more structured preference model.

An exploration strategy seeks informative comparisons. Exploitation favors options already believed to perform well under the chosen objective. Algorithms differ in how they handle uncertainty, choose challengers and stop exploring.

A Condorcet winner is an option that beats every other option with probability greater than one half in the strict version of the definition. Some methods assume such a winner exists. Others use different solution concepts. Contextual variants allow preferences to depend on features of the user or situation, rather than assuming a single preference matrix applies everywhere.

Three practical examples

An interface study adaptively chooses pairs of layouts for participants to compare.
An answer-evaluation system selects pairs of model responses to learn which alternatives users prefer.
A search experiment compares ranking strategies using a carefully defined relative-feedback procedure rather than an assumed absolute satisfaction score.

Think of it this way

Think of organizing a sequence of taste comparisons when judges find relative choices easier than numerical ratings. Testing every possible pair equally may waste effort on distinctions that are already clear.

The analogy stops at the feedback structure. Judges can change preferences, presentation order can influence choices and a taste winner is not necessarily healthier or cheaper. Those additional goals require their own measurements or constraints.

Worked example

Finding a winner and recognizing a cycle

Suppose the true pairwise probabilities for three options are known in a teaching example: A beats B with probability 0.65, A beats C with probability 0.60, and B beats C with probability 0.55. A is the strict Condorcet winner because both of its relevant probabilities exceed 0.5.

A real learner would not know these values. Seeing A win six of ten comparisons with B gives an empirical rate of 0.6, not certainty that the true probability exceeds one half. More evidence and an appropriate uncertainty procedure may be needed.

Now change the setup: A beats B with probability 0.6, B beats C with probability 0.6, and C beats A with probability 0.6. No option beats both others. Repeating comparisons more often can clarify the cycle but cannot create a Condorcet winner.

The team must then choose a suitable alternative objective instead of reporting that its search simply needs a little more data. These probabilities are invented to illustrate the distinction between uncertainty about preferences and the absence of a particular kind of winner.

Illustrative case study

Fictional case: comparing help answers

A fictional software company studies three styles of help answer. It presents controlled pairs and asks participants which better resolves the specified problem. A sequential policy chooses later comparisons using earlier results.

The team notices that beginners and experienced users favor different styles. It investigates whether pooling all users into one preference matrix hides a useful distinction. It also records ties and unclear judgments rather than silently converting every response into a forced win.

Before choosing a deployment policy, the company checks factual correctness separately. A preferred answer can still be wrong. The fictional study demonstrates how relative preference evidence supports a decision while leaving other product requirements to separate evaluation.

Common misconceptions and mistakes

A duel is not necessarily an ordinary A/B test with a measured reward for each arm. Its defining feedback is relative comparison, although practical systems can collect additional information.

One observed win does not establish a stable preference. Position bias, repeated judges and changing content can distort a naive estimate. The observation model should reflect the collection process.

There is also no universal dueling-bandit regret formula. Regret depends on the winner concept and how the two selected arms are evaluated. Comparing algorithm guarantees requires checking those definitions and assumptions, not merely matching the word regret.

Business use

Use this framework when pairwise judgments are credible and sequential selection can reduce unnecessary comparisons. Define the comparison question precisely, including whether a tie or abstention is allowed.

Keep the candidate set, presentation policy and participant population visible in the results. Decide in advance how the study will handle cycles or heterogeneous preferences. Evaluate whether adaptively collected evidence supports the intended conclusion; a changing comparison policy should not be analyzed as though every pair received an identical random sample.

Check your understanding

A beats B more often than not, B beats C more often than not, and C beats A more often than not. Is A a Condorcet winner?

Answer: No. A loses its comparison with C in probability. The three-way cycle contains no strict Condorcet winner.

Related terms

Preference learning
Regret
Multi-armed bandit
exploration and exploitation
Condorcet winner
contextual bandit
pairwise comparison

Learn next

Compare best-option identification with regret minimization. Then study a preference matrix containing a cycle and see how alternative winner definitions change the learning objective.

Regret

Sources

Primary sources: Yue, Broder, Kleinberg and Joachims, The K-armed dueling bandits problem: https://www.sciencedirect.com/science/article/pii/S0022000012000281

Dudik and colleagues, Contextual Dueling Bandits, on contextual feedback and alternatives to Condorcet assumptions: https://arxiv.org/abs/1502.06362

The probability examples and company case are original illustrations.

Previous: Duality gap Next: Dueling deep Q-network