Does it estimate the reward of every action?

Not directly; it learns preferences governing a policy.

Why subtract a baseline?

A suitable baseline can reduce gradient variability without changing its expected direction under the stated independence condition.

Does reward zero always mean no update?

No. The update depends on reward minus baseline.

AI Glossary / Reinforcement learning and robotics / Intermediate

Gradient bandit algorithm

Definition

A gradient bandit algorithm learns action preferences and converts them into action-selection probabilities, typically with a softmax function. After observing a reward, it updates preferences using a stochastic gradient of expected reward. It learns how strongly to favor each action rather than directly estimating each action's expected reward as an action-value method does.

What it means

A bandit problem involves repeated choices with uncertain immediate rewards. There is no modeled chain of future states in the basic noncontextual setting. The learner must balance trying alternatives with exploiting choices that appear promising.

Gradient bandits maintain relative preferences. A larger preference makes an action more likely, but its numerical value is not an estimated reward, profit or success probability. Reward feedback changes the probability distribution over choices. A baseline, often based on previous rewards, helps distinguish a better-than-usual outcome from one that merely has a positive numerical value.

How it works

Assign each action a real preference H(a). Compute its probability as exp[H(a)] divided by the sum of exponentials across all actions. Equal preferences give equal probabilities, and adding the same constant to every preference leaves the probabilities unchanged.

Sample an action A from this distribution, observe reward R, and compare it with baseline b. With learning rate alpha, update the selected preference by alpha times (R minus b) times [1 minus pi(A)]. Update every other preference by minus alpha times (R minus b) times its current probability.

Thus an above-baseline reward raises the selected action's preference relative to the others; a below-baseline reward reverses the direction. Use probabilities from before the update throughout the calculation.

For the standard unbiased-gradient argument, the baseline is independent of the current sampled action conditional on the past. A running reward average can be maintained with an explicitly specified update order. Numerical softmax implementations commonly subtract the largest preference before exponentiation to avoid overflow without changing the probabilities.

Three practical examples

1. A simulated interface experiment learns probabilities for choosing among several fixed layouts using immediate task-completion rewards.

2. An engineering study allocates repeated trials among alternative operating settings when each trial yields a comparable measured score.

3. A reinforcement-learning course compares preference learning with epsilon-greedy action-value learning on the same stationary bandit environment.

Think of it this way

Think of adjusting the sizes of slices on a choice wheel. A favorable result enlarges the selected slice and shrinks the others, increasing its chance of selection next time. The hidden preference numbers control slice sizes, but are not labels saying how much reward each slice is worth.

Worked example

Suppose three actions A, B and C begin with preferences zero. Each therefore has probability one third. Use learning rate 0.1 and a baseline of two, computed from earlier observations before the current action is sampled.

Select B and observe reward five. The reward advantage is five minus two = three. B's preference increases by 0.1 times three times two thirds = 0.2. Each other preference decreases by 0.1 times three times one third = 0.1.

The new preferences are minus 0.1, 0.2 and minus 0.1. Softmax gives B probability exp(0.2)/[exp(0.2) + 2 exp(minus 0.1)], approximately 0.4030. A and C each receive approximately 0.2985.

The preference increments sum to zero: minus 0.1 + 0.2 minus 0.1 = zero. Adding 100 to all three new preferences would leave the same action probabilities, confirming that only relative preference matters.

If the reward had instead equaled the baseline, the advantage would be zero and all preferences would remain unchanged. A reward of one would reduce B's relative preference despite being numerically positive. These are single-step updates, not evidence that B has the greatest true expected reward.

Illustrative case study

Fictional case: Birch Labs compares three presentation formats in a controlled simulator. Each simulated session yields a completion score, and a gradient bandit adjusts how often each format is selected.

The team logs the selection probabilities and monitors whether unusually noisy outcomes drive premature concentration on one format. It compares results across repeated runs and checks the reward definition against the intended experience. When session characteristics matter, analysts recognize that the noncontextual bandit ignores relevant differences and investigate a contextual model rather than interpreting one global preference as universally best.

Common misconceptions and mistakes

A preference is not an action-value estimate. Interpreting H(a) as expected reward confuses two different representations.

The algorithm updates unselected preferences as well as the selected one. Omitting those terms changes the gradient. The baseline is also not a target reward that every action must achieve; it is a reference used in the learning signal.

Softmax assigns positive probabilities for finite preferences, but this does not guarantee adequate practical exploration: probabilities can become extremely small. A single favorable outcome can be noise. Large learning rates, poor scaling and changing reward distributions can produce unstable or misleading behavior.

Business use

Business relevance lies in repeated low-latency choice experiments where immediate outcomes are meaningful and the bandit assumptions are appropriate. Teams should define the reward carefully, record action probabilities and assess cumulative performance as well as the final preferred action. If choices affect future states or delayed outcomes, a simple bandit may omit important consequences and require a sequential decision model.

Check your understanding

Why does B reach only about 40 percent probability after its favorable outcome?

Answer: The finite preference update changes a softmax distribution; it does not force a deterministic choice.

What happens if all preferences increase equally?

Answer: Nothing happens to the action probabilities.

Related terms

Policy gradient
multi-armed bandit
softmax policy
reward baseline
action preference
exploration and action-value estimation

Learn next

Next, derive the gradient of log softmax and recover the selected and unselected update terms. Then compare learning curves across reward scales, baseline choices and learning rates using repeated simulated runs.

Sources

Primary source: Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, author textbook draft, section Gradient Bandit Algorithms: https://web.stanford.edu/class/psych209/Readings/SuttonBartoRL.pdf

The three-action calculation and fictional case are original teaching material. The worked example uses a baseline fixed from past data before the current action.

Previous: Gradient accumulation Next: Gradient-based method