What happens when all values are equal?

Every available action receives the same probability under the basic formula.

Does increasing temperature reverse the ranking?

No. For positive temperature, higher values still have higher probabilities; the distribution becomes flatter.

Is it the same as Thompson sampling?

No. Thompson sampling draws from an uncertainty model and chooses according to that draw. Basic Boltzmann exploration directly transforms current value estimates into probabilities.

AI Glossary / Probabilistic machine learning / Intermediate

Boltzmann exploration

Definition

Boltzmann exploration is an action-selection strategy that samples actions with probabilities determined by a softmax of their estimated values. Better-valued actions receive more probability, while other actions can still be tried. A positive temperature parameter controls how strongly the selection favors the currently highest estimated values.

What it means

An agent often faces a choice between exploiting what currently looks best and exploring alternatives that might prove better. Always choosing the highest estimate can overlook an action whose estimate is poor because it has rarely been tried.
Boltzmann exploration converts value estimates into a distribution over available actions. It differs from a simple random exploration step because the alternatives do not generally receive equal probabilities.
The policy responds to estimated values, not automatically to how uncertain those estimates are. Two actions with equal estimated value receive equal probability under the basic rule even if one estimate comes from far more observations.

How it works

For a finite action set, assign probability P(a) = exp(Q(a)/tau) divided by the sum of exp(Q(b)/tau) over available actions b. Here Q is the estimated value and tau is a positive temperature.
Sample an action from this distribution, observe the result, and update the value estimates using the chosen learning method. Boltzmann exploration specifies the action-selection rule; it is not itself the entire value-learning algorithm.
For fixed finite values, a high temperature makes probabilities more similar. As temperature approaches zero, probability concentrates on maximizing actions; tied maxima share that limiting mass under the standard formula.
Temperature must be interpreted relative to the value scale. Numerical implementations commonly subtract the largest scaled value before exponentiating, which preserves the probabilities while helping avoid overflow. Changing temperature through time requires care rather than an assumption that cooling always produces good exploration.

Three practical examples

A simulated routing agent tries several permitted routes while favoring those with higher estimated reward.
A game-playing agent samples among candidate moves using learned action values instead of always taking the current maximum.
A bandit experiment allocates trials among approved alternatives using value-weighted random selection, while recording which alternative was selected and its outcome.

Think of it this way

Think of drawing a ticket from a box where promising options have more tickets. Less promising options remain possible, but the proportions can be made more or less concentrated.
The analogy has limits: ticket counts reflect current value estimates, which can be wrong. They do not inherently represent confidence in those estimates, and putting more tickets on an option does not make its real outcome better.

Worked example

Consider an original three-action example with values Q(A) = 0, Q(B) = log(2), and Q(C) = log(4), using natural logarithms. At temperature tau = 1, the exponential weights are 1, 2, and 4.
Their sum is 7, so the selection probabilities are 1/7, 2/7, and 4/7: approximately 14.29%, 28.57%, and 57.14%. The highest-valued action C is favored but is not certain.
At tau = 0.5, the weights become 1, 4, and 16. The probabilities are 1/21, 4/21, and 16/21, approximately 4.76%, 19.05%, and 76.19%. Lowering temperature concentrates more probability on C.
Adding 10 to every Q value leaves the probabilities unchanged because the common exponential multiplier cancels. Multiplying all Q values by two while keeping tau fixed has the same effect as halving tau. These are selection probabilities, not probabilities that an action is optimal or successful.

Illustrative case study

In a fictional project, Juniper Robotics compares exploration policies in a simulator. After changing the reward units, the team finds that the same temperature produces much more concentrated action choices.
The engineers inspect the value-to-temperature ratio and repeat the comparison using consistently scaled rewards. They also examine whether rarely selected actions have enough evidence behind their low estimates.
This invented case illustrates a tuning issue. It does not report that Boltzmann exploration outperformed another policy or that a simulation result transferred to physical equipment.

Common misconceptions and mistakes

Do not confuse temperature with the learning-rate parameter used to update Q values. Some literature uses an inverse-temperature multiplier and calls it a learning rate, so read the formula before interpreting its direction.
Do not treat a 70% action-selection probability as 70% confidence that the action is best.
Do not assume that all non-greedy actions are selected equally, as they may be under a simple uniform exploration component.
Do not set temperature to zero directly in the division formula; a greedy limit requires separate handling.
Finally, basic Boltzmann selection does not automatically solve exploration difficulty or provide optimal regret guarantees.

Business use

For applied experimentation, the strategy offers a simple way to favor promising options while continuing to sample others. Its behavior is understandable when the value scale, temperature, and candidate actions are recorded.
Assess cumulative reward, coverage of alternatives, and sensitivity to early estimation errors. A policy that quickly concentrates on one option can look efficient while learning too little about its competitors. If uncertainty is central to the decision, compare methods that explicitly account for uncertainty rather than assuming a softmax over mean estimates already does so.

Check your understanding

Values are multiplied by five after a change of reward units. What temperature change preserves the same Boltzmann probabilities?

Answer: Multiply temperature by five as well. Each ratio Q(a)/tau then remains unchanged, so the normalized exponential weights remain unchanged.

Related terms

Softmax
Temperature
Epsilon greedy policy
Thompson sampling
exploration–exploitation tradeoff
action value
multi-armed bandit

Learn next

calculate softmax probabilities for two nearly tied actions and one distant alternative. Then compare how Boltzmann and uncertainty-aware policies respond when the same estimated values are supported by very different numbers of observations.

Softmax

Sources

Sources: Poole and Mackworth, Exploration and Exploitation, author-hosted textbook section: https://www.cs.ubc.ca/~poole/aibook/2e/html2e/ArtInt2e.Ch12.S5.html Cesa-Bianchi and colleagues, Boltzmann Exploration Done Right, original paper: https://papers.nips.cc/paper/2017/file/b299ad862b6f12cb57679f0538eca514-Paper.pdf The calculations and fictional case are original teaching examples.

Previous: Boid Next: Boltzmann Machine Network