Is Thompson sampling uniformly random exploration?

No. Its action choices depend on the posterior and which actions appear best under sampled parameters.

Must every action have a Beta distribution?

No. Beta distributions are convenient for a particular binary-reward model. Other reward models use other posteriors.

Does it always choose the true best action?

No. It learns under uncertainty and may select inferior actions while gathering evidence. Performance depends on the problem and modeling assumptions.

AI Glossary / Generative AI and language / Intermediate

Thompson sampling

Definition

Thompson sampling is a method for making repeated decisions when the rewards of available actions are uncertain. It samples a plausible model of those rewards from a Bayesian posterior, chooses the action that is best under that sampled model, observes the outcome, and updates its beliefs. This naturally combines using promising actions with exploring actions whose value is still uncertain.

What it means

Suppose a service can show several versions of a help panel, but does not know which most often resolves a user's question. Always selecting the current average winner can overlook an alternative that has received little testing. Choosing randomly without using evidence wastes what has already been learned.

Thompson sampling lets uncertainty influence selection. An option can be chosen because a plausible sampled value makes it look best, even if its current average is lower.

Despite its placement in a language glossary, the method is much broader than text generation. It selects actions in sequential decision problems; it is not another name for temperature sampling of a language model's next token.

How it works

Define the available actions, the observed reward, and a probability model for uncertain reward parameters. Start with a prior and update it using collected observations.

At each decision, sample parameters from the current posterior distribution. Compute the expected reward of each action under those sampled parameters and choose the best permitted action. Observe the actual result and use it to update the posterior before subsequent decisions.

In a simple independent binary-reward model, each action has a Beta distribution for its unknown success probability. An observed success increments one parameter, and a failure increments the other. This convenient update is specific to that model; richer problems may require different distributions and approximate posterior calculations.

Three practical examples

1. A help-center experiment chooses among approved page layouts using a clearly defined successful-resolution signal, updating its estimates as outcomes arrive.

2. A recommendation prototype balances established items with less-tested candidates, provided its reward model and permitted action set match the intended application.

3. An AI workflow experiment allocates requests among several approved prompt variants and learns from reviewed task outcomes rather than simply selecting the variant with the best early average.

Think of it this way

Think of drawing one plausible performance estimate for each competitor, then letting the winner of that imagined contest take the next turn. Competitors with uncertain potential can sometimes win the draw and receive another opportunity to provide evidence.

The analogy is limited because the draws must come from a specified statistical model, not optimistic guesses. The next observed result is also distinct from the sampled performance estimate. A promising action can still fail on an individual attempt.

Worked example

Consider two actions with independent Beta(1,1) priors and binary success observations. Action A has produced eight successes and two failures, giving posterior Beta(9,3), whose mean is 9/12 = 0.75. Action B has produced two successes and one failure, giving Beta(3,2), whose mean is 3/5 = 0.60.

For one illustrative decision, suppose posterior draws are 0.68 for A and 0.77 for B. Thompson sampling selects B, even though B's posterior mean is lower. If B succeeds, its posterior becomes Beta(4,2), with mean 4/6, approximately 0.667. A's posterior is unchanged in this independent model.

The sampled 0.77 is a plausible success probability, not the observed reward and not B's permanent score. New draws are made at later decisions. The numbers illustrate the update mechanics; they do not show that B is truly better.

Illustrative case study

Fictional case: ClearPath Guides tests three versions of an onboarding explanation. Every version passes a content review before entering the experiment. The team defines success as completing a specified setup step, rather than merely clicking the explanation.

A Thompson-sampling prototype adjusts which version is shown as completed outcomes arrive. The team logs selections and delays, and separately checks whether particular user groups encounter unresolved difficulties.

This fictional design illustrates reward definition and monitoring. It does not guarantee that adaptive allocation identifies the best explanation under changing user behavior, and it does not equate more clicks with a better onboarding experience.

Common misconceptions and mistakes

Do not select the largest posterior mean and call that Thompson sampling. The defining step samples uncertain parameters before choosing an action.

Do not confuse a parameter draw with a simulated observed success or failure. The action is chosen using a sampled reward model, after which a real outcome is observed.

Avoid assuming the basic independent, stationary binary model fits every application. Delayed feedback, changing audiences, correlated actions, and misleading reward signals can require a different design. The algorithm optimizes the reward it is given, which may be narrower than the business outcome people actually want.

Business use

Thompson sampling can support adaptive experiments when learning and earning useful outcomes must happen together. Its value depends on meaningful rewards, appropriate priors, reliable observation, and enough opportunities to learn.

Keep a clear record of which actions were available, which were chosen, and when rewards arrived. If the objective is formal comparison or causal estimation, plan the analysis for adaptive allocation instead of automatically applying formulas intended for fixed randomization. Operational constraints should be part of the decision design.

Check your understanding

Action A has the highest posterior mean, but a posterior draw makes B look best. Which action does ordinary Thompson sampling choose for that round?

Answer: B, assuming it is permitted. Selecting by the sampled model is the mechanism that allows uncertainty to influence exploration.

Related terms

Bayesian inference
Prior (regularizer)
Posterior
Reward
Online learning
Multi-armed bandit
exploration and exploitation
prior
Beta distribution
contextual bandit

Learn next

Learn the multi-armed bandit problem next to clarify the difference between cumulative reward and identifying a winner. Then study Bayesian updating and contextual bandits, where information about the current situation can affect predicted rewards.

Reward

Sources

Russo and colleagues, A Tutorial on Thompson Sampling: https://arxiv.org/abs/1707.02038 and https://arxiv.org/pdf/1707.02038

The primary tutorial's Beta-Bernoulli model, posterior update, and sampling algorithm were read. The worked counts, practical suggestions, and ClearPath Guides case are original teaching illustrations, not reported experimental results.

Previous: Third normal form (3NF) Next: Thought vector