Is UCT the only selection policy?

No. It is one influential approach among several.

Does selection execute a real action?

Usually it chooses an internal search branch; execution follows the completed planning decision.

Can a low-average action be selected?

Yes, when the exploration component makes further investigation attractive.

AI Glossary / Reinforcement learning and robotics / Advanced

Tree selection policy

Definition

A tree selection policy is a rule for choosing which branch to follow while traversing a search tree. In Monte Carlo tree search, it directs successive simulations through the existing tree, using accumulated evidence to decide which actions deserve more investigation.

What it means

Search has a limited computation budget. A promising branch may deserve further attention, but an overlooked branch may turn out to be better. The selection policy determines how the search distributes its effort between these possibilities.
The policy operates inside the planning process. Choosing a branch for another simulation does not necessarily mean choosing that action for real execution. A separate decision rule can select the final move after the search budget is exhausted. Keeping these two decisions distinct helps explain why a search sometimes investigates an action with a lower current average outcome.

How it works

Monte Carlo tree search repeatedly traverses its current tree, expands an eligible frontier, obtains an outcome estimate and updates statistics along the visited path. Selection uses those statistics during later traversals. Expansion and outcome estimation are related stages, but they are not the selection rule itself.
One common rule combines a mean return with an exploration bonus. In a UCT-style convention, the score for an already visited child is Q + c sqrt(ln N / n). Here Q is its average return, N is the parent visit count, n is the child visit count and c controls the bonus. This convention absorbs any additional fixed scaling factor into c.
For a fixed parent count, the bonus is larger for less-visited children. Unvisited actions need an explicit initialization or selection convention because division by zero is undefined. Other selection policies can use learned action priors or different exploration rules. The chosen formula must also respect the value perspective in competitive games: an opponent's turn cannot simply maximize the original player's reward as though both players cooperate.

Three practical examples

Board-game planning: a search investigates candidate moves repeatedly, using simulation outcomes to update the priority of their branches.
Scheduling research: a simulator explores sequences of task assignments. A selection rule allocates further simulated trials among partial schedules rather than exhaustively enumerating every continuation.
Robot planning experiments: a team compares possible action sequences in a model before selecting a command. Search results depend on the model, evaluation method and computation budget as well as the tree selection policy.

Think of it this way

Imagine testing several recipes before choosing one for an event. A consistently good recipe deserves attention, but a recipe tried only once might deserve another trial before you dismiss it. Deciding which recipe to test next is different from deciding which recipe to serve. The selection policy makes the first kind of decision inside a branching search.

Worked example

Consider an original search snapshot with one parent and two visited actions. The parent has 100 visits. Action A has 80 visits and accumulated return 56, giving mean Q_A = 56/80 = 0.70. Action B has 20 visits and accumulated return 12, giving Q_B = 12/20 = 0.60. Assume returns are on a consistent zero-to-one scale.
Use score Q + c sqrt(ln N / n), with natural logarithms and c = 0.5. Since ln 100 is approximately 4.60517, A receives bonus 0.5 sqrt(4.60517/80), approximately 0.11996. Its score is therefore approximately 0.81996.
B receives bonus 0.5 sqrt(4.60517/20), approximately 0.23993, giving score 0.83993. The selection rule chooses B for the next traversal even though B has the lower average return. Its smaller visit count produces enough exploration bonus to reverse the ranking.
Now reduce c to 0.1 while keeping the same snapshot. A's score becomes approximately 0.72399, while B's becomes approximately 0.64799. This version chooses A. The comparison isolates how the exploration coefficient affects one selection; it does not establish which coefficient performs best across a complete search.
If the selected B traversal produces return 1, B's accumulated return becomes 13 across 21 visits, so its new mean is 13/21, approximately 0.61905. The parent count becomes 101. Subsequent scores must use these updated statistics. A final-action rule based on visit count would still choose A at this point. The calculation demonstrates selection and updating, not a complete search run or a guarantee of the optimal action.

Illustrative case study

Fictional company Alder Planning compares search settings in a simplified scheduling simulator. Its analysts hold the simulation budget and test scenarios constant while changing the selection coefficient. They record final schedule quality, runtime and how visits are distributed among root actions.
One setting investigates unfamiliar branches more often. The team checks whether that extra exploration improves final decisions rather than treating exploration itself as the success metric. It also tests sensitivity to the simulator's assumptions. This fictional evaluation illustrates a comparison method; it reports no measured commercial improvement.

Common misconceptions and mistakes

The highest selection score is not necessarily the highest estimated reward. The bonus deliberately changes the ranking. Nor is that score automatically a calibrated probability or an ordinary confidence interval for the final decision.
A search policy is also different from the policy used to complete a rollout beyond the expanded tree. Confusing them makes experiments difficult to interpret. Another mistake is changing the reward scale without reconsidering the exploration coefficient: multiplying rewards changes the relative influence of the same numerical bonus. Finally, good finite-budget performance is not guaranteed merely because a rule has favorable asymptotic properties under particular assumptions.

Business use

Tree selection policies matter when a business evaluates branching alternatives through simulation. They influence how computation is spent and which possibilities receive attention before a decision deadline. Teams should compare final outcomes at realistic budgets, include model costs and monitor sensitivity to reward definitions. More simulations cannot correct an inaccurate simulator by themselves.

Check your understanding

Why did B win the first comparison?

Answer: Its larger exploration bonus outweighed its lower mean return.

Why did reducing c change the result?

Answer: It reduced the contribution of exploration relative to estimated reward.

Why update the parent count?

Answer: Future scores depend on the amount of search conducted at that parent.

Related terms

Learn next

Recalculate the example after another outcome, then compare the next simulation choice with a final-action rule based on visit counts.

Sources

Gelly and Silver, Monte-Carlo Tree Search and Rapid Action Value Estimation in Computer Go, 2011: https://www.cs.utexas.edu/~pstone/Courses/394Rspring13/resources/mcrave.pdf . AIMA glossary supplies the listed terminology: https://github.com/aimacode/aima-glossary/blob/master/glossary.md . The numerical example and company scenario are original illustrations.

Previous: Tree-of-thought prompting Next: Tree width