Must every action have positive probability?

No; some can have probability zero.

Can a stochastic policy be learned?

Yes. Policy-gradient methods often optimize parameterized action distributions.

Does choosing the most probable action preserve the policy?

No. It replaces the sampling rule with a deterministic one, subject to tie handling.

AI Glossary / Reinforcement learning and robotics / Intermediate

Stochastic policy

Definition

A stochastic policy specifies a probability distribution over actions for each relevant state or observation. The agent samples from that distribution when deciding what to do, so repeated encounters with the same input can lead to different actions. The probabilities may be fixed, learned, or constrained by the task; they do not have to be equal.

What it means

A deterministic policy selects one action for a given input. A stochastic policy can assign positive probability to several actions. Deterministic policies can also be viewed as the special case in which one action has probability 1.
Policy randomness is different from environmental randomness. A stochastic policy can act in a deterministic simulator, and a deterministic policy can operate in an environment with uncertain outcomes. Separating these sources of variation helps explain why repeated runs differ.

How it works

For a finite action set, the policy assigns nonnegative probabilities that sum to 1. In state s, the notation pi(a|s) gives the probability of choosing action a. Continuous-action policies instead specify a probability distribution such as a parameterized density, subject to the required action limits.
During training, randomness may support exploration or form part of the optimization objective. A policy can learn to place more probability on useful actions while retaining alternatives. At evaluation time, sampling from the policy and selecting its most probable action are different decision rules and can yield different results.
Randomization is not always necessary for optimality. Standard finite, fully observed discounted Markov decision processes admit a deterministic stationary optimal policy. Other settings, including restrictions on memory, partial observations, or additional objectives, can make randomization useful. The task assumptions determine the relevant conclusion.
An implementation must specify its inputs and sampling behavior. A policy conditioned on the full history has more information than one conditioned only on the current observation. Probability values alone do not reveal whether the policy has access to a clock, memory, or other state information.

Three practical examples

Exploration: sample alternative actions while learning their consequences in a controlled environment.
Continuous control research: represent actions with a parameterized distribution during policy optimization.
Partially observed tasks: investigate how randomized decisions compare with deterministic rules when the controller has limited information or memory.

Think of it this way

Think of a decision rule that uses a weighted draw instead of always choosing the same option. Learning changes the weights. The draw can be deliberate and mathematically specified, but its usefulness depends on what information the decision-maker has and what the task rewards.

Worked example

Consider a fictional two-decision task. Both decisions present exactly the same observation, and the controller is restricted to a stationary memoryless policy: it cannot use the decision number, previous action, or internal memory. The environment gives reward 1 only if the action sequence is Left followed by Right. Every other two-action sequence gives 0. The environment itself is deterministic.
Let the policy choose Left with probability p and Right with probability 1-p, independently on each decision conditional on the identical observation. The probability of the successful sequence is p times (1-p). Expected reward is therefore p(1-p).
If p = 0.20, success probability is 0.20 times 0.80 = 0.16. If p = 0.50, it is 0.25. If p = 0.80, it returns to 0.16. The expression can be written 0.25 - (p - 0.50)^2, so its maximum over p from 0 to 1 is 0.25 at p = 0.50.
A deterministic stationary memoryless policy must always choose Left or always choose Right for this observation. Those choices correspond to p = 1 or p = 0, and both have success probability 0. Randomization therefore improves the best achievable performance within this deliberately restricted policy class.
Now allow the policy to observe the decision number or retain one bit of memory. It can choose Left first and Right second deterministically, achieving reward 1 every episode. This does not contradict the earlier calculation because the available information and policy class changed.
Across 400 independent episodes with p = 0.50, expected successes are 400 times 0.25 = 100. The standard deviation of the success count is sqrt(400 times 0.25 times 0.75), approximately 8.66. Actual successes need not equal 100. The example isolates action randomness and information restrictions; it is not evidence that stochastic policies generally outperform deterministic policies in fully observed tasks.

Illustrative case study

Fictional case: MapleControl trains a stochastic simulator policy and reports its sampled evaluation results. An application later replaces sampling with the most probable action at every step. Performance changes, and the team discovers that it has deployed a different policy from the one evaluated.
The team records whether each test samples actions or uses a deterministic selection rule, and evaluates both explicitly. This invented case illustrates a reproducibility issue rather than a claim that either selection rule is universally better.

Common misconceptions and mistakes

Common mistakes include treating stochastic as uniform, assuming action probabilities are confidence scores about factual correctness, and conflating random transitions with random action selection. Another error is claiming randomization solves missing information. It can improve some restricted controllers, but it does not reveal hidden state or guarantee the performance of a controller with richer observations.

Business use

Businesses evaluating a learned policy should ask which decision rule will actually operate, what inputs it receives, and how repeated-run variability is measured. A policy's average return alone may hide undesirable outcome variation. Comparing sampled and deterministic deployment modes can be useful, but each comparison must use the same task conditions and relevant constraints.

Check your understanding

Why does p = 0.50 achieve expected reward 0.25?

Answer: The required Left-then-Right sequence has probability 0.50 times 0.50.

Why do the restricted deterministic policies fail?

Answer: They repeat the same action under the identical observation.

What allows a deterministic policy to succeed perfectly?

Answer: Access to decision number or suitable memory.

Related terms

Policy
Random policy
Exploration
Policy gradient
Entropy
deterministic policy
action distribution
partial observability
memoryless policy
expected return

Learn next

Next, compare sampled and most-probable-action execution for one policy. Then add a time indicator to a partially observed toy task and examine how the available policy class changes.

Policy

Sources

Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, second edition, policy definitions and policy-gradient discussion: https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf Tor Lattimore and Csaba Szepesvari, Bandit Algorithms, stochastic action-selection policies: https://tor-lattimore.com/downloads/book/book.pdf

Previous: Stochastic parrot Next: Stochastic process