Does the theorem require a differentiable environment?

The standard likelihood-ratio formulation differentiates policy probabilities; it need not differentiate environment transitions.

Is this the same as REINFORCE?

REINFORCE is a sample-based learning method using return estimates; the theorem is an underlying gradient identity.

Can a baseline depend on the state?

Yes, an action-independent state baseline can preserve the expected gradient under the standard conditions. Arbitrary action-dependent subtraction needs additional care.

AI Glossary / Probabilistic machine learning / Intermediate

Policy gradient theorem

Definition

The policy gradient theorem expresses how an agent's expected cumulative reward changes when the parameters of its action-selection policy change. For a differentiable stochastic policy, it combines the gradient of the log action probability with the action's expected future return, averaged using the appropriate policy-dependent visitation weights.

What it means

A reinforcement-learning policy determines which actions an agent takes. Changing it affects both immediate choices and the situations the agent encounters later. This makes differentiating total performance appear complicated: the distribution of future states changes too.
The theorem provides a useful expression that accounts for those consequences without explicitly differentiating the state-visitation distribution. It connects observed decisions and their value to a direction for adjusting policy parameters. It does not say that future states are unaffected by the policy, or that every sampled update improves performance.

How it works

For a finite-horizon, undiscounted task, write J(theta) as expected total reward. Under suitable differentiability and regularity conditions, its gradient is the expectation of the sum over times t of grad log pi_theta(A_t|S_t) multiplied by Q_t(S_t,A_t). Here Q_t is expected reward from that decision onward under the policy. Time may be included in the state.
The log-probability gradient describes how changing parameters changes the selected action's probability. Q supplies its future value. Returns from sampled trajectories can estimate Q; a learned critic offers another approach but can introduce approximation bias.
Discounted and average-reward versions require their corresponding visitation weights and objective conventions. A discounted occupancy measure need not integrate to one; silently normalizing it can change a scale factor. A state-dependent, action-independent baseline can be subtracted without changing the expected score-function gradient under the usual conditions.

Three practical examples

Simulated scheduling: estimate how a stochastic job-selection policy affects total simulated completion rewards rather than judging each selection only by its immediate outcome.
Game agents: use complete episode returns to guide action probabilities, while recognizing that a fortunate episode can produce a noisy estimate.
Control research: connect a learned action-value critic to a stochastic controller, then investigate whether critic errors distort the actor's update.

Think of it this way

Think of adjusting a recipe-selection dial after observing the results of a sequence of meals. The adjustment combines how the dial affected a chosen recipe with how that choice contributed to the overall outcome. This analogy stops short of a proof: reinforcement learning has explicit probabilities, parameter derivatives, and temporal credit assignment. A memorable success alone is not an unbiased gradient estimate.

Worked example

Consider an original one-decision task. Action A earns 3 points and action B earns 1. Let p = sigmoid(theta) be the probability of A, so expected reward is J = 3p + (1 - p) = 1 + 2p.
At theta = 0, p = 0.5 and dp/dtheta = p(1 - p) = 0.25. Direct differentiation therefore gives dJ/dtheta = 0.5.
The log-probability derivative is 1 - p = 0.5 for A and -p = -0.5 for B. Applying the policy-gradient expression gives 0.5 × 0.5 × 3 + 0.5 × (-0.5) × 1 = 0.75 - 0.25 = 0.5, agreeing with the direct calculation.
Subtracting a baseline of 2 gives 0.5 × 0.5 × (3 - 2) + 0.5 × (-0.5) × (1 - 2) = 0.5 again. In this special example either sampled baseline-adjusted contribution equals 0.5; such zero variance is not a general property of baselines.

Illustrative case study

Fictional case: a research team at Harbour Queue builds a small scheduling simulator. Its initial report celebrates one policy update because the next simulated day has a higher reward. A reviewer asks whether this is an expected improvement or merely a different random workload.
The team first checks its implementation on the two-action calculation above, where the exact derivative is available. It then compares repeated estimates on fixed simulator settings, records their variability, and separates critic training from evaluation. These checks help identify inconsistent probability calculations and noisy conclusions. This is an invented testing scenario, not evidence of operational savings or convergence.

Common misconceptions and mistakes

The theorem is not an assurance of a global optimum. It describes a gradient; optimization still faces noisy estimates, imperfect representations, and step-size choices. Nor is a high reward automatically a good advantage: the relevant comparison may include a baseline.
The absence of an explicit state-distribution derivative does not make off-policy samples interchangeable with on-policy samples. Appropriate corrections or a different objective may be necessary. Finally, substituting an arbitrary critic for the true action value does not automatically preserve the exact gradient.

Business use

For a business evaluating reinforcement learning, the theorem helps frame technical questions about an agent's learning signal. Ask what reward objective is being differentiated, which policy generated the data, how future returns are estimated, and whether the implementation's weighting matches its claimed objective.
A useful demonstration includes a small environment with a known gradient before a complex simulation. Report variability and evaluation results separately from theoretical identities; neither a correct formula nor a convincing single episode establishes deployment performance.

Check your understanding

In the worked example, is the contribution from a sampled B action without a baseline equal to the exact gradient?

Answer: No. It is -0.5 × 1 = -0.5. The exact gradient is the probability-weighted average across possible actions, which equals 0.5.

Related terms

Policy
Reinforcement learning
Score function estimator
REINFORCE
Advantage function
Baseline
action-value function
actor-critic
discounted occupancy measure

Learn next

Differentiate a two-action expected reward directly, then reproduce it with log-probability derivatives. Add a baseline and compare sample variability. Next study trajectory returns and the assumptions needed when a critic or off-policy data replaces the exact on-policy expectation.

Sources

Sutton, McAllester, Singh and Mansour, Policy Gradient Methods for Reinforcement Learning with Function Approximation, original abstract: https://papers.nips.cc/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html Silver and colleagues, Deterministic Policy Gradient Algorithms, section 2.2 reviews the stochastic theorem and section 2.3 critic approximation: https://proceedings.mlr.press/v32/silver14.pdf Silver and colleagues Section 2.2 and its discounted-occupancy convention were independently consulted.

Previous: Policy-gradient baseline Next: Policy improvement