Is REINFORCE a policy-gradient method?

Yes, it is a basic Monte Carlo example.

Does every policy-gradient method use a critic?

No. Some use sampled returns directly.

Can the learned policy remain stochastic?

Yes. The parameterization and objective determine whether randomization is retained or reduced.

AI Glossary / Reinforcement learning and robotics / Advanced

Policy gradient

Definition

A policy gradient is the derivative of a policy's expected performance with respect to the parameters that control its action choices. Policy-gradient methods use this information, or an estimate of it, to adjust the policy toward higher expected return. They optimize a parameterized decision rule directly rather than requiring every decision to be selected by maximizing a learned action-value table.

What it means

A small change in a policy parameter may make some actions more likely and others less likely. The policy gradient indicates how that change affects the objective locally. With many parameters, it is a vector of sensitivities rather than a single number.
The gradient concerns expected performance, not a guarantee about the next episode. A sampled action may receive a good outcome by chance, and a useful learning method must separate that variability from a persistent improvement signal. The policy's representation also determines which behaviors the optimizer can reach.

How it works

Start with a differentiable policy πθ and an objective J(θ), such as expected episodic return. A gradient-ascent update has the form θ_new = θ + αg, where α is a step size and g estimates the gradient of J. Software that minimizes losses can instead minimize a suitable negative objective.
For stochastic policies, a common estimator combines the gradient of log action probability with a return or advantage estimate. This connects sampled outcomes to the parameters that made the actions likely. In sequential problems, the sampling distribution, discounting, and treatment of future rewards must match the chosen objective.
An action-independent baseline can be subtracted from the return without changing the expected score-function gradient under the usual assumptions. It can reduce variance, although not every baseline reduces variance in every situation. A critic supplies value estimates that can make learning more efficient while introducing its own approximation considerations.
The policy-gradient theorem provides a way to express the performance gradient without explicitly differentiating the environment's state-distribution dynamics. It does not mean that state visitation is irrelevant. Nor does gradient ascent generally guarantee a globally optimal policy or improvement after every finite noisy update.

Three practical examples

Continuous control: adjust a robot controller's action distribution using simulated task returns.
Game agents: change move probabilities based on estimated contributions to longer-term outcomes.
Sequential resource decisions: learn a parameterized allocation rule when the objective depends on consequences across multiple decisions.

Think of it this way

Think of adjusting several knobs on a decision-making machine. The gradient estimates which small knob changes would improve the average result around the current setting. It does not provide a map of every possible setting, and a noisy estimate can temporarily point in an unhelpful direction.

Worked example

Consider a one-step fictional task with two actions. A always gives reward 5, and B always gives reward 1. Let p = πθ(A) = 1/(1 + exp(−θ)), so πθ(B) = 1 − p. The expected reward is J(θ) = 5p + 1(1 − p) = 1 + 4p.
Because dp/dθ = p(1 − p), the exact gradient is dJ/dθ = 4p(1 − p). At θ = 0, p = 0.50 and the gradient equals 1. An exact-gradient ascent step with α = 0.20 moves θ to 0.20. The new probability of A is approximately 0.549834 and the expected reward becomes approximately 3.199336, compared with 3 before the update.
Now examine a sampled score-function estimate at the original θ = 0. The log-probability derivative is 1 − p = 0.50 if A is selected, and −p = −0.50 if B is selected. Multiplying by reward gives gradient estimate 2.5 for A and −0.5 for B. Their expectation is 0.50 × 2.5 + 0.50 × (−0.5) = 1, matching the exact gradient.
Subtract the action-independent baseline 3. The A estimate becomes (5 − 3) × 0.50 = 1, and the B estimate becomes (1 − 3) × (−0.50) = 1. This special example has zero estimator variance after that subtraction because rewards are deterministic and the numbers align. General tasks do not inherit this zero-variance property.
Notice that B's positive reward initially produced a negative parameter update. The sign depends on both the outcome and how the parameter changes that action's probability.

Illustrative case study

Fictional case: LindenControl trains a simulated controller using return-weighted policy updates. The team sees widely varying learning curves across random seeds and introduces a learned value baseline to compare outcomes with expectations for each state.
It evaluates whether the change improves learning under the same interaction budget and checks the critic's errors alongside policy performance. This fictional example illustrates a variance-management question, not a claim that adding a critic always helps. The team keeps a separate evaluation set of runs rather than treating lower training loss as proof of a better controller.

Common misconceptions and mistakes

Policy gradient does not mean differentiating a raw reward through every physical action. Score-function methods can estimate gradients without a differentiable simulator. A positive reward does not always mean every relevant parameter increases. A baseline is not a replacement reward objective. Very large steps can also harm performance even when the local gradient estimate is useful.

Business use

This family is relevant when teams need a flexible action policy, especially with continuous or large action spaces. Assess interaction cost, sensitivity to step sizes, variation across runs, and performance under the actual deployment action rule. Define the return carefully before optimizing it. Faster optimization of an incomplete objective can reinforce an undesirable decision pattern rather than improve the intended business outcome.

Check your understanding

What does the gradient equal at θ = 0 in the example?

Answer: One.

Why does the baseline preserve its expectation?

Answer: It is independent of the sampled action, and its expected score contribution cancels.

Does the exact-gradient improvement prove every sampled update helps?

Answer: No. Sample estimates can vary and finite steps need not improve performance.

Related terms

Policy
REINFORCE
Baseline
Score function estimator
Stochastic policy
gradient ascent
actor-critic
advantage
expected return
policy optimization

Learn next

Next, derive a log-probability gradient for a small softmax policy. Then compare Monte Carlo returns with critic-based advantage estimates and examine the effect of step size on actual performance.

Sources

Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, second edition, Sections 13.1–13.4: https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf Richard S. Sutton and colleagues, Policy Gradient Methods for Reinforcement Learning with Function Approximation, NIPS 1999, primary abstract: https://papers.nips.cc/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html

Independent review checked policy-gradient and action-independent-baseline derivations.

Previous: Policy evaluation Next: Policy-gradient baseline