Is a value network required?

No. A learned baseline can help, but the basic estimator can use rewards or returns directly.

Is REINFORCE the same as supervised learning on the best action?

No. It estimates a reward gradient from sampled actions and their probabilities rather than requiring a correct action label.

Does adding a baseline change the objective?

An action-independent baseline in the standard score-function term leaves its expectation unchanged, under the stated sampling assumptions.

AI Glossary / Probabilistic machine learning / Advanced

REINFORCE

Definition

REINFORCE is a family of stochastic gradient methods that adjusts a policy using sampled rewards or returns multiplied by the gradient of the log probability of sampled actions. It provides a way to improve expected performance without differentiating through the environment's response to those actions.

What it means

A stochastic policy assigns probabilities to possible actions. REINFORCE uses the outcomes of actions actually sampled from that policy to estimate how changing its parameters would change expected reward. In episodic applications, the signal can include rewards received later in the episode.
The central distinction is between a random gradient estimate and the exact expected gradient. Individual updates may point in an unhelpful direction, especially when returns are noisy. Averaging over suitable samples recovers the intended gradient under the relevant assumptions; it does not make every sampled action or training update beneficial.

How it works

For a one-step objective J(theta) = E[R], with action a sampled from pi_theta and reward having no additional direct dependence on theta, the score-function identity gives gradient J = E[R gradient log pi_theta(a)]. The environment need not supply a derivative of reward with respect to action.
A baseline b that is independent of the sampled action conditional on the state can replace R by R - b without changing the expected policy-gradient term. An appropriate baseline can reduce variance, although an arbitrary baseline need not do so.
For episodic learning, accumulate policy log-probability gradients weighted by returns, with discounting handled consistently with the chosen objective. Standard Monte Carlo REINFORCE uses sampled returns rather than bootstrapped value targets. The actor update should treat its return and baseline weights as fixed quantities when differentiating this estimator.

Three practical examples

Simulated control: a policy samples actions, receives episode outcomes, and improves its action probabilities without requiring a differentiable simulator.
Discrete design search: a model samples candidate configurations and receives a quality score from an external evaluation procedure.
Sequence-generation research: a stochastic generator receives a sequence-level score that is difficult to differentiate directly, and a score-function update adjusts its sampling distribution.

Think of it this way

Think of repeatedly trying choices according to an adjustable lottery, then changing the lottery based on how the sampled choices performed relative to a reference level. The lottery's probabilities are what the model can differentiate. This analogy does not mean simply repeating every rewarded choice or attributing a noisy outcome entirely to one action.

Worked example

Consider an original one-step bandit with two actions. Action A always earns 4 and action B always earns 1. Let p = sigmoid(theta) be the probability of A, currently p = 1/4. Expected reward is J = 4p + 1(1 - p) = 1 + 3p, so its exact derivative is 3p(1 - p) = 9/16 = 0.5625.
The log-probability derivative is 1 - p = 3/4 for A and -p = -1/4 for B. Without a baseline, a sampled REINFORCE gradient is therefore 3 for A or -1/4 for B. Its expectation is (1/4)3 + (3/4)(-1/4) = 9/16, matching the exact derivative.
Now use fixed baseline b = 2. The gradients become (4 - 2)(3/4) = 3/2 for A and (1 - 2)(-1/4) = 1/4 for B. Their expectation remains (1/4)(3/2) + (3/4)(1/4) = 9/16.
The baseline changes the samples while preserving their expected gradient. In this special deterministic example both baseline-adjusted outcomes favor increasing A's probability. General noisy tasks do not guarantee this behavior, and this chosen baseline is not claimed to be universally optimal.

Illustrative case study

In a fictional project, Spruce Design samples discrete component combinations and evaluates them in a simulator. Early REINFORCE training is erratic because a few unusually high scores dominate updates. The team logs sampled probabilities and rewards, then compares batches of estimates with and without a separately maintained baseline.
It evaluates learned sampling policies across repeated seeds and independent simulation scenarios. Before drawing conclusions, it also checks whether the scoring rule rewards desirable designs or merely exploits a shortcut in the simulator. The case illustrates estimator and objective validation rather than claiming that a baseline alone guarantees reliable design optimization.

Common misconceptions and mistakes

REINFORCE does not require the sampled action to be differentiable, but it does require the relevant log-probability derivative. Replacing that derivative with the derivative of reward is a different estimator.
A baseline that depends on the sampled action generally cannot be subtracted without an appropriate correction. Reusing data from a different behavior policy also needs careful treatment; ordinary on-policy formulas are not automatically valid for arbitrary logged actions. A positive reward can still produce an update away from an action when its reward is below the baseline. Finally, low-variance estimates can still optimize a badly specified reward.

Business use

Use REINFORCE when decisions are stochastic and outcomes supply a useful learning signal, especially where direct differentiation through the outcome is unavailable. Budget for repeated evaluations because score-function estimates can be noisy. Track reward distributions, estimator variance, and policy entropy alongside average training reward.
Document the sampling policy, return definition, baseline, and update timing. Compare the trained policy with simple baselines on independent trials. For applications where experimentation has real costs, define the permissible action set and evaluation conditions separately from the optimization method.

Check your understanding

In the bandit example, why is the baseline-adjusted gradient positive even when B is sampled?

Answer: B earns less than the baseline, giving a negative reward difference. Its log-probability derivative with respect to A's logit is also negative. Their product is positive, shifting probability toward A.

Related terms

Policy gradient
Score function estimator
Stochastic policy
Return
Baseline
likelihood-ratio gradient
variance reduction
actor-critic
Monte Carlo estimation

Learn next

Enumerate both outcomes in the bandit example and calculate the gradient variance for several fixed baselines. Then replace deterministic rewards with noisy rewards and compare sample averages with the exact expected gradient.

Sources

Williams, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning, original1992 paper, REINFORCE theorem, baselines, and episodic extension: https://gwern.net/doc/reinforcement-learning/model-free/1992-williams.pdf Original publisher record and abstract: https://doi.org/10.1007/BF00992696 Williams Sections 4–5 and 8.3 independently consulted for baseline independence, episodic updates and variance limitations.

Previous: Reification (computer science) Next: Reinforcement