Can a baseline depend on the state?

Yes; state-value baselines are common.

Must it equal the true value?

No, and the worked example shows a different constant can reduce variance further.

Does lower variance guarantee a better final policy?

No. Optimization settings, exploration and model limitations also affect results.

AI Glossary / Reinforcement learning and robotics / Advanced

Policy-gradient baseline

Definition

A policy-gradient baseline is a reference quantity subtracted from a return or action-value estimate when forming a policy-gradient update. Under the appropriate action-independence condition, it changes the estimator's variability without changing its expected gradient. A common baseline is a learned state-value function, which turns the learning signal into an estimate of advantage relative to the state's usual outcome.

What it means

An outcome can be good in absolute terms but ordinary for the situation in which it occurred. A baseline helps a learner distinguish those cases. The policy update responds to how much better or worse the outcome was than the reference, rather than only to its raw magnitude.

The purpose is usually variance reduction, making noisy sampled gradients more useful. It is not a rule that the baseline must be zero, the average reward or the exact value function. Those are possible choices, and their effectiveness depends on the estimator and problem. A poorly chosen baseline can increase variance even while preserving the expected gradient.

How it works

For a sampled action a in state s, a standard score-function policy-gradient contribution has the form grad log pi(a given s) multiplied by a return estimate G. Replace G with G minus b(s), using a baseline that does not depend on the sampled action conditional on the information being conditioned on.

The baseline contribution has zero expectation because the policy-weighted sum of grad log pi equals the gradient of the sum of action probabilities, which is the gradient of one. This explains why a fixed state-dependent baseline can preserve the expected policy gradient.

A value baseline is often trained by regression toward returns or other suitable value targets. In the policy loss, the baseline or advantage is normally treated as a fixed learning signal; inadvertently differentiating through it can introduce an unintended term. Shared network components need deliberate handling of policy and value losses.

Action-dependent baselines require compensating constructions rather than naive subtraction. Likewise, a baseline fitted using the same sampled action or return can introduce dependence that the simple zero-expectation argument does not cover. Sampling, fitting and gradient conventions should be explicit.

Three practical examples

1. An episodic policy-gradient learner subtracts a state-value prediction so unexpectedly good outcomes receive positive advantage estimates.

2. A research comparison measures gradient variability with no baseline, a constant baseline and a learned state-dependent baseline.

3. An implementation review checks that the actor's loss does not accidentally backpropagate through an advantage target intended to be held fixed.

Think of it this way

Think of assessing a runner against the expected time for the course and conditions. A performance that looks slow in isolation may be strong on a difficult course.

Worked example

Consider an original one-step, two-action bandit. Action A gives reward 4 and B gives reward 0, both deterministically. The policy selects A with probability p equal to 0.25 and B with probability 0.75. Parameterize p by a logistic logit theta.

The derivative of log pi with respect to theta is 1 minus p, or 0.75, when A is selected. It is minus p, or -0.25, when B is selected. The true expected reward is 4p, whose derivative is 4p(1 minus p), or 0.75.

Without a baseline, the sampled gradient is 3 for A and 0 for B. Its expectation is 0.25 times 3, or 0.75. Its variance is 0.25 times (3 minus 0.75) squared plus 0.75 times (0 minus 0.75) squared, giving 1.6875.

Choose baseline b equal to 1, the mean reward. The two sampled gradients become 0.75 times (4 minus 1), or 2.25, and -0.25 times (0 minus 1), or 0.25. Their weighted mean remains 0.75, while their variance falls to 0.75.

Now choose baseline b equal to 3. The A gradient is 0.75 times 1, or 0.75. The B gradient is -0.25 times -3, also 0.75. Every sample produces the correct gradient, so variance is zero in this particular example.

This demonstrates why the expected reward baseline need not be variance-optimal. For this scalar score estimator, the optimal constant baseline weights returns by squared scores: E[score squared times reward] divided by E[score squared]. Here that is 0.5625 divided by 0.1875, giving 3. The result relies on the stipulated one-step deterministic rewards; it is not a promise of zero-variance policy learning in general.

Illustrative case study

In a fictional project, Pine Control Lab finds that a policy-gradient experiment is sensitive to random seeds. The team adds a learned state baseline and measures the variability of gradient estimates as well as task reward.

An implementation check discovers that one version allowed the actor loss to differentiate through the baseline unintentionally. The team separates the intended gradient paths and evaluates again.

Common misconceptions and mistakes

A baseline is not the same thing as changing the environment's reward function. It modifies the gradient estimator while leaving the intended objective intact under stated conditions.

Subtracting any arbitrary quantity is not automatically unbiased. Dependence on the sampled action needs special treatment, and data reuse can complicate the argument.

Do not assume that an inaccurate state baseline necessarily biases a Monte Carlo policy-gradient estimator. Action-independent approximation can preserve its mean while changing variance. In contrast, replacing returns with bootstrapped critic estimates can introduce additional approximation bias; that is a separate issue from baseline subtraction alone.

Business use

Baselines are relevant when a learning system needs many interactions because policy updates are noisy. Evaluation should compare data requirements, stability and final task performance, accounting for the extra cost of training a baseline model. For reviewers, the key questions are what reference is used, how it is learned and whether the intended unbiasedness conditions match the implementation.

Check your understanding

Why is the expected gradient unchanged across all three calculations?

Answer: Each baseline contribution has zero mean under the specified policy.

Why does b equal to 3 produce zero variance here?

Answer: It makes both possible sampled gradient contributions equal to 0.75.

Related terms

Policy gradient
REINFORCE
Advantage function
Control variates
actor-critic
control variate
score-function estimator and variance reduction

Learn next

Next, repeat the calculation with probability 0.5 and compare the optimal baseline. Then study state-dependent baselines and the distinction between unbiased Monte Carlo advantages and bootstrapped estimates.

Monte Carlo

Sources

Primary source: Evan Greensmith, Peter L. Bartlett and Jonathan Baxter, Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning: https://jmlr.org/papers/volume5/greensmith04a/greensmith04a.pdf

The logistic bandit derivation and fictional case are original teaching material. The example uses a scalar score estimator and action-independent fixed baselines, not the paper's full GPOMDP setting.

Previous: Policy gradient Next: Policy gradient theorem