Is reward-to-go the same for every action?

Generally no, because each decision has a different remaining reward sequence.

Yes, when remaining costs outweigh rewards under the stated convention.

A complete sampled reward-to-go is observed retrospectively; a value function predicts its expectation.

AI Glossary / Reinforcement learning and robotics / Intermediate

Reward-to-go

Definition

Reward-to-go is the cumulative reward remaining from a particular decision onward in a trajectory. It excludes rewards received before that decision and may discount later rewards according to the task definition. In policy-gradient learning, sampled reward-to-go can help evaluate an action using its subsequent outcomes rather than weighting it with the entire episode's past and future rewards.

What it means

An episode contains many decisions, and each decision has a different amount of the episode still ahead. Reward-to-go captures the relevant suffix of the reward sequence. The first decision may have a long horizon remaining, while the final decision has only its immediate resulting reward.
The term often refers to a sampled return calculated after observing an episode. It should not be confused with an expected value function, which averages over possible future outcomes under a policy. One observed reward-to-go can be unusually high or low even when the underlying expected value is unchanged.

How it works

Use a consistent indexing convention. If action A_t produces reward R_(t+1), then discounted reward-to-go G_t is R_(t+1) plus gamma times R_(t+2), plus gamma squared times R_(t+3), continuing until termination. Setting gamma to 1 gives the undiscounted sum for a finite episode.
An efficient calculation works backward. Start with zero beyond a true terminal state, then set each preceding return to its resulting reward plus gamma times the next return. This produces a separate learning weight for each decision.
In a standard on-policy policy-gradient derivation, rewards that occurred before an action have zero expected contribution when multiplied by that action's log-probability gradient. Removing those terms avoids attributing past outcomes to a later choice and can reduce irrelevant sampling noise. A state-dependent baseline can further adjust the learning weight without changing the expected gradient under the usual conditions.
Complete sampled reward-to-go requires the future portion of the episode. If data collection stops before the underlying task ends, the observed suffix is incomplete. Bootstrapping from a value estimate can address that cutoff, but the result is then an estimated return target with its own assumptions.

Three practical examples

Episodic policy learning: assign different return weights to actions taken early and late in a game.
Training diagnostics: inspect which later outcomes influence the learning signal for a particular decision.
Simulator analysis: distinguish complete episode returns from truncated rollout targets before comparing algorithms.

Think of it this way

Think of reviewing a journey from each junction. From the first junction, all remaining travel time matters. From the last junction, earlier delays have already happened. The remaining journey can still be uncertain, but the later decision cannot change the delays that preceded it.

Worked example

Consider a fictional episode with three decisions and resulting rewards 2, -3, and 8. The third reward ends the task. With discount factor 0.50, the last decision's reward-to-go is G2 = 8. Working backward gives G1 = -3 + 0.50 times 8 = 1, then G0 = 2 + 0.50 times 1 = 2.5.
Checking directly, G0 = 2 + 0.50 times (-3) + 0.25 times 8 = 2.5. The weights for the three decisions are therefore 2.5, 1, and 8. The value 8 for the final decision is not inconsistent with the first decision's 2.5: the starting points and discounting differ.
Without discounting, the corresponding reward-to-go values are 7, 5, and 8. Using the total episode reward 7 for every decision would include the earlier reward 2 in the second decision's weight and both earlier rewards in the third decision's weight. The reward-to-go construction removes those past contributions.
Now suppose collection stops after the second reward, while the task remains active. Treating this cutoff as true termination gives the incomplete discounted target 2 + 0.50 times (-3) = 0.5 for the first decision. If a next-state value estimate of 6 is used for bootstrapping, the target becomes 2 - 1.5 + 0.25 times 6 = 2. The eventual complete return in the original episode was 2.5 because the realized final reward was 8 rather than the estimate 6.
These are three different quantities: an incomplete observed suffix, a bootstrapped estimate, and the complete sampled return. None should be silently relabeled as another. The calculations also do not imply that one trajectory equals the expected value of a policy; estimating that expectation requires suitable averaging or modeling.

Illustrative case study

Fictional case: ElmAgents trains a simulator policy from batches of fixed-length rollouts. A report calls every batch's accumulated reward a complete reward-to-go, even when some tasks continue beyond the batch boundary. This makes long-running tasks appear less valuable than they are.
The team distinguishes true terminal states from collection cutoffs and documents its bootstrap targets. It also logs complete episodes for independent evaluation. This invented case illustrates data-handling and interpretation, not a claim that a particular correction guarantees successful learning.

Common misconceptions and mistakes

Common mistakes include including rewards from before the action, dropping the reward immediately caused by the action, and confusing a sampled return with expected future value. Discounted and undiscounted conventions must not be mixed. For policy-gradient implementations, the full objective and time weighting also matter; replacing one array of rewards without understanding the derivation can change the intended estimator.

Business use

Businesses reviewing sequential-learning experiments should ask whether reported return weights cover complete tasks, how future outcomes are discounted, and what happens at data-collection boundaries. These details affect the meaning of training signals and comparisons. Clear reward-to-go calculations are particularly helpful when early decisions create costs now and benefits much later.

Check your understanding

What is G1 at discount 0.50?

Answer: One.

Why is the incomplete first target only 0.5?

Answer: It omits the unobserved final outcome.

What does the bootstrapped value 2 depend on?

Answer: The next-state value estimate of 6.

Related terms

Return
Reward
Policy gradient
REINFORCE
Baseline
Discount factor
value function
advantage
bootstrapping
episode truncation

Learn next

Next, calculate returns backward for several short episodes with different terminal locations. Then compare complete Monte Carlo returns with bootstrapped targets when rollouts end before the task does.

Monte Carlo

Sources

OpenAI Spinning Up, Introduction to Policy Optimization, reward-to-go derivation: https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, second edition, Sections 3.3 and 13.3: https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf

Previous: Reward shaping Next: Rhetorical structure theory