What happens when n equals one?

The return becomes a one-step bootstrapped target.

Does n reaching the episode end remove bootstrapping?

Yes, for that starting point under the stated terminal convention.

Must n be the same throughout training?

No, although changing it requires an algorithm and evaluation procedure that account for the varying targets.

AI Glossary / Reinforcement learning and robotics / Intermediate

n-step return

Definition

An n-step return is a reinforcement-learning target that combines the next n observed rewards with an estimated continuation value at the end of that horizon. If an episode terminates before n steps, the return stops at termination and contains no further bootstrap value. It connects one-step temporal-difference targets with longer observed returns.

What it means

A learner often needs to estimate future reward before an entire episode has finished. A one-step target uses one observed reward and immediately relies on a prediction of what follows. An n-step return waits for more actual rewards before using that prediction.

The horizon n controls how much of the target comes from observed experience. Larger horizons can carry delayed outcomes farther back in one update, but they also introduce delay and may increase variability. More observed steps do not make every individual target more accurate.

How it works

Starting at time t, add the next reward, gamma times the following reward, and successive rewards multiplied by increasing powers of gamma. After n nonterminal steps, add gamma to the power n times the estimated value of the reached state.

For state-value prediction, that last estimate is V(S at t plus n). For n-step Sarsa, it is the action value Q(S, A) for the action selected at the horizon. Other control algorithms may use an expected or maximum action-value bootstrap; their policy assumptions differ.

If termination occurs first, use only rewards through the terminal transition. A time limit imposed by data collection is not automatically a genuine terminal state; the intended continuing or episodic task determines whether to bootstrap there.

Once the target is available, a tabular learner can move its earlier estimate toward it using a learning rate. An online implementation needs to retain recent transitions and finish outstanding updates after an episode ends. When data come from another policy, a raw reward sequence can reflect that behavior policy unless the chosen algorithm applies appropriate corrections.

Three practical examples

1. A navigation learner uses several rewards to carry information about reaching a goal back toward earlier state estimates.

2. A simulator evaluates the effect of n on prediction accuracy, target variability and the delay before updates become available.

3. A sequential decision model stores contiguous transition segments so multi-step targets preserve the order of rewards and terminal boundaries.

Think of it this way

Think of estimating the cost of a journey already under way. You can combine one known leg with a forecast for the rest, or wait until several legs have been completed before forecasting the remainder. Waiting gives more observed information, but the eventual estimate still depends on the remaining forecast and the particular journey experienced.

Worked example

Consider an original four-transition episode with rewards 2, -1, 3 and 4, followed by termination. Set gamma to 0.8. Hold bootstrap estimates fixed: after the first transition the estimate is 5, after the second it is 6, and after the third it is 7.

The one-step return from the initial state is 2 plus 0.8 times 5, giving 6. Only the first reward is observed in this target; all later rewards are represented by the bootstrap estimate.

The two-step return is 2 plus 0.8 times -1 plus 0.8 squared times 6. This gives 2 minus 0.8 plus 3.84, or 5.04.

The three-step return is 2 minus 0.8 plus 0.64 times 3 plus 0.512 times 7. The total is 6.704. The power 0.512 belongs to the continuation estimate because it begins after three transitions.

The four-step return reaches termination, so no bootstrap is added. It is 2 minus 0.8 plus 0.64 times 3 plus 0.512 times 4, giving 5.168. Choosing n greater than four produces the same terminally truncated return for this starting point.

If the initial value estimate is 4 and the learning rate is 0.25, using the three-step target updates it to 4 plus 0.25 times (6.704 minus 4), or 4.676.

The targets are not ordered by horizon: 6, 5.04, 6.704 and 5.168. Different bootstrap errors and the realized rewards explain that pattern. One episode alone cannot establish which horizon gives the best expected learning performance.

Illustrative case study

In a fictional project, Ash Simulation studies a delayed-reward maintenance task. A short-horizon learner initially propagates information slowly, so the team experiments with longer returns.

It preserves episode boundaries in replay and distinguishes genuine task completion from a simulator time limit. The report compares several horizons over repeated runs and includes both learning speed and variability. A larger n helps in some settings but adds unstable targets in others. The team chooses its configuration from separate evaluation results rather than assuming that using more observed rewards must always be better.

Common misconceptions and mistakes

Do not discount the first reward by gamma; it has exponent zero under the standard convention. Also do not add a value after genuine termination.

An n-step return is one horizon's target. A lambda return combines multiple horizons, and eligibility traces provide related backward computations. These concepts are connected but not interchangeable.

Longer returns still depend on policy and sampling assumptions. Reusing a trajectory from an older policy without the appropriate correction can change what is being estimated. Finally, multiple overlapping targets from one episode are correlated; they are not independent experiences simply because their starting times differ.

Business use

N-step returns help technical teams balance immediate learning against incorporating delayed outcomes. Relevant measurements include interactions needed, memory for recent transitions, update latency and eventual task performance. Business reviewers should ask whether the reward horizon reflects the real decision process and whether episode endings are modeled correctly. A faster reduction in training loss is less informative than improved predictions or decisions on separate evaluation episodes.

Check your understanding

Why is there no final value term in the four-step example?

Answer: The fourth transition ends the episode.

What does gamma control?

Answer: How rewards and continuation values at later times are discounted, independently of the chosen horizon.

Related terms

Lambda return
temporal-difference learning
bootstrapping
Monte Carlo return
n-step Sarsa
eligibility trace and experience replay

Learn next

Next, compute returns starting from the second transition of the same episode. Then compare a fixed n-step target with a weighted mixture of horizons.

Sources

Primary source: Kristopher De Asis and colleagues, Multi-Step Reinforcement Learning: A Unifying Algorithm, section on atomic multi-step algorithms and n-step returns: https://arxiv.org/pdf/1703.01327

The four-transition calculation and fictional case are original teaching material. The example uses fixed state-value bootstrap estimates and explicit termination; the source also develops action-value variants.

Previous: N-step Q(sigma) Next: N-step tree backup