Can lambda equal one?

Yes in the episodic formulation, where it selects the full observed return.

Must the lambda return wait until an episode ends?

Its full forward episodic form does; truncated and backward-view algorithms provide other computational arrangements.

Is it always between its component returns?

With lambda between zero and one and correctly normalized weights, the scalar mixture lies between their minimum and maximum.

AI Glossary / Reinforcement learning and robotics / Advanced

Lambda return

Definition

A lambda return is a weighted combination of multi-step returns used as a learning target in reinforcement learning. The parameter lambda determines how much weight is given to longer returns relative to shorter ones. It connects one-step temporal-difference targets with full episodic Monte Carlo returns, providing a forward-looking interpretation of eligibility-trace methods.

What it means

A one-step target combines the next reward with an estimated value of the next state. A longer return uses more actual rewards before relying on a value estimate. The lambda return blends these different horizons rather than choosing only one fixed horizon.

Lambda does not discount rewards directly. Gamma controls the importance of future rewards within each component return, while lambda controls how the component returns are mixed. The blend trades reliance on value estimates against exposure to longer sequences of random rewards. Neither a larger nor a smaller lambda is universally best for prediction or control.

How it works

For a nonterminal n-step horizon, form the return from the next n rewards, discounted by powers of gamma, plus gamma to the nth power times a value estimate at the reached state. If the episode terminates first, stop at termination and omit any terminal bootstrap value.

For an episode ending at T, the lambda return at t assigns weight (1 minus lambda) times lambda to the power n minus one to each shorter n-step return, for n from one through T minus t minus one. The remaining full-episode return receives weight lambda to the power T minus t minus one.

These weights sum to one. The final term collects the remaining weight that would otherwise extend beyond termination. At lambda zero, the target is the one-step return; at lambda one, it is the full episodic return.

A forward target depends on future rewards, so it may not be immediately available online. Eligibility traces provide related backward computations. Exact equivalence depends on the update convention and whether estimates change during the episode; ordinary online TD(lambda) should not be casually identified with every fixed-value forward calculation.

Three practical examples

1. A value-prediction experiment compares different lambda targets to study how quickly delayed rewards influence earlier states.

2. A simulation team blends short and longer returns when intermediate value estimates are useful but not fully trusted.

3. A reinforcement-learning instructor expands a short episode's lambda return explicitly to explain how trace-based updates relate to multi-step learning.

Think of it this way

Think of asking several forecasters who look different distances ahead. The shortest-horizon forecaster relies heavily on an estimate of what comes next; the longest waits for more actual outcomes. Lambda sets the weights given to their answers. It does not change how each forecaster values rewards at different times - that is gamma's role.

Worked example

Consider a state followed by three rewards, 1, 2 and 3, after which the episode terminates. Set gamma = 0.5. Hold the value estimates fixed at four after the first reward and six after the second reward; the terminal value is zero.

The one-step return is 1 + 0.5 times 4 = 3. The two-step return is 1 + 0.5 times 2 + 0.5 squared times 6 = 3.5. The complete three-step return is 1 + 0.5 times 2 + 0.5 squared times 3 = 2.75.

With lambda = 0.6, the first two weights are 0.4 and 0.4 times 0.6 = 0.24. The terminal return receives the remaining weight 0.6 squared = 0.36. Their sum is one.

The lambda return is 0.4 times 3 + 0.24 times 3.5 + 0.36 times 2.75 = 3.03. Lambda zero would give three, while lambda one would give 2.75.

It would be incorrect to multiply the final weight by another factor of 0.4. Doing that would leave the weights summing to only 0.784 and would discard part of the intended mixture. The example uses fixed bootstrap estimates to make the forward target unambiguous; it does not prescribe a particular online parameter-update schedule.

Illustrative case study

Fictional case: Rowan Learning evaluates a warehouse-control policy in simulation. Early value estimates are inaccurate, while final delivery rewards are noisy. Analysts compare several lambda settings instead of assuming that either the shortest or longest target is best.

They evaluate prediction error across independent simulated episodes and track how results change as training improves the value estimates. A setting that helps early learning may behave differently later. The team records gamma, lambda and the update convention separately so that the experiment can be reproduced and interpreted.

Common misconceptions and mistakes

Lambda is not gamma, a learning rate or a fixed number of steps. It changes the weighting across multiple horizons.

The last episodic weight differs from the ordinary shorter-horizon weights because it absorbs the remaining probability mass. Ignoring that distinction changes the target.

The lambda return is not inherently an estimate with no bias. Bootstrapped value estimates can introduce error, while longer sampled returns can add variability. In continuing or truncated settings, convergence conditions and the treatment of the remaining tail must be stated. Off-policy use also requires appropriate corrections; simply reusing on-policy returns can target the wrong policy.

Business use

Business relevance is in designing learning targets for sequential prediction and simulated decision systems. Lambda provides a controllable way to combine model estimates with observed delayed outcomes. Teams should compare sample efficiency, variability and final prediction quality rather than optimizing training loss alone. Keep the reward definition, discount and target-policy assumptions explicit, especially when evaluating models from logged trajectories.

Check your understanding

Why is the final weight 0.36 rather than 0.144?

Answer: It collects all remaining weight after the two shorter returns.

Which parameter produces the factors 0.5 and 0.25 inside the example's returns?

Answer: Gamma, not lambda.

Related terms

n-step return
Eligibility trace
TD(lambda)
temporal-difference learning
Monte Carlo return
bootstrapping and generalized advantage estimation

Learn next

Next, expand a four-step episode and verify that its weights sum to one. Then compare the fixed-value forward target with accumulating-trace and true-online implementations, noting when their update behavior agrees or differs.

Sources

Primary source: Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, author textbook draft, Chapter 12 on the lambda return and eligibility traces: https://web.stanford.edu/class/psych209/Readings/SuttonBartoRL.pdf

The three-reward calculation and fictional case are original teaching material. The worked target uses fixed bootstrap estimates and an explicit terminal convention.

Previous: Lambda Next: Lamport's distributed mutual exclusion algorithm