Does lambda one always mean ordinary Monte Carlo code?

No. The forward-return interpretation reaches Monte Carlo targets, but online update conventions matter.

Must every trace be stored forever?

No. Traces decay and episodic algorithms reset them at episode boundaries.

Can it use neural networks?

Gradient-based traces are possible, but computational and stability issues require additional attention.

AI Glossary / Probabilistic machine learning / Advanced

TD(lambda)

Definition

TD(lambda), written TD(λ), is a family of temporal-difference learning methods that use eligibility traces to spread prediction errors across recently visited states or recently active model parameters. The parameter lambda controls how strongly earlier activity remains eligible for an update. Here the focus is state-value prediction under a fixed policy, using ordinary accumulating traces.

What it means

A reward may arrive several steps after the events that helped produce it. One-step temporal-difference learning updates the current prediction using that reward and the next state's estimated value. Eligibility traces let the same newly observed error also influence earlier predictions.

Lambda is not the reward discount factor. Discounting determines how future rewards enter the value being predicted; lambda influences how learning information is distributed through time. A longer trace can accelerate credit assignment, but it may also increase sensitivity to noisy rewards and interacting updates. The appropriate setting depends on the data, representation and learning rate.

How it works

For tabular prediction, maintain a value V(s) and a trace e(s) for each state. Start an episode with zero traces. At a transition from state S to S-prime with reward R, compute the temporal-difference error delta = R + gamma V(S-prime) minus V(S). The terminal state's value is zero.

Multiply every existing trace by gamma times lambda, then add one to the current state's trace. Update every state's value by alpha times delta times its trace. Alpha is the learning rate. Repeated visits can accumulate trace strength; replacing traces and other variants use different rules.

The forward interpretation combines returns of different lengths. Lambda zero gives one-step TD. In the episodic forward lambda-return, lambda one gives the full Monte Carlo return. Ordinary online TD(lambda) should not be described as exactly identical to every forward-return update at finite learning rates; true-online variants address a particular exact-equivalence objective.

With function approximation, traces apply to value gradients rather than simply to state indicators. Stability and convergence then depend on additional conditions, especially when behavior and target policies differ.

Three practical examples

1. A game-playing research system estimates the expected return from board positions, allowing an eventual reward to influence several recently visited positions.

2. A simulated maintenance planner evaluates a fixed inspection policy when costs appear after sequences of equipment states.

3. A teaching environment compares short and long traces on the same episodic task to study learning speed, variability and sensitivity to step size.

Think of it this way

Think of recently visited states as leaving fading chalk marks. A new prediction error changes values according to how visible each mark remains. Lambda controls part of the fading rate, while gamma controls how the task discounts the future. The marks are temporary learning aids, not additional rewards.

Worked example

Consider one episode A to B to terminal, with rewards zero and then two. Initialize V(A) = V(B) = 0 and all traces to zero. Choose gamma = 1, lambda = 0.8 and alpha = 0.1. These are illustrative tabular settings.

At A, the temporal-difference error is zero + V(B) minus V(A) = 0. The trace for A becomes one, but neither value changes because the error is zero.

At B, decay the A trace to 0.8 and set the B trace to one. The terminal transition produces error two + zero minus V(B) = 2. Update V(A) by 0.1 times 2 times 0.8 = 0.16. Update V(B) by 0.1 times 2 times 1 = 0.20.

With lambda zero, A's trace would have vanished before the final reward, so this episode would leave V(A) at zero while V(B) becomes 0.20. With lambda one, A would also become 0.20 in this particular episode.

These values are learning updates, not exact expected returns. Their simplicity depends on distinct states, zero initial values and the stated accumulating-trace convention. Repeated states or different initial estimates change the arithmetic.

Illustrative case study

Fictional case: Birch Simulation tests a warehouse-routing policy in a simulator where successful delivery yields a reward only after several moves. Its analysts compare value prediction with different lambda values while keeping the policy fixed.

Longer traces spread late rewards to earlier route states faster in some runs, but noisy congestion outcomes increase variation. The team selects settings using separate simulated episodes and records both average error and variability. It does not interpret faster training on one trajectory as proof that the route policy itself is better.

Common misconceptions and mistakes

Lambda is not the learning rate and is not a substitute for gamma. Increasing it does not universally improve learning.

Eligibility traces are not replay buffers: they summarize recent eligibility rather than storing arbitrary past transitions for later sampling. Accumulating, replacing and Dutch traces should not be mixed without specifying the algorithm.

Basic state-value TD(lambda) evaluates a policy; it does not by itself specify action selection or policy improvement. Off-policy use also requires care. Combining bootstrapping, function approximation and off-policy data can produce instability that simply adjusting lambda does not automatically fix.

Business use

The business relevance is mainly in sequential-decision research and simulation-based policy evaluation. Traces can make delayed feedback more useful during learning. Before operational use, compare prediction accuracy, sample efficiency and stability across multiple runs. Keep the reward definition and evaluation policy explicit so that improved value prediction is not confused with an improvement in business outcomes.

Check your understanding

In the example, why does A change after leaving A?

Answer: Its eligibility trace is still 0.8 when the final error arrives.

Which parameter discounts future rewards?

Answer: Gamma; lambda controls trace persistence and return mixing.

Related terms

Eligibility trace
Lambda return
SARSA(lambda)
temporal-difference learning
Monte Carlo return
bootstrapping
true-online TD and policy evaluation

Learn next

Next, compute the same episode with a repeated state and compare accumulating with replacing traces. Then examine the forward lambda-return and the assumptions behind stable policy evaluation.

Sources

Primary source: Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, author textbook draft, Chapter 12, reproduced by Stanford: https://web.stanford.edu/class/psych209/Readings/SuttonBartoRL.pdf

The worked episode and fictional case are original teaching examples. The update equations here specifically describe tabular accumulating traces.

Independent review could not retrieve the cited textbook copy. It consulted van Seijen and colleagues, True Online Temporal-Difference Learning, Sections 2.3–2.4 and the forward-view distinction: https://arxiv.org/pdf/1512.04087

Previous: Taylor expansion Next: Teacher forcing