What happens when lambda is zero?

The correction terms cancel appropriately and the method reduces to one-step linear TD.

Does it need to store the entire episode?

The efficient backward view does not.

Is it the same as replacing TD(lambda)?

No; both its trace and weight correction are different.

AI Glossary / Reinforcement learning and robotics / Advanced

True online TD(lambda)

Definition

True online TD(lambda) is a temporal-difference prediction algorithm whose efficient backward-view updates exactly match a specified online forward view for linear value approximation. It combines Dutch eligibility traces with a correction for predictions changing during an episode. The word true describes this algebraic equivalence; it does not mean that every prediction is the true value or that every learning problem converges automatically.

What it means

Traditional lambda-return reasoning combines short and long returns to explain credit assignment. Its familiar exact equivalence with conventional TD(lambda) assumes an offline setting where the relevant estimates remain fixed during the episode. Online updates change those estimates as experience arrives.

True online TD(lambda) was derived to match a forward view that explicitly accommodates such changes. It produces the same weights without repeatedly revisiting the entire growing history after each transition.

The method is informative when features recur and finite step sizes make update ordering matter. It remains an on-policy prediction algorithm in its basic form. Separate extensions address other settings, and the linear equivalence should not be assumed for an arbitrary nonlinear neural approximation.

How it works

For the constant-step-size linear version, predict V as theta dotted with current features phi and V' as theta dotted with next features phi'. Compute delta as reward plus gamma times V' minus V. Maintain a trace vector e and a stored scalar V_old, both reset to zero at episode start.

Update the trace to gamma times lambda times the old trace, plus phi, minus alpha times gamma times lambda times (old e dotted with phi) times phi. This is the Dutch trace.

Then update theta by adding alpha times (delta plus V minus V_old) times the new trace, and subtracting alpha times (V minus V_old) times phi. Store the pre-update V' as the next V_old. For a terminal state, use zero continuation features or an equivalent terminal-value convention.

Both the special trace and the prediction correction matter. Substituting Dutch traces into ordinary TD(lambda) without the other correction generally loses the intended equivalence. Time-varying step sizes require the corresponding formulation rather than an unexamined substitution into the constant-step-size equations.

Three practical examples

1. A prediction experiment compares algorithms when the same feature repeatedly activates before a delayed reward.

2. A developer validates an efficient backward-view implementation against a deliberately slow online forward-view calculation on a short episode.

3. A teaching exercise shows why different trace conventions produce different finite-step-size updates even when the rewards and features are identical.

Think of it this way

Think of maintaining a running account that must agree exactly with recalculating all eligible adjustments as new information arrives. Repeating the full calculation would be expensive. True online TD(lambda) carries enough bookkeeping to reproduce that particular account efficiently. Agreement with the account proves implementation equivalence, not that its forecasts perfectly describe the world.

Worked example

Consider a two-transition episode with a shared feature equal to one at both nonterminal visits and zero at termination. Start with scalar weight theta equal to zero. Set alpha to 0.5, gamma to 0.9 and lambda to 0.8. Rewards are 1 and then 2. This can represent revisiting a state or two states sharing the same feature.

On the first transition, V and V' are zero, so delta is 1. The Dutch trace becomes 1 and theta becomes 0.5. Store V_old equal to the pre-update next prediction, zero.

On the terminal transition, V is 0.5, V' is zero and delta is 1.5. The new Dutch trace is 0.72 times 1 plus 1 minus 0.5 times 0.72 times 1, or 1.36. The prediction change V minus V_old is 0.5.

The weight becomes 0.5 plus 0.5 times (1.5 plus 0.5) times 1.36 minus 0.5 times 0.5, or 1.61.

An independent forward-view check reaches the same number. At the final horizon, the first interim lambda target is 0.2 times the original one-step target 1, plus 0.8 times the full return 1 plus 0.9 times 2. This gives 2.44. Replaying from the initial weight zero gives 1.22 after the first update, then 1.22 plus 0.5 times (2 minus 1.22), or 1.61, after the second.

Ordinary accumulating traces would instead reach trace 1.72 and final weight 0.5 plus 0.5 times 1.5 times 1.72, or 1.79. The difference illustrates finite-step-size equivalence, not a claim that 1.61 is necessarily a better prediction in every environment.

Illustrative case study

In a fictional project, Rowan Signals predicts future equipment rewards using overlapping binary features. A researcher notices that trace-based algorithms disagree on a short recorded episode, despite identical initial parameters.

The team constructs a slow forward-view reference and compares every intermediate weight with its true-online implementation. A discrepancy reveals that V_old had been stored after the parameter update instead of preserving the required earlier prediction.

The corrected reference calculations agree. Prediction quality still requires separate evaluation. This is an illustrative validation process, not a deployment or evidence of universal superiority.

Common misconceptions and mistakes

Do not interpret true online as a guarantee of true values. Do not confuse Dutch traces with replacing traces. Do not omit the prediction-change correction or recompute the stored next prediction after updating theta.

Another mistake is presenting the method as having no additional arithmetic. Its asymptotic per-step complexity remains linear in feature count, but the extra terms have a real computational cost. Exact equivalence also depends on using the matching forward-view definition.

Business use

The method offers a precise reference for teams using linear value prediction and eligibility traces. Its strongest practical advantage for review is that a short forward-view calculation can validate the optimized implementation. Compare prediction accuracy, sensitivity to alpha and lambda, and runtime on the actual representation. Algebraic equivalence alone does not settle whether the features or reward target suit the business problem.

Check your understanding

What is the final weight in the example?

Answer: 1.61.

What does the independent forward-view calculation verify?

Answer: Agreement with the specified online update semantics.

Why is V_old zero on the second step?

Answer: It stores the earlier next-state prediction computed before the first parameter update.

Related terms

TD(lambda)
Lambda return
Accumulating eligibility trace
Dutch trace
Online forward view
Linear function approximation
True online SARSA

Learn next

Study ordinary TD(lambda) and lambda returns first. Then implement a small online forward-view reference before studying Dutch traces and prediction corrections. Treat variable step sizes and off-policy extensions separately.

Sources

Van Seijen, Mahmood, Pilarski, Machado and Sutton, True Online Temporal-Difference Learning, JMLR 2016, sections 3–4 and 6.1: https://www.jmlr.org/papers/volume17/15-599/15-599.pdf. The constant-step-size equations and scope were checked directly. The repeated-feature example and fictional case are original teaching illustrations.

Previous: True negative Next: True positive