Does alpha affect it?

Yes, through the overlap adjustment.

Is the trace the complete learning algorithm?

No; the corresponding parameter update must also be specified.

AI Glossary / Reinforcement learning and robotics / Advanced

Dutch eligibility trace

Definition

A Dutch eligibility trace is a recursively updated feature-memory vector used in reinforcement-learning algorithms such as true online TD(lambda). It modifies the usual accumulating trace by accounting for the current feature vector's overlap with the previous trace and the learning step size. Together with the appropriate weight-update correction, it supports an exact correspondence with the online lambda-return algorithm for linear value approximation.

What it means

An eligibility trace records which features remain involved in learning from later prediction errors. An accumulating trace decays the old vector and adds the current features. A Dutch trace adjusts that addition using information about how much the current features already overlap with the trace.
This adjustment matters when features recur while learning is happening. It is not simply a rule that resets every active component to one. It also does not, on its own, constitute the complete true online learning algorithm: that algorithm includes a corresponding correction to the parameter update.

How it works

For a linear value approximation V(s) = w transpose x(s), let x_t be the current feature vector and z_previous the trace before processing it. With discount gamma, trace parameter lambda and step size alpha, the Dutch trace is:
z_t = gamma lambda z_previous + [1 - alpha gamma lambda (z_previous transpose x_t)] x_t.
The inner product measures overlap between the previous trace and the current feature vector. The trace starts at zero at the beginning of an episode. If the overlap is zero, its update matches decay plus addition of the current features. If lambda is zero, the trace reduces to the current feature vector.
True online TD(lambda) combines this trace with a parameter-update correction that accounts for changes in predictions during learning. For the linear setting described in the source, the complete algorithm matches the online forward view at each step. Replacing only the trace in another algorithm does not establish that equivalence.

Three practical examples

Repeated states: a tabular learner revisits the same state within an episode, allowing the effects of accumulating, replacing and Dutch traces to be compared directly.
Overlapping features: a linear predictor uses features shared by multiple states, so the trace adjustment depends on a vector inner product rather than simply whether the state name repeats.
Implementation checking: a developer computes a short feature sequence by hand and compares the trace and parameter updates separately.

Think of it this way

Imagine a running record of recently used features. Every step fades the record, but a repeated feature is added with an adjustment that recognizes its existing presence and the learning rate. The record is a computational aid for allocating updates; it is not a probability distribution or a literal count of visits.

Worked example

Consider an original two-feature illustration with one-hot vectors A = (1,0) and B = (0,1). Set alpha = 0.2, gamma = 0.9 and lambda = 0.8. The decay product gamma lambda is 0.72. Begin with trace (0,0), and process the feature sequence A, A, B.
On the first A, the old trace has zero overlap with A. The new trace is therefore (1,0).
On the second A, the old trace's inner product with A is 1. The coefficient multiplying A is 1 - 0.2 times 0.72 times 1 = 0.856. The decayed old trace is (0.72,0). Adding (0.856,0) gives the Dutch trace (1.576,0).
For comparison, an accumulating trace after the second A would be (1.72,0), because it adds a full one. A replacing trace in this one-hot setting would be (1,0), because it resets the active component to one. The Dutch result is different from both.
On the following B, the inner product of (1.576,0) with B is zero. The old trace decays to (1.13472,0), and adding B produces (1.13472,1). The accumulating trace for the same sequence would be (1.2384,1), while the replacing trace would be (0.72,1).
Now examine a separate variation: suppose the third feature were A instead of B. Starting from (1.576,0), the new coefficient is 1 - 0.2 times 0.72 times 1.576 = 0.773056. The decayed first component is 1.13472, so the resulting trace is (1.907776,0).
The step size also affects the trace. If alpha were 0.1 instead of 0.2, the second A would produce 0.72 + (1 - 0.1 times 0.72) = 1.648 in the first component. Smaller alpha moves this particular update closer to the accumulating value 1.72.
These calculations isolate the trace recurrence. No rewards, initial parameter vector or prediction-correction terms have been specified, so they do not determine a complete sequence of learned values.

Illustrative case study

Fictional company Birch Learning checks a linear predictor in a simulator with repeated observations. Its engineers test short feature sequences before running longer learning experiments. They record the step size and trace convention alongside the resulting vectors.
After checking the trace recurrence, they validate the full parameter update against the online forward view on a small episode. They do not attribute a mismatch solely to the trace before examining initialization and prediction bookkeeping. This fictional workflow reports no measured accuracy or efficiency gain.

Common misconceptions and mistakes

A Dutch trace is not a replacing trace with another name. It is also not independent of the learning rate: alpha appears directly in its recurrence.
Another error is claiming that installing Dutch traces alone makes any TD implementation true online. The parameter correction is essential. The exact linear-function result should not be assumed to apply unchanged to an arbitrary nonlinear neural-network predictor. Finally, feature scaling affects the overlap calculation, so the numerical behavior of one-hot features should not be generalized casually to every representation.

Business use

Dutch traces are relevant when teams study efficient incremental learning with linear representations. A useful experiment documents feature encoding, step size, discounting, trace decay and the full update rule. Small exact examples help establish correctness before interpreting performance results on larger simulated tasks.

Check your understanding

Why was the second A trace 1.576?

Answer: It combined decay 0.72 with adjusted addition 0.856.

Why was B added without an overlap reduction?

Answer: Its inner product with the previous trace was zero.

Why were no learned values reported?

Answer: Rewards and the complete parameter-update information were not supplied.

Related terms

Eligibility trace
True online TD(lambda)
Feature vector
Accumulating trace
Replacing trace
Online lambda-return
Linear function approximation

Learn next

Repeat the calculation with overlapping, non-one-hot features, then validate a complete short true online TD(lambda) episode.

True online TD(lambda)

Sources

Sutton and Barto, Reinforcement Learning: An Introduction, second edition, Sections 12.5–12.6 and Equation 12.11: https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf . The feature sequences and company scenario are original illustrations.

Previous: Durbin - Wu - Hausman test Next: Dyadic distribution