AI Glossary / Reinforcement learning and robotics / Advanced
Retrace(lambda)
Definition
Retrace(lambda) is a multi-step off-policy reinforcement-learning method that combines target-policy temporal-difference errors using truncated importance-ratio trace coefficients. It limits how strongly behavior-policy trajectories propagate later corrections while preserving the target-policy value as the fixed point of its expected operator under the stated assumptions. Lambda controls the extent of multi-step credit propagation.
What it means
An agent may collect experience using one policy while learning the value of another. Simply summing rewards from that experience can estimate the wrong policy. Full importance weighting can correct the mismatch, but products of large probability ratios can make updates highly variable.
Retrace uses a different construction: it combines expected target-policy TD errors and limits the ratios that carry later errors backward. When behavior and target policies agree, the off-policy truncation does not shorten the traces beyond lambda. When they disagree, it reduces the influence of incompatible action sequences.
The word safe in the original paper concerns off-policy learning properties. It is not a guarantee of physical safety, legal compliance or harmless decisions in an application.
How it works
For each observed transition, compute delta as reward plus gamma times the next state's action value averaged under the target policy, minus the value of the observed current state-action pair. Use zero continuation after genuine termination.
For each later sampled action, compute its target probability pi divided by its behavior probability mu. Set its trace coefficient c to lambda times the smaller of one and that ratio. Thus c lies between zero and lambda.
Form the return by starting with the current Q estimate and adding the first TD error. Add each later TD error multiplied by its power of gamma and the product of intervening c coefficients. The first correction is not multiplied by an unnecessary ratio for the initial action, because the return is conditioned on that state-action pair.
Recorded behavior probabilities must correspond to the policy that actually generated each action. Data coverage still matters. The expected-operator contraction and tabular convergence results rely on their assumptions; they should not be promoted into an unconditional guarantee for arbitrary neural approximators or optimizer settings.
Three practical examples
1. A replay-based critic learns about a newer target policy from trajectories collected by older exploratory policies.
2. A research team compares trace lengths under full importance ratios, tree backup and Retrace using identical experience.
3. An implementation audit checks the indexing of ratios and ensures that the initial TD correction is not incorrectly attenuated.
Think of it this way
Think of passing a chain of evidence backward through a sequence of decisions. Each link asks how compatible that decision was with the policy being studied. Retrace weakens links that were relatively unlikely under the target policy, but does not amplify a link without bound merely because the behavior policy rarely chose it.
Worked example
Consider an original three-error return calculation with current Q equal to 5, gamma equal to 0.9 and lambda equal to 0.8. Suppose the already computed target-policy TD errors along the segment are 2, -1 and 3. These inputs are held fixed to isolate the return weighting.
For the action connecting the first and second errors, the target probability is 0.25 and the behavior probability is 0.5. The ratio is 0.5, so c1 is 0.8 times 0.5, or 0.4.
For the next connecting action, the target probability is 0.6 and the behavior probability is 0.2. The ratio is 3, but it is truncated to one. Therefore c2 is 0.8, not 2.4.
The Retrace target is 5 plus 2 plus 0.9 times 0.4 times -1 plus 0.9 squared times 0.4 times 0.8 times 3. The contributions are 5, 2, -0.36 and 0.7776, totaling 7.4176.
With learning rate 0.1, an illustrative tabular update moves the initial estimate to 5 plus 0.1 times (7.4176 minus 5), or 5.24176.
If the second ratio were left untruncated in the same lambda-weighted calculation, its coefficient would be 2.4 and the final contribution would rise to 2.3328. That comparison shows the amplification the truncation avoids.
If lambda were zero, every later correction would vanish, leaving Q plus the first TD error, or 7. The method would then use the one-step expected target rather than this multi-step correction.
Illustrative case study
In a fictional project, Aspen Simulation trains a critic using stored trajectories from several earlier policies. Large importance-ratio products occasionally dominate its updates.
The team evaluates Retrace and records the distribution of trace coefficients, effective propagation lengths and prediction errors on separate simulator runs. It discovers that some useful long trajectories are still heavily attenuated because behavior and target policies differ substantially. The report treats this as a tradeoff between using distant information and controlling variability, and verifies that behavior probabilities were saved accurately when the data were collected.
Common misconceptions and mistakes
Retrace does not simply clip a full-trajectory importance weight and multiply a raw return. Its target-policy TD errors and products of per-decision trace coefficients are essential to the construction.
Clipping does not mean the expected operator must have the wrong fixed point. Preserving that fixed point is different from claiming each finite sampled target is an unbiased Monte Carlo return.
Avoid equating a tabular contraction result with guaranteed stability under all function approximation. Also distinguish replay-sampling weights from behavior-to-target action ratios: they address different distribution changes and are not interchangeable.
Business use
Retrace is relevant when teams want to reuse sequential experience while evaluating or improving a policy that differs from data-collection behavior. Useful evaluation includes interaction savings, numerical stability, prediction accuracy and the amount of delayed information retained. Accurate policy-probability logging and coverage diagnostics are practical prerequisites for interpreting results. A stable critic still needs separate assessment of the policy's actual operational performance.
People also ask
What happens on-policy?
The action ratios are one, so the trace coefficients equal lambda.
Can a coefficient exceed lambda?
Not under the stated Retrace formula.
Does the method eliminate all sampling variability?
No. Rewards, state transitions and finite data remain sources of randomness.
Check your understanding
Why is c2 equal to 0.8 rather than 2.4?
Answer: The importance ratio 3 is truncated at one before multiplication by lambda.
Why is the first TD error added without c1?
Answer: The initial state-action pair is already the return's conditioning point.
Related terms
- Importance sampling
- Eligibility trace
- off-policy learning
- tree backup
- target policy
- experience replay and V-trace
Learn next
Next, calculate the same correction weights when target and behavior probabilities match. Then compare Retrace's action-value return with the state-value correction used by V-trace.
More in Reinforcement learning and robotics
- Relational reinforcement learning - Relational reinforcement learning combines reinforcement learning with representations of objects and the relationships between them.
- Replacing eligibility trace - A replacing eligibility trace is a short-term credit-assignment record that is reset to a fixed active value, usually one, whenever its state or feature is...
- Residual-gradient reinforcement learning - Residual-gradient reinforcement learning updates a value approximator by differentiating a squared Bellman-style residual through both the current...
- Reward - A reward is the numerical feedback signal that defines what a reinforcement learning agent is encouraged to achieve.
- Reward hypothesis - The reward hypothesis is the proposal that goals and purposes can be represented through maximizing the expected cumulative value of a scalar reward signal.
- Reward shaping - Reward shaping adds supplementary reward information to a reinforcement learning task to guide learning.
Work in finance? Get the free CFO AI Prompt Pack: 7 prompts for finance leaders.
Sources
Primary source: Remi Munos and colleagues, Safe and Efficient Off-Policy Reinforcement Learning, Sections 2 and 3: https://arxiv.org/pdf/1606.02647
The three-error calculation and fictional case are original teaching material. The example isolates finite-segment weighting and explicitly distinguishes it from an unbiased full-return claim or a general function-approximation guarantee.
Previous: Retention period Next: Retraining