AI Glossary / Reinforcement learning and robotics / Advanced

GTD2

Definition

GTD2 is a gradient temporal-difference algorithm for learning a linear value-function approximation using a primary parameter vector and an auxiliary vector. It is designed to minimize the mean squared projected Bellman error through inexpensive incremental updates. Under its stated sampling, feature and learning-rate assumptions, it supports stable policy evaluation in settings that include off-policy sampling.

What it means

A value estimator predicts future discounted reward under a fixed target policy. Linear approximation expresses that prediction as a weighted sum of features. When the sampling distribution and target dynamics differ, ordinary TD learning can become unstable even though each update looks locally reasonable.

GTD2 approaches the problem through an explicit projected-error objective. Its auxiliary vector tracks information needed to estimate the objective's gradient without storing and inverting a full feature matrix on every step. The number two distinguishes this method from the original GTD algorithm; it does not mean two-step returns or a two-action environment.

How it works

For the transition-sampling formulation used here, let phi be the current-state feature vector, phi-prime the next-state vector, R the reward and gamma the discount. Compute delta = R + gamma phi-prime-transpose theta minus phi-transpose theta.

Using old values of both parameter vectors, update theta by adding alpha times (phi minus gamma phi-prime) times (phi-transpose w). Update w by adding beta times (delta minus phi-transpose w) times phi. Alpha and beta are positive learning rates selected according to the intended convergence setting.

The auxiliary vector approximates C-inverse E[delta phi], where C = E[phi phi-transpose]. This supports a gradient direction for the projected Bellman objective. Updates require vector operations with cost linear in the feature count rather than an explicit matrix inverse.

The original sampled-transition formulation draws current states under a chosen distribution and transitions according to target dynamics. When transitions instead come from a different behavior policy, the appropriate importance-weighted GTD2 formulation must be used. The displayed equations should not be copied unchanged into arbitrary off-policy action data. Convergence also depends on coverage, regularity and suitable step-size conditions.

Three practical examples

1. A reinforcement-learning researcher evaluates a fixed policy using linear features under a state-sampling distribution different from the policy's stationary distribution.

2. A simulation team compares GTD2 with conventional TD on a small example where off-policy approximation can cause divergence.

3. An implementation audit checks that primary and auxiliary updates use the same pre-update parameter snapshot rather than accidentally mixing old and new values.

Think of it this way

Think of steering toward a destination while a second instrument estimates how the error surface is tilted. The primary vector changes the prediction; the auxiliary vector improves the steering information. The second instrument is not another destination, and its readings should not be interpreted as a second set of predicted returns.

Worked example

Take current features phi = (1,2), next features phi-prime = (2,1), reward one and discount 0.9. Let theta = (0.5, minus 0.25), w = (0.2,0.1), alpha = 0.1 and beta = 0.2. Assume the displayed transition-sampling formulation applies.

The current value is 1 times 0.5 + 2 times minus 0.25 = zero. The next value is 2 times 0.5 minus 0.25 = 0.75. Hence delta = 1 + 0.9 times 0.75 minus zero = 1.675.

The auxiliary prediction phi-transpose w is 0.2 + 0.2 = 0.4. The feature difference phi minus 0.9 phi-prime is (minus 0.8,1.1).

The theta increment is 0.1 times 0.4 times that difference, giving (minus 0.032,0.044). Thus theta-new = (0.468, minus 0.206).

The w increment is 0.2 times (1.675 minus 0.4) times (1,2), or (0.255,0.510). Therefore w-new = (0.455,0.610).

Both updates used the old w and theta. Substituting w-new into the theta calculation would implement a different update sequence. This is one numerical transition update; it does not demonstrate convergence, objective reduction on every sample or accurate value prediction for a complete environment.

Illustrative case study

Fictional case: Elm Simulation builds a linear policy evaluator from sampled transitions. Engineers implement GTD2 and initially see unexpectedly large parameter changes. A hand calculation reveals that their code updates w first and then uses the new w for theta, contrary to the specified simultaneous-update convention.

After correcting the implementation, they compare multiple learning-rate settings and independent runs against known values in a small simulator. The team then checks coverage and importance corrections before using behavior-policy data. Passing one arithmetic check is treated as necessary but insufficient validation.

Common misconceptions and mistakes

GTD2 is not the original GTD objective with a renamed variable, and it is not ordinary TD plus the correction term used by TDC. These related algorithms have distinct update equations.

The auxiliary vector does not directly estimate the value function. If w starts at zero, the displayed first theta increment is zero even when delta is nonzero; w learns information that affects subsequent primary updates.

A convergence theorem does not imply that every stochastic update decreases the objective. It also does not remove approximation error from inadequate features or permit unsupported actions in off-policy evaluation. Constant learning rates and nonlinear extensions require their own analysis.

Business use

Business relevance is mainly in research systems that need stable, incremental value prediction from reused or deliberately sampled data. GTD2 adds auxiliary state and an additional learning-rate choice, so evaluate its practical cost and convergence speed alongside alternatives. Report both the optimized projected-error criterion and actual predictive accuracy where reference returns are available. Keep the data-generation assumptions explicit.

People also ask

Why are there two vectors?

One represents values and the other estimates gradient-related expectations.

Does GTD2 choose actions?

The basic algorithm evaluates a specified policy; action selection and policy improvement require additional machinery.

Does it need a full covariance matrix?

Its standard incremental updates avoid explicitly maintaining and inverting that matrix.

Check your understanding

What is the TD error in the example?

Answer: 1.675, calculated from the old value parameters.

Would using updated w in the theta step be equivalent?

Answer: No; it changes the specified simultaneous-update computation.

Related terms

• Gradient temporal-difference learning
• TDC
• projected Bellman error
• linear value approximation
• off-policy evaluation
• importance sampling and stochastic approximation

Learn next

Next, derive the expected GTD2 direction from the projected Bellman objective. Then compare its first few updates with TDC on identical transitions and study the sampling corrections needed for behavior-policy data.

More in Reinforcement learning and robotics

• Global costmap - A global costmap is a spatial cost representation used to support planning a route across an operating area.
• Gradient bandit algorithm - A gradient bandit algorithm learns action preferences and converts them into action-selection probabilities, typically with a softmax function.
• Greedy policy - A greedy policy chooses an action with the highest current estimated value at each decision point.
• Hierarchical reinforcement learning - Hierarchical reinforcement learning is an approach to learning sequential decisions at more than one level of abstraction.
• Human - robot collaboration - Human - robot collaboration is coordinated activity in which people and robots contribute to a shared task or goal.
• Human - robot interaction

Sources

Primary source: Richard S. Sutton and colleagues, Fast Gradient-Descent Methods for Temporal-Difference Learning with Linear Function Approximation, Equations 8 and 9 and the associated convergence assumptions: https://icml.cc/2009/papers/546.pdf

The vector calculation and fictional case are original teaching material, using the paper's explicit transition-sampling convention.

Previous: Grubbs's test Next: Guard (information security)