Is this a policy-gradient method?

No. Gradient TD usually optimizes a value-prediction objective, whereas policy gradients adjust action selection.

Why use auxiliary weights?

They estimate expectation-based quantities needed for efficient gradient updates.

Does convergence mean the true value is recovered?

Only under sufficient representation and other required conditions; otherwise it may be a projected approximation.

AI Glossary / Reinforcement learning and robotics / Advanced

Gradient temporal-difference learning

Definition

Gradient temporal-difference learning refers to methods that estimate value functions by following gradients of carefully defined temporal-difference objectives. Important members include GTD, GTD2 and temporal-difference learning with gradient correction. They were developed to address instability that can arise when ordinary bootstrapping methods combine off-policy data with function approximation, particularly in linear policy evaluation.

What it means

A value function predicts discounted future reward under a specified policy. Off-policy learning estimates that value using data generated under another policy or sampling distribution. Ordinary TD updates can be useful, but they do not generally constitute full gradient descent on a fixed scalar error objective.

Gradient-TD methods introduce an explicit objective and often an auxiliary weight vector to estimate quantities needed for its gradient. Their purpose is stable prediction under stated assumptions, not automatically finding a better policy. Different family members use different objectives or update arrangements, so the name should not be treated as one universal update equation.

How it works

Represent value linearly as v-theta(s) = phi(s)-transpose theta. A sampled transition provides a current feature vector, next feature vector and reward. The TD error is reward plus discounted next value minus current value.

GTD2 and TDC target the mean squared projected Bellman error, which measures the discrepancy after projecting the Bellman-updated value back into the feature space. The original GTD formulation uses a different objective based on the expected TD update. These quantities are distinct from simply averaging squared sampled TD errors.

For a common linear formulation, let A = E[phi(phi minus gamma phi-prime)-transpose], b = E[R phi] and C = E[phi phi-transpose]. With appropriate coverage and nonsingular C, the projected objective is (b minus A theta)-transpose C-inverse (b minus A theta).

An auxiliary estimate can track C-inverse times the expected TD update, avoiding repeated matrix inversion. Sampling conventions, importance corrections and learning rates must match the chosen method. Convergence results require conditions on features, coverage, moments and step sizes; they are not unconditional guarantees for arbitrary neural-network implementations.

Three practical examples

1. A policy-evaluation study predicts the return of a target policy using transitions collected under a different exploration policy.

2. A reinforcement-learning researcher compares ordinary TD with gradient-TD methods on a linear approximation problem known to expose off-policy instability.

3. A simulation team estimates several value predictions from a shared stream of observations, checking that the sampling distribution supports each target prediction.

Think of it this way

Think of adjusting a map using a measured error surface rather than following correction arrows that may not point downhill on any single surface. Gradient-TD methods define the surface carefully, but the surface itself still reflects the chosen features and sampling weights. Reaching its minimum does not guarantee that every location on the map is represented accurately.

Worked example

Consider one continuing state with feature phi = 1, deterministic reward two, discount gamma = 0.5 and a transition back to itself. Its true value is 2 + 0.5 times 2 + 0.25 times 2 and so on, totaling four. The approximation v-theta is simply theta.

Here C = 1, A = 1 times (1 minus 0.5) = 0.5 and b = 2. The projected Bellman objective is J(theta) = (2 minus 0.5 theta) squared. Because the single feature represents every scalar value in this toy model, projection introduces no additional restriction.

At theta zero, J is four. Its full derivative is minus (2 minus 0.5 theta), so the derivative at zero is minus two. An exact population-gradient step with learning rate 0.1 gives theta-new = 0.2. The new objective is (2 minus 0.1) squared = 3.61.

The objective is minimized at theta four, where the Bellman discrepancy vanishes. This calculation demonstrates the objective and its exact gradient. It is not a sampled GTD2 or TDC update trace and does not demonstrate off-policy instability, because the example has only one state and one transition pattern.

Illustrative case study

Fictional case: Pine Simulation estimates the value of an evaluation policy from a fixed exploration dataset. Ordinary linear TD behaves inconsistently for some feature choices, so analysts compare it with a gradient-TD implementation.

They verify the target-policy corrections, feature coverage and two learning-rate schedules before interpreting results. A more stable objective trace is encouraging, but the team also compares predicted returns with long simulated rollouts. Stability of a chosen projected objective is treated separately from accuracy of the resulting business prediction.

Common misconceptions and mistakes

Gradient TD is not merely ordinary TD implemented with automatic differentiation. The objective and the handling of the bootstrapped target matter.

Minimizing the average squared sampled TD error is not generally equivalent to minimizing the mean squared Bellman error or projected Bellman error. Random next-state variation can change these objectives differently.

The auxiliary weights are not a second independently learned value policy. They estimate gradient-related quantities. Also, linear convergence guarantees should not be transferred automatically to nonlinear function approximation, unsupported target actions or poorly chosen constant step sizes. An exact projected solution may still have substantial value error when features are inadequate.

Business use

The business relevance is reliable off-policy value prediction in research and simulation workflows. Teams should distinguish prediction from control, check data coverage and compare computational costs with alternative estimators. Report the target objective alongside actual return-prediction error where it can be measured. An algorithm's theoretical stability is useful only when its assumptions and numerical implementation match the application.

Check your understanding

Why is the toy optimum four?

Answer: It satisfies theta = 2 + 0.5 theta.

Why does the example not prove off-policy stability?

Answer: It contains no meaningful difference between behavior and target transition patterns.

Related terms

GTD2
Mean squared projected Bellman error
TDC
semi-gradient TD
off-policy learning
linear function approximation and policy evaluation

Learn next

Next, derive the matrices A, b and C for a small multi-state model. Then compare GTD2 and TDC update rules and examine how changing the sampling distribution changes the projection being optimized.

Sources

Primary source: Richard S. Sutton and colleagues, Fast Gradient-Descent Methods for Temporal-Difference Learning with Linear Function Approximation: https://icml.cc/2009/papers/546.pdf

The scalar objective calculation and fictional case are original teaching material. The explanation distinguishes the algorithm family from its individual members and their assumptions.

Previous: Gradient step Next: Graduated optimization