Why are two vectors needed?

One represents values; the other estimates information required by the gradient correction.

Does TDC need a full covariance matrix?

Its incremental updates avoid explicitly maintaining one.

Does convergence mean exact true values?

Not when the feature class cannot represent them; the projected fixed point is the relevant target.

AI Glossary / Reinforcement learning and robotics / Advanced

Temporal-difference learning with gradient correction

Definition

Temporal-difference learning with gradient correction, usually called TDC, is a linear value-prediction algorithm that adds an auxiliary correction to the ordinary TD update. It belongs to the gradient temporal-difference family and is designed around the mean squared projected Bellman error. A second parameter vector tracks information needed for the correction, supporting convergence under conditions where conventional off-policy linear TD can be unstable.

What it means

Ordinary TD moves a value estimate toward reward plus an estimated continuation. With shared features and an unsuitable sampling distribution, those updates can amplify errors instead of settling. TDC retains the familiar TD component but adjusts its direction using an auxiliary estimate.

The objective involves projection because the chosen features may not represent every possible value function. The algorithm seeks consistency after projecting the Bellman target into the representable space. That is different from minimizing raw sampled TD errors or guaranteeing exact true values when the feature representation is inadequate.

TDC is a prediction method for evaluating a policy. Adding policy improvement, arbitrary neural features or a changing target policy requires further analysis; the original linear prediction theorem does not automatically cover those changes.

How it works

Let the value prediction be theta dotted with the current feature vector phi. Let phi' denote next-state features. Compute delta as reward plus gamma times theta dotted with phi', minus theta dotted with phi. Maintain an auxiliary vector w of the same dimension.

In the base transition-sampling formulation, update theta by adding alpha times delta times phi and subtracting alpha times gamma times phi' times the scalar phi dotted with w. Update w by adding beta times (delta minus phi dotted with w) times phi. Both updates use the old vectors and the same pre-update delta.

The auxiliary vector tracks a feature-covariance-adjusted expected TD update. This allows a gradient-related correction without explicitly storing and inverting a full feature covariance matrix. Time and storage scale linearly with feature count for dense vectors.

The original theorem uses assumptions including suitable moments, nonsingular matrices and two-timescale diminishing step sizes, with alpha/beta tending to zero. Its generalized sampling setup permits a different state distribution while conditional transitions follow the evaluated policy. If actions come from a different behavior policy, use the appropriate off-policy correction and coverage conditions; the unweighted base formula is not valid for arbitrary mismatched action data.

Three practical examples

1. A prediction researcher compares conventional linear TD and TDC under a state-sampling distribution that challenges ordinary TD stability.

2. An engineer audits an auxiliary-vector implementation to ensure the correction uses pre-update values rather than an inadvertently updated w.

3. A team with many features explores an incremental alternative to methods requiring a dense feature-by-feature matrix.

Think of it this way

Think of steering with a second instrument that estimates a systematic distortion in the main signal. The main TD term still points toward a newly formed target, while the auxiliary instrument changes how that correction is applied. The second instrument must itself be learned accurately; merely adding another vector does not make every update reliable.

Worked example

Consider an original two-feature calculation using the unweighted base formulation. The current features are phi equal to (1, 2), next features are phi' equal to (2, -1), theta is (0.5, 0.25), and w is (0.2, 0.1). Let reward be 2, gamma be 0.8, alpha be 0.1 and beta be 0.2.

The current prediction is 1 and the next prediction is 0.75. Thus delta equals 2 plus 0.8 times 0.75 minus 1, or 1.6. The auxiliary projection phi dotted with w is 0.4.

The ordinary TD increment is 0.1 times 1.6 times (1, 2), or (0.16, 0.32). The correction to subtract is 0.1 times 0.8 times 0.4 times (2, -1), or (0.064, -0.032). The net increment is therefore (0.096, 0.352), yielding theta equal to (0.596, 0.602).

The auxiliary increment is 0.2 times (1.6 minus 0.4) times (1, 2), or (0.24, 0.48). Its new value is (0.44, 0.58). This update uses the old delta and old w, not the newly calculated theta.

If w had instead started at zero, the first theta update would equal ordinary TD because the correction would be zero. Subsequent updates would generally differ as w learned. The fixed step sizes here illustrate one arithmetic step; they do not satisfy or demonstrate an asymptotic diminishing-step-size convergence schedule.

Illustrative case study

In a fictional project, Elm Forecasting estimates discounted machine-utilization rewards with a linear feature model. The team wants to understand why a replay-based predictor behaves differently from an on-policy baseline.

Researchers implement TDC in a small controlled model first. They verify the two-vector arithmetic, inspect feature conditioning and document whether the sampled transitions match the evaluation policy. They do not assume that changing the state-sampling distribution and changing the action policy are the same issue.

The comparison separates algorithmic correction from data-coverage and representation problems. This is an illustrative investigation, not a deployed predictor or measured improvement.

Common misconceptions and mistakes

Do not confuse TDC with GTD2: they share the auxiliary update but use different primary-vector updates. Do not describe TDC as full residual-gradient descent on a sampled squared TD error. Its objective is the projected Bellman error.

Do not transfer the theorem to arbitrary constant step sizes, nonlinear approximators or changing policies. Also avoid assuming that a zero initial correction means the auxiliary vector can be omitted permanently.

Business use

The method is relevant when linear policy evaluation needs an incremental algorithm with an explicit stability analysis. Practical evaluation should include feature scaling, conditioning, both step-size schedules and data provenance. Compare value accuracy and computation as well as the optimization objective. A representationally limited model can converge cleanly while still making inaccurate predictions for important states.

Check your understanding

What is the example's net theta increment?

Answer: (0.096, 0.352).

Why does the second correction component increase the net update?

Answer: Subtracting -0.032 adds 0.032.

When is the first correction zero?

Answer: When the auxiliary vector is initialized to zero.

Related terms

Gradient temporal-difference learning
GTD2
Mean squared projected Bellman error
Linear function approximation
Off-policy learning
Two-timescale stochastic approximation

Learn next

Study linear TD and projected Bellman equations first. Then compare TDC and GTD2 updates term by term. Read the sampling and step-size assumptions before interpreting either method's convergence statement.

Sources

Sutton and colleagues, Fast Gradient-Descent Methods for Temporal-Difference Learning with Linear Function Approximation, ICML 2009, sections 4 and 6: https://icml.cc/2009/papers/546.pdf. The base update and original TDC convergence conditions were checked directly. The numerical step and fictional project are original illustrations.

Previous: Temporal-difference error Next: Temporal expressions