Does a full update change successor values?

It can, as the worked example shows.

Is double sampling necessary in deterministic transitions?

The stochastic conditional-expectation issue disappears for that deterministic calculation.

Is residual gradient the same as GTD2?

No. GTD2 targets a projected Bellman objective using an auxiliary estimate and a different update construction.

AI Glossary / Reinforcement learning and robotics / Advanced

Residual-gradient reinforcement learning

Definition

Residual-gradient reinforcement learning updates a value approximator by differentiating a squared Bellman-style residual through both the current prediction and its bootstrapped continuation prediction. This differs from semi-gradient temporal-difference learning, which treats the continuation target as fixed during an update. The exact objective depends on whether the residual uses expected transitions or individual sampled transitions.

What it means

A bootstrapped target contains another prediction made by the same model. If changing a parameter changes both sides of the comparison, a full gradient must account for both effects. Residual-gradient methods follow that logic explicitly.

This gives a direct optimization interpretation, but choosing the residual objective still matters. Reducing squared inconsistency need not minimize error relative to the true value function. In stochastic settings, squaring a sampled TD error includes variability that is absent when the conditional expectation is taken before squaring. These objectives should be named separately rather than bundled under one ambiguous loss label.

How it works

For a deterministic fixed-policy transition, define delta as reward plus gamma V_theta(next state) minus V_theta(current state). Let the sample loss be one-half delta squared. Its negative gradient is delta times [grad V_theta(current state) minus gamma grad V_theta(next state)].

A residual-gradient update adds a learning-rate multiple of that expression to theta. In a linear approximator, the value gradients are simply the feature vectors, so the direction is delta times current features minus discounted next features. A semi-gradient update retains only the current-feature term.

For mean squared Bellman error in a stochastic environment, the expectation over transition outcomes occurs inside the residual before squaring. Its gradient involves products of conditional expectations. Independent successor samples from the same conditioning state can address the resulting double-sampling problem. Using one sampled successor for both factors generally instead differentiates a mean squared sampled TD objective.

Control variants must also specify their policy or maximization backup and treatment of nondifferentiable ties. Gradient-based optimization can improve stability under stated conditions, but it does not give universal global optimality for arbitrary nonlinear models. The broader family of residual algorithms can mix direct and residual-gradient directions; the pure residual-gradient update is one particular choice.

Three practical examples

1. A researcher compares full-gradient and semi-gradient updates on a deterministic prediction problem to isolate the effect of differentiating the target.

2. A model-based experiment computes expected Bellman residuals directly rather than substituting noisy single-transition squares.

3. An implementation audit checks whether automatic differentiation follows the continuation branch, matching the intended objective rather than accidentally changing the algorithm.

Think of it this way

Think of comparing two measurements connected to the same adjustable mechanism. Turning a knob changes both measurements. A full gradient accounts for both changes; a semi-gradient temporarily treats one measurement as a fixed reference. Neither description alone decides which learning objective is more useful for the task.

Worked example

Consider an original deterministic transition from A to B with reward 1 and gamma equal to 0.5. Use separate tabular parameters theta_A and theta_B, initially 2 and 4. This example isolates the loss for that one transition; it does not specify all of B's future dynamics.

The TD residual is 1 plus 0.5 times 4 minus 2, which is 1. The one-half-squared loss is 0.5. The derivative with respect to theta_A is -1, while the derivative with respect to theta_B is 0.5. The second derivative component arises because B's value appears inside the target.

With learning rate 0.2, a full negative-gradient step increases theta_A to 2.2 and decreases theta_B to 3.9. The new residual is 1 plus 0.5 times 3.9 minus 2.2, or 0.75. The new loss is one-half times 0.75 squared, or 0.28125.

For comparison, a semi-gradient step from the same original values increases theta_A to 2.2 but leaves theta_B at 4. Its new residual is 0.8 and its loss is 0.32. The residual-gradient step reduces this particular sample loss more, partly by lowering the successor estimate.

That lowering is precisely why the comparison must not be interpreted as proof of better value prediction. If separate evidence requires B's value to remain near 4, changing it to make A's equation easier to satisfy creates a competing error elsewhere. A full training objective must include the relevant transitions and weighting.

Both calculations use deterministic data; stochastic transitions require separate analysis.

Illustrative case study

In a fictional project, Maple Prediction Lab tests value-learning methods on a known simulator. A residual-gradient model has a smoothly decreasing training objective but reaches useful prediction accuracy more slowly than a semi-gradient baseline.

The team examines both current-state and successor-state parameter changes. It also measures errors against simulator-derived values, finding that a low residual alone does not capture every relevant difference.

Common misconceptions and mistakes

Residual-gradient learning is not a residual neural-network architecture and does not mean merely logging TD residuals. The defining operation is differentiating the residual through its parameter-dependent target.

Do not equate sampled squared TD error with mean squared Bellman error when outcomes are stochastic. Their minima can differ.

An objective with a well-defined gradient does not make every learning rate stable or eliminate nonconvexity. Residual methods can also learn slowly or spend representation capacity reducing discrepancies that are not the best proxy for value accuracy. Conversely, their limitations do not justify assuming semi-gradient learning is always stable, particularly off-policy with function approximation.

Business use

The concept helps technical teams understand what a training loss actually optimizes and why gradient flow through targets matters. Compare objective reduction, prediction accuracy, computation and policy outcomes separately. If a system changes its own continuation estimates to reduce a residual, assess whether that behavior supports the intended task rather than only producing a more favorable internal metric.

Check your understanding

Why does theta_B decrease in the example?

Answer: Lowering the target helps reduce a positive residual, and the full gradient includes that route.

What is missing from the isolated loss comparison?

Answer: Other transitions that constrain B and the wider value function.

Related terms

Bellman residual
Mean squared Bellman error
Double sampling
mean squared TD error
semi-gradient learning
function approximation and GTD2

Learn next

Next, add a terminal transition from B and inspect the combined two-transition objective. Then compare deterministic residual minimization with the stochastic expectation-based formulation.

Sources

Primary source: Richard S. Sutton and colleagues, Fast Gradient-Descent Methods for Temporal-Difference Learning with Linear Function Approximation, discussion of residual-gradient methods and objective distinctions: https://icml.cc/2009/papers/546.pdf

The deterministic transition derivation and fictional case are original teaching material. The example uses a half-squared sample loss and explicitly separates it from stochastic mean squared Bellman error.

Previous: Residual flow Next: Residual Neural Network