Is the update only for linear models?

No; a differentiable nonlinear value model can also use it.

Does zero TD error prove correct values?

No. One transition can have zero error even when other transitions or expectations disagree.

Is it necessarily gradient descent on a fixed objective?

No. Its general learning dynamics need not have that interpretation.

AI Glossary / Reinforcement learning and robotics / Advanced

Semi-gradient temporal-difference learning

Definition

Semi-gradient temporal-difference learning updates a parameterized value estimate using a bootstrapped target while treating that target as constant during differentiation. The update differentiates the current prediction, but ignores the target's dependence on the same parameters. Semi-gradient describes this choice of derivative; it does not mean using half of a gradient or randomly omitting half the parameters.

What it means

A value function predicts future cumulative reward. With function approximation, many states share parameters, allowing experience at one state to influence predictions elsewhere. A one-step TD target combines the observed reward with a discounted prediction for the next state.

Because both predictions may use the same parameters, changing those parameters can move the prediction and its target simultaneously. Semi-gradient TD deliberately ignores the second dependency when forming the update direction.

This produces a simple incremental learning rule. It is generally not ordinary gradient descent on the mean squared Bellman error or on a fixed supervised-learning loss. Understanding that difference helps explain why supervised-learning intuition does not automatically provide a convergence guarantee.

How it works

For one-step state-value prediction, observe state S, reward R and next state S'. Compute delta equal to R plus gamma times v(S',theta), minus v(S,theta), using the current parameter vector theta. Then add alpha times delta times the gradient of v(S,theta) to theta.

For a linear approximation v(S,theta) equal to theta dotted with features phi(S), that gradient is simply phi(S). The update becomes theta plus alpha times delta times phi(S). A terminal transition has zero continuation value.

In automatic differentiation, the bootstrapped target is detached from the gradient calculation. This does not prevent the next-state prediction from changing indirectly when shared parameters change. It only prevents its derivative from contributing to this update.

On-policy linear prediction has convergence results under suitable assumptions about the data process, features and step sizes. Nonlinear approximators or off-policy sampling require separate analysis. In particular, the combination of bootstrapping, function approximation and off-policy learning can be unstable. The short rule alone does not specify a universally convergent algorithm.

Three practical examples

1. A simulator learns values from a compact set of engineered features instead of maintaining an independent number for every possible state.

2. A neural critic implementation detaches its target and audits the resulting parameter gradient against a hand calculation.

3. A research team compares semi-gradient TD with a residual-gradient method, checking both the update direction and the quantity each method seeks to reduce.

Think of it this way

Think of adjusting a forecast against a reference estimate that you temporarily pin in place. You move the current forecast toward that reference, even though both are produced by the same forecasting machinery. Once the machinery changes, the reference may move too. Pinning it during one calculation does not make it permanently fixed.

Worked example

Consider an original linear example with theta equal to (1, 2). The current-state features are (1, 2), and next-state features are (2, 1). The observed reward is 3, gamma is 0.5 and alpha is 0.1. The transition is not terminal.

The current prediction is 1 times 1 plus 2 times 2, or 5. The next-state prediction is 1 times 2 plus 2 times 1, or 4. The target is 3 plus 0.5 times 4, or 5. Thus delta is zero, and the semi-gradient update makes no change.

Now consider a separate transition with the same states and parameters but reward 4. Its target is 6 and delta is 1. The semi-gradient increment is 0.1 times 1 times (1, 2), giving new parameters (1.1, 2.2).

The new current prediction is 5.5. The new next-state prediction is 4.4, so recomputing the target gives 6.2 and the residual becomes 0.7. This differs from the residual 0.5 against the old, temporarily fixed target of 6.

For comparison, differentiating the complete sampled half-squared residual would use the direction phi(S) minus gamma times phi(S'), or (0, 1.5). That separate update would give parameters (1, 2.15), not (1.1, 2.2). The comparison identifies the omitted derivative; it does not establish which method learns better, nor does a sampled residual gradient generally equal the gradient of the squared expected Bellman error.

Illustrative case study

In a fictional project, Meadow Systems builds a critic for a warehouse simulation. A developer assumes that detaching the next-state target freezes its numerical value across updates. A small two-feature test shows that shared parameters still alter both state predictions.

The team adds checks comparing analytical updates with the implementation and records the target before and after each test step. It then evaluates the critic under the actual sampling policy and feature scaling used in training.

The review clarifies a gradient convention and the need for empirical stability checks. It is not evidence of a production improvement or a claim that one small example establishes convergence for the full simulator.

Common misconceptions and mistakes

Do not interpret semi-gradient as an unbiased estimate of the full gradient of the squared Bellman error. Do not assume that detaching a target prevents parameter sharing from changing its next numerical value. Do not transfer an on-policy linear convergence theorem to an arbitrary neural network or replay distribution.

Also distinguish a target network from target detachment. A separate slowly updated network changes how targets evolve; detachment specifies which derivatives are used in a particular calculation.

Business use

This concept helps teams review learning code and diagnose unstable critics. Specify the prediction model, sampling distribution, target construction and derivative convention together. Monitor value magnitudes and held-out prediction quality alongside training residuals. A decreasing detached-target loss within one optimization step is not by itself evidence that long-run policy evaluation is improving.

Check your understanding

What is the semi-gradient increment in the second example?

Answer: (0.1, 0.2).

Why does the recomputed target rise from 6 to 6.2?

Answer: The shared parameter update changes the next-state prediction.

What derivative is omitted?

Answer: The derivative of the bootstrapped next-state value inside the target.

Related terms

Residual-gradient reinforcement learning
Mean squared Bellman error
Target network
Gradient temporal-difference learning
Temporal-difference learning
Function approximation
Bootstrapping

Learn next

Study tabular TD and linear features first. Then compare detached-target derivatives with complete residual derivatives. Continue to projected Bellman equations and the assumptions needed for on-policy and off-policy stability results.

Sources

Brandfonbrener and Bruna, Geometric Insights into the Convergence of Nonlinear TD Learning, NeurIPS workshop 2019, sections 1–2: https://optrl2019.github.io/assets/accepted_papers/7.pdf. The paper states the semi-gradient update and examines conditional convergence of expected learning dynamics. The worked calculation and fictional case are original illustrations, not reproductions of its experiments.

Previous: Semi-amortized variational inference Next: Semi-partial correlation ( r 0(1.2) )