Can the correct value function have positive MSTDE?

Yes, when realized targets vary around their conditional mean.

When do MSTDE and MSBE coincide?

When the sampled target has zero conditional variance under the specified state distribution.

Is a frozen-target squared loss meaningless?

No. Its interpretation differs from jointly differentiating a bootstrap target that changes with the current parameters.

AI Glossary / Reinforcement learning and robotics / Advanced

Mean squared temporal-difference error

Definition

Mean squared temporal-difference error is the expected square of the error between a sampled reward-plus-continuation target and the current value estimate. It is often abbreviated MSTDE or described as mean squared TD error. Unlike mean squared Bellman error, it squares each sampled discrepancy before averaging over possible rewards and successor states.

What it means

A temporal-difference error compares what a predictor currently says with what one observed transition suggests. The target combines the reward just received and a discounted estimate of what follows. Squaring these errors produces a familiar-looking loss that can be averaged over transitions or a dataset.

A large value can reflect incorrect predictions, unpredictable outcomes, or both. Even the true value function can produce nonzero TD errors because it predicts an expectation rather than every realized outcome. Consequently, driving this loss down is not always the same as improving expected-return predictions, especially when the continuation estimate is also adjustable.

How it works

For fixed-policy state-value prediction, define delta as r plus gamma times V(next state) minus V(current state). At genuine termination, set continuation value to zero. Mean squared TD error is E[delta squared] under a specified transition-sampling distribution.

Condition on the current state. The expected squared error equals the square of the expected error plus its conditional variance. Averaging over states therefore gives mean squared Bellman error plus the expected conditional variance of the sampled target.

That variance can depend on model parameters because the target contains estimated successor values. A full gradient of squared TD error differentiates through both the current value and the successor value. A semi-gradient treats the bootstrap target as fixed during the update. These are different procedures, even when their displayed numerical loss looks identical.

Deep-learning implementations may also use a separately frozen target network. State the network-update convention, terminal handling and sampling weights before comparing losses. Changing the replay distribution changes the average being measured; a lower number across two different distributions is not automatically a better estimator.

Three practical examples

1. A learning dashboard tracks average squared TD errors while also showing realized episode returns and prediction calibration.

2. A research experiment compares full residual-gradient updates with semi-gradient TD to examine their different optimization targets.

3. A simulator distinguishes noisy transitions from systematically misestimated states by comparing sampled residuals with expected Bellman residuals.

Think of it this way

Think of predicting the average outcome of a variable process. A correct average still differs from individual observations. If you can also change part of the quantity used as tomorrow's reference, reducing observed discrepancies may make the reference less variable rather than make today's prediction more accurate. Bootstrapped targets create precisely this extra possibility.

Worked example

Consider an original episodic process with one nonterminal state S. Every transition gives reward 1. With probability 0.5, the process returns to S; with probability 0.5, it terminates. Set gamma to 0.5 and let the predicted value of S be theta.

For a return to S, the TD error is 1 plus 0.5 theta minus theta, or 1 minus 0.5 theta. For termination, it is 1 minus theta. Mean squared TD error is therefore 0.5 times (1 minus 0.5 theta) squared plus 0.5 times (1 minus theta) squared.

Expanding gives 1 minus 1.5 theta plus 0.625 theta squared. Its derivative is -1.5 plus 1.25 theta, so its minimizing parameter is 1.2. At that value, the possible errors are 0.4 and -0.2. Their mean square is 0.5 times 0.16 plus 0.5 times 0.04, which is 0.10.

The true value instead satisfies V equal to 1 plus 0.5 times 0.5 V. Thus V is 4/3, approximately 1.3333. At this correct value, the two TD errors are +1/3 and -1/3, and their mean square is 1/9, approximately 0.1111. The accurate value has a slightly larger sampled squared-error objective.

To understand why, the expected TD error is 1 minus 0.75 theta. At theta equal to 1.2, its square is 0.01, while the conditional target variance is 0.0625 times theta squared, or 0.09. Together they give 0.10. At the true value, the expected error is zero but target variance is 1/9. Minimizing MSTDE has traded some systematic error for a reduction in target variability.

Illustrative case study

In a fictional project, Cedar Simulation evaluates prediction methods for repeated service attempts. One experiment reports lower squared TD loss after switching to a full-gradient residual objective.

The team checks expected returns using the known simulator and discovers that the value estimate has moved away from the correct answer. It documents how the learned continuation target affects the loss, then compares methods using both objective values and held-out prediction errors. The result becomes a lesson in choosing evaluation criteria, rather than a claim that every squared TD loss or every residual-gradient method is unsuitable.

Common misconceptions and mistakes

Nonzero TD loss does not by itself prove that a value function is wrong. Randomness can prevent zero sampled errors.

Conversely, a lower loss does not necessarily mean more accurate values when the target depends on the same parameters.

Do not describe ordinary semi-gradient TD as exact gradient descent on this full objective. Also distinguish an empirical minibatch average from its population expectation. Finite samples, changing target networks and prioritized sampling can all affect interpretation.

Business use

For business applications, squared TD error is useful as a diagnostic but needs context. Ask whether a change improves actual sequential decisions, whether the measured data distribution stayed comparable, and whether targets were held fixed during optimization. Pair training losses with task outcomes and separate validation. This helps avoid selecting a system mainly because its internal target has become easier to predict.

Check your understanding

What is the true value in the example?

Answer: Four-thirds, obtained from the expected Bellman equation.

Why does theta equal to 1.2 achieve lower MSTDE?

Answer: It reduces target variability enough to outweigh its small expected Bellman error.

Related terms

Temporal-difference error
Mean squared Bellman error
residual gradient
semi-gradient learning
bootstrapping
target network and conditional variance

Learn next

Next, change the probability of returning to S and derive both minimizing parameters. Then compare full-gradient and semi-gradient updates on exactly the same sampled transition.

Sources

Primary source: Richard S. Sutton and colleagues, Fast Gradient-Descent Methods for Temporal-Difference Learning with Linear Function Approximation, discussion of Bellman and sampled TD objectives: https://icml.cc/2009/papers/546.pdf

The one-state stopping process, derivation and fictional case are original teaching material. The example differentiates the complete sampled TD objective, including its parameter-dependent continuation target.

Previous: Mean squared projected Bellman error Next: Mean time to failure