Can a positive reward produce a negative TD error?

Yes, when the reward-plus-continuation target is below the prediction.

Does a zero TD error prove the value is correct?

No; one sample can agree accidentally, and an incomplete dataset can hide errors.

Is the same formula used by every TD algorithm?

The target changes across state-value prediction, SARSA, Q-learning and entropy-regularized methods.

AI Glossary / Reinforcement learning and robotics / Intermediate

Temporal-difference error

Definition

A temporal-difference error, often shortened to TD error and written delta, is the difference between a bootstrapped learning target and the current value estimate. For one-step state-value learning, it equals the observed reward plus the discounted estimated next-state value, minus the current state's estimated value. It provides a local correction signal for learning predictions of future reward.

What it means

The error compares two estimates made at successive points in a decision sequence. The first is the earlier prediction. The second combines newly observed reward with a prediction of what remains. This lets learning begin before the full sequence finishes.

A positive TD error means the sampled target exceeds the current estimate; a negative error means it falls below it. Neither sign means the immediate outcome was good or bad in absolute terms. A positive reward can still produce a negative error when the earlier prediction was too optimistic.

The target depends on the algorithm. SARSA uses the next action selected by its policy, while Q-learning uses the largest estimated next-state action value. Calling both quantities TD errors does not make their targets interchangeable.

How it works

For state-value TD(0), calculate delta as R plus gamma times V(S') minus V(S). Use gamma to express the chosen discounting convention. At genuine episode termination, the continuation value is zero, making delta equal to terminal reward minus the earlier prediction.

A tabular update adds alpha times delta to V(S). With function approximation, the error scales a parameter update associated with the current prediction. Eligibility-trace methods use it to adjust several recently visited states or state-action pairs.

Keep the target and prediction associated with the correct transition and parameter snapshot. Recomputing one term after changing the values can alter the intended update. Artificial truncation, such as a time limit, may require bootstrapping even when a data collector marks a rollout as finished.

In a stochastic environment, an individual TD error can be nonzero even when the value function is correct. Under the evaluated policy and correct values, the conditional expected state-value TD error is zero. Sampling variability is different from a systematic Bellman inconsistency.

Three practical examples

1. A navigation learner increases an earlier prediction after a transition produces a better reward-and-continuation target than expected.

2. A replay system uses absolute TD error as one possible priority signal, while recognizing that noisy experiences can repeatedly have large errors.

3. A critic diagnostic compares average errors across states to identify systematic underprediction or overprediction, rather than treating every surprising transition as a model defect.

Think of it this way

Think of revising a journey forecast after completing one leg. You combine the benefit or cost already observed with your updated expectation for the remaining journey, then compare that total with your earlier forecast. The discrepancy guides a correction. A single unexpectedly good leg can produce a surprise even if the earlier forecast was accurate on average.

Worked example

Consider an original state-value example. Before a transition, V(S) is 7 and V(S') is 5. The observed reward is 2, gamma is 0.8 and alpha is 0.25. The transition is nonterminal.

The target is 2 plus 0.8 times 5, or 6. The TD error is 6 minus 7, or -1. The updated V(S) is 7 plus 0.25 times -1, or 6.75. Although the immediate reward is positive, the error is negative because the full target falls below the earlier prediction.

Now consider a separate one-step terminal prediction problem. A state produces reward 0 or 4 with equal probability and then terminates. Its true expected return is 2. If V(S) equals 2, the TD errors are -2 and +2, each with probability one half.

The mean error is zero, but the mean squared error is 4. Thus zero expected TD error does not imply zero sampled squared TD error. For alpha equal to 0.1, one update after reward 4 would move the estimate to 2.2 despite starting at the true mean. That fluctuation is a sampling effect, not evidence that the true value was larger than 2.

Finally, if the first transition had genuinely terminated with reward 2, its target would be 2 and its error would be -5. The next-state estimate of 5 would no longer belong in the calculation. The examples are separate scenarios with explicitly stated assumptions.

Illustrative case study

In a fictional project, Birch Service models the long-run value of service-routing decisions. An analyst flags every large TD error as a faulty prediction. A review finds that some routes have variable outcomes even when their average predictions are calibrated.

The team examines signed averages, error distributions and outcome variability separately. It also discovers that time-limited simulation runs were incorrectly treated as true terminal events, removing continuation values from some targets.

This illustrative project distinguishes implementation errors from sampling variation; it does not establish a deployment or improved customer outcomes.

Common misconceptions and mistakes

Do not equate TD error with immediate reward. Do not equate it with the difference from a complete Monte Carlo return. Do not assume its absolute size is a reliable measure of business importance, model uncertainty or learning usefulness in every setting.

Another mistake is expecting every sampled error to vanish at correct values. Conversely, small errors on a narrow replay dataset do not establish accurate values in unvisited states or under a different policy.

Business use

TD errors are useful for implementation checks, critic monitoring and some data-selection rules. Report the target definition, reward scale and sampling policy with any error metric. Compare value-learning diagnostics with independent task outcomes. A model can become internally more consistent while optimizing an incomplete reward definition, so prediction consistency and business success should remain distinct evaluation questions.

Check your understanding

Answer: What is the first example's error? -1.

Why is the second example's mean squared error 4 at the true value?

Answer: Both possible errors have magnitude 2, reflecting outcome variability.

What changes at genuine termination?

Answer: The bootstrapped continuation is removed.

Related terms

SARSA
Q-learning
Eligibility trace
Prioritized experience replay
Temporal-difference learning
Bootstrapping
Bellman error

Learn next

Learn one-step value prediction and discounting first. Then compare sampled TD errors with expected Bellman errors and complete-return errors. Practice terminal and nonterminal calculations before interpreting training dashboards.

Sources

Sutton and Barto, Reinforcement Learning: An Introduction, author draft, including the state-action TD formulation in section 7.5: https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf.

Brandfonbrener and Bruna, Geometric Insights into the Convergence of Nonlinear TD Learning, section 2: https://optrl2019.github.io/assets/accepted_papers/7.pdf. The arithmetic and fictional case are original teaching illustrations.

Previous: Temporal-difference Next: Temporal-difference learning with gradient correction