Why is it called temporal-difference?

Learning uses differences between predictions associated with successive stages of experience, adjusted for intervening rewards in the standard RL form.

Does TD need a transition model?

Basic TD prediction can learn from sampled transitions without an explicit model of transition probabilities.

Is a TD target always the true return?

No. It usually includes an estimated future value, so it can differ from the return later observed.

AI Glossary / Graphs and symbolic AI / Intermediate

Temporal-difference

Definition

Temporal-difference, usually abbreviated TD, refers to a learning approach that updates a prediction using newly observed information and a later prediction. In reinforcement learning, a TD error commonly compares a state's current value estimate with an observed reward plus the discounted estimated value of the next state. This lets learning proceed incrementally without waiting for a complete episode's final outcome.

What it means

A value estimate predicts future accumulated reward under a specified policy. After one transition, the learner has a new reward and a new state from which to estimate what remains. Together, these provide a revised target for the earlier prediction.

The target partly depends on another estimate, a practice called bootstrapping. It is not simply the difference between two clock readings, nor merely subtracting successive raw observations. The relevant difference concerns predictions of related future outcomes. TD learning can be used for prediction and can form part of control methods such as Sarsa and Q-learning, whose targets differ.

How it works

For one-step state-value learning, write the error as δ = r + γV(s′) − V(s). Here s is the current state, s′ is the next state, r is the reward received on the transition, γ is the discount factor, and V is the current value estimate.

Tabular TD(0) updates the current state's estimate by V(s) ← V(s) + αδ, where α is a step size. For a true terminal state in an episodic problem, the future-value contribution is zero. A time-limit truncation is not automatically the same as true termination.

Repeat the process across experience generated under the relevant policy. A single update is a noisy adjustment, not a proof that the new estimate is correct. Learning behavior depends on state representation, sampling, step sizes, and whether data come from the policy being evaluated. Function approximation and off-policy learning introduce additional stability issues.

Three practical examples

1. Game evaluation: a learner updates its estimate of a position using the next position and any immediate reward, rather than waiting for the game's end after every update.

2. Simulated navigation: an agent learns expected future reward for locations from transitions through an environment, with the reward definition and policy made explicit.

3. Sequential prediction: a system revises a long-term prediction as intermediate evidence arrives. The application must define what is predicted and why the later estimate provides a useful learning target.

Think of it this way

Think of estimating the remaining effort in a multi-stage task. After completing one stage, you combine the effort just observed with a revised estimate of what remains, then reconsider your earlier estimate.

The analogy is limited because TD is a specified statistical update with a target, step size, and assumptions. A revised estimate can itself be wrong. Bootstrapping shares information efficiently but does not create an independent ground-truth label at every step.

Worked example

Suppose a toy process moves from state A to state B and then terminates. The initial estimates are V(A)=4 and V(B)=6. Set γ=0.9 and α=0.2. The transition A→B gives reward 1.

The TD target for A is 1+0.9×6=6.4. The error is 6.4−4=2.4, so the update gives V(A)=4+0.2×2.4=4.48. The estimate moves toward the target rather than jumping all the way to it.

Next, B transitions to a true terminal state with reward 2. Its target is 2+0=2. The error is 2−6=−4, giving V(B)=6+0.2×(−4)=5.2.

For comparison, the actual discounted return from A in this completed episode is 1+0.9×2=2.8. A Monte Carlo update using the original V(A)=4 and the same step size would produce 4+0.2×(2.8−4)=3.76. The TD update instead produced 4.48 because it used the earlier, overly high estimate of B. Subsequent experience can correct that estimate and propagate the correction. This example shows a difference in targets, not a universal ranking of the two methods.

Illustrative case study

Fictional case: StepPilot trains a value predictor in a small simulation. The team notices that estimates rise early even when completed episodes have modest rewards. Inspecting the TD targets reveals optimistic next-state values, rather than an arithmetic error in the update.

Engineers compare target components, episode returns, and learning curves over repeated runs. They also check terminal handling and distinguish it from a simulation time limit. The case illustrates debugging through the learning signal; it does not report a real deployment or promise convergence under arbitrary training conditions.

Common misconceptions and mistakes

A positive TD error does not mean the immediate reward was positive. It means the complete target exceeded the current estimate.

TD(0) does not wait for the entire return, unlike a basic episodic Monte Carlo update. Its bootstrapped target can therefore be biased by inaccurate successor estimates.

Do not equate all TD algorithms with Q-learning. State-value prediction, on-policy action-value learning, and off-policy control have different targets and assumptions.

Convergence statements need their conditions. Combining off-policy data, function approximation, and bootstrapping can produce instability; success in a tiny table does not establish stability for a neural implementation.

Business use

TD methods are relevant when decisions or predictions unfold sequentially and feedback arrives over time. They can spread learning across transitions and make use of experience before the final outcome is known.

Define rewards and episode boundaries carefully, and keep policy evaluation separate from claims about improved decisions. Monitor target values, errors, and actual returns. Evaluate learned behavior on appropriate held-out scenarios or controlled simulations. A decreasing training error is not enough if the reward misses the business objective or if estimates fail under a changed policy or environment.

Check your understanding

Why did V(A) increase even though the completed episode's return was below 4?

Answer: The TD target used V(B)=6 before B was corrected. That target was 6.4, larger than the original estimate of A.

Related terms

Reinforcement learning
Bellman equation
SARSA
Q-learning
Eligibility trace
TD error
bootstrapping
value function
Monte Carlo learning

Learn next

Study TD(0) and Monte Carlo prediction side by side. Repeat the worked transitions after B's estimate changes to 5.2 and calculate the next target for A. Then examine eligibility traces to see how TD ideas can update several earlier states from one transition.

Monte Carlo

Sources

Primary reference: Sutton, Mahmood, and White, “An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning,” https://arxiv.org/pdf/1503.04269, introduction and TD(0) preliminaries. These passages were read for the learning signal and stability scope. The two-state arithmetic and fictional case are original hand-checked examples.

Previous: Temporal data Next: Temporal-difference error