AI Glossary / Reinforcement learning and robotics / Advanced

Emphatic temporal-difference learning

Definition

Emphatic temporal-difference learning is a family of value-prediction methods that changes the emphasis placed on updates to address instability in off-policy learning with function approximation. It combines importance sampling with a follow-on trace and a user-specified interest in states. Its foundational stability results concern linear value approximation under stated assumptions, not every possible learning system.

What it means

Off-policy prediction learns about a target policy using behavior generated by another policy. Correcting action probabilities alone does not necessarily make ordinary temporal-difference learning stable when values share parameters. The distribution of updates across states also matters.
Emphatic learning adjusts that distribution through a history-dependent emphasis. Interest expresses which states the user wants to value accurately; emphasis is the algorithm's resulting weighting and need not equal that immediate interest. The method remains a predictor of a specified policy's values rather than automatically being a policy-optimization algorithm.

How it works

Let rho_t be the target policy's probability of the observed action divided by its behavior-policy probability. In a constant-discount illustration, the follow-on trace is F_t = i_t + gamma rho_previous F_previous, initialized with F_0 = i_0. Here i_t is the nonnegative interest in the current state.
For trace parameter lambda, the emphasis is M_t = lambda i_t + (1-lambda) F_t. A linear eligibility trace is updated as z_t = rho_t [gamma lambda z_previous + M_t x_t], where x_t is the current feature vector. The weight update is alpha times the TD error times z_t.
These quantities serve different roles: importance ratios correct action sampling, the follow-on trace carries earlier interest forward, and the eligibility trace directs the parameter update. The target policy requires suitable behavior-policy coverage. The theoretical setting also imposes conditions on the process and feature representation; a formula alone does not establish convergence for an arbitrary dataset or step size.

Three practical examples

• Logged-policy evaluation: researchers predict a fixed target policy's returns from data collected by another policy with known action probabilities.
• Linear prediction experiments: a team compares ordinary importance-sampled TD with emphatic updates in a small model where the expected-update behavior can be analyzed.
• Interest-weighted prediction: analysts specify nonuniform interest in states and inspect how the resulting emphasis differs from the raw interest values along a trajectory.

Think of it this way

Imagine assigning importance not only to the current item in a sequence but also to how it follows earlier items that mattered. The resulting weight depends on the path and sampling corrections.

Worked example

Consider an original trace calculation with constant interest i = 1, discount gamma = 0.9 and lambda = 0. Use a single scalar feature x = 1 and start with F_0 = 1. Suppose the observed action-probability ratios at three consecutive steps are rho_0 = 2, rho_1 = 0.5 and rho_2 = 1.
Such ratios are possible, for example, if the observed actions have target/behavior probabilities 0.8/0.4, 0.2/0.4 and 0.4/0.4 at the respective states.
Because lambda is zero, emphasis equals the follow-on trace, and the eligibility trace is simply z_t = rho_t F_t x_t. At step 0, z_0 = 2 times 1 times 1 = 2.
At step 1, the follow-on trace uses the previous ratio: F_1 = 1 + 0.9 times 2 times 1 = 2.8. The current ratio is then used in eligibility, giving z_1 = 0.5 times 2.8 = 1.4. Confusing the current and previous ratios would change this calculation.
At step 2, F_2 = 1 + 0.9 times 0.5 times 2.8 = 2.26. Since rho_2 is 1, z_2 = 2.26. Constant interest therefore does not imply constant emphasis, and a current importance ratio of one does not erase the influence of earlier steps.
To see the update scale, consider a separate unit-error diagnostic: if the TD error at each inspected step were 1 and alpha were 0.1, the corresponding scalar weight increments would be 0.2, 0.14 and 0.226. Ordinary importance-sampled TD(0) with the same unit feature and error would instead yield 0.2, 0.05 and 0.1.
These are conditional update calculations. A complete training trajectory would require rewards, next-state features and changing predictions to determine the actual TD errors; the example does not claim they naturally remain one.
For another comparison, if every action ratio were one, the follow-on sequence from the same initialization would be 1, 1.9 and 2.71. Thus even an on-policy setting need not make this emphatic update identical to conventional TD(0).

Illustrative case study

Fictional company Alder Prediction evaluates a linear value estimator on simulator logs. Its analysts retain behavior action probabilities and define the target policy and state-interest function before training. They examine follow-on traces alongside prediction errors.
When a run produces unusually large updates, the team investigates both the errors and the weights multiplying them. It compares multiple runs and checks coverage rather than treating a theoretical expected-update result as a guarantee about every finite sample path. This fictional workflow reports no measured accuracy or stability gain.

Common misconceptions and mistakes

Emphasis is not the same as interest, reward or the current importance ratio. Earlier ratios and interests can affect it through the follow-on recursion. The example also shows why setting all current ratios to one does not automatically recover ordinary TD.
Another mistake is reading the linear expected-update stability result as a universal guarantee for neural networks, arbitrary step sizes or finite-data accuracy. Large or variable weights can produce high variance and difficult practical learning. Finally, stable learning does not mean the chosen features can exactly represent the target value function.

Business use

Emphatic TD is relevant to research on learning predictions from behavior that differs from the policy of interest. Reports should identify the target, behavior probabilities, coverage, features and interest function. They should distinguish mathematical stability conditions from observed prediction quality and computational behavior.

People also ask

Is emphasis always equal to interest?

No; it incorporates earlier information through the follow-on trace.

Does the original theory cover every neural network?

No; its central stability result concerns linear approximation.

Can a stable expected update still have noisy samples?

Yes; variance remains a practical concern.

Check your understanding

Why was F_1 equal to 2.8?

Answer: It used the previous ratio 2, discount 0.9 and initial trace 1.

Why was z_1 only 1.4?

Answer: The current ratio 0.5 scaled the emphasis.

Why did ratio one at step 2 not imply unit emphasis?

Answer: Earlier history remained in F_2.

Related terms

• Importance sampling
• Eligibility trace
• Off-policy learning
• Follow-on trace
• Interest function
• Linear function approximation
• Gradient-TD

Learn next

Compare the follow-on and eligibility recursions with a nonzero lambda, keeping the current and previous importance ratios distinct.

More in Reinforcement learning and robotics

• Dutch eligibility trace - A Dutch eligibility trace is a recursively updated feature-memory vector used in reinforcement-learning algorithms such as true online TD(lambda).
• Dyna-Q - Dyna-Q is a reinforcement-learning approach that integrates direct Q-learning, model learning and planning.
• Dyna-Q with exploration bonus (Dyna-Q+) - Dyna-Q with exploration bonus, usually called Dyna-Q+, extends Dyna-Q by encouraging the agent to revisit state-action pairs that have gone untried for a...
• Epsilon greedy policy - An epsilon greedy policy is a decision rule that usually chooses an action with the highest current estimated value but sometimes explores through random...
• Evolutionary robotics - Evolutionary robotics applies evolutionary search methods to robot controllers, physical designs, or both.
• Expected SARSA - Expected SARSA is a temporal-difference method that updates an action-value estimate using the policy-weighted average of the next state's action values.

Sources

Sutton, Mahmood and White, An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning: https://jmlr.org/papers/v17/14-488.html . Sutton and Barto, Section 12.11: https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf . The trace calculations and company scenario are original illustrations.

Previous: Emission probabilities Next: Empirical Bayes