Can the residual be negative?

Yes, under a signed convention; norms and squared summaries are nonnegative.

Can sample errors persist when the residual is zero?

Yes, as the random-reward example shows.

Does a small residual always mean a small value error?

The relationship depends on assumptions, discounting and the norm or coverage used.

AI Glossary / Reinforcement learning and robotics / Advanced

Bellman residual

Definition

A Bellman residual measures the discrepancy between a proposed value function and the result of applying a Bellman update to it. For a fixed policy, one common sign convention is residual = T_pi V - V, where T_pi averages immediate reward plus discounted next-state value under that policy. A zero residual at every state means the value function satisfies that policy's Bellman equation.

What it means

A value estimate should be consistent with the rewards and future values it implies. If the estimate says a state is worth 8, but the expected one-step reward plus discounted continuation is 8.2, the residual is 0.2 under the stated convention.
This is a consistency check, not directly the difference between the estimate and the true value. That difference can be larger because an error propagates through future visits. The residual also depends on the operator: evaluating a fixed policy and applying an optimality operator that selects the best action are different tasks. A report should specify which equation is being checked.

How it works

For a fixed policy, calculate the expected quantity R + gamma V(S_next), conditional on the current state and actions drawn from that policy, then subtract V(s). This expectation defines the residual at state s. Repeating the calculation across states gives a residual vector.
A temporal-difference error from one observed transition is a sample quantity, R + gamma V(S_next) - V(s). Its conditional expectation is the fixed-policy Bellman residual when the sampling and value conventions match. One noisy transition need not equal that expectation.
An overall summary can use a norm or a weighted mean of squared residuals. Squaring each sampled temporal-difference error and averaging is generally different from squaring the expected error: the former includes conditional variability. Projected Bellman error is another distinct quantity used with function approximation. Terminology sometimes varies, so an implementation should state its formula, weighting and sign rather than relying on the label alone.

Three practical examples

Policy evaluation: a team checks whether estimated state values are consistent with a known simulation model and a fixed policy.
Planning diagnostics: an iterative solver monitors Bellman discrepancies while updating values. The stopping criterion must match the operator and error measure being used.
Learning analysis: researchers compare residual estimates with prediction accuracy and policy performance. A small measured discrepancy on sampled states does not establish accuracy everywhere the system might operate.

Think of it this way

Imagine a budget whose opening balance should match the expected income and expenses implied by its own assumptions. A mismatch reveals internal inconsistency. Correcting one line may affect later lines, so the local mismatch need not equal the total error in the budget. The analogy concerns consistency, not a guarantee that the assumptions describe reality.

Worked example

Consider one continuing state with one action. Every transition returns to that state. The reward is 0 or 2 with equal probability, so its expectation is 1. Set gamma = 0.9 and suppose the current value estimate is V = 8.
The Bellman update is T V = 1 + 0.9 times 8 = 8.2. The residual T V - V is therefore 0.2. The true expected discounted value solves V_true = 1 + 0.9 V_true, giving V_true = 10. The value error is 10 - 8 = 2, ten times the residual in this example.
Now examine individual transitions. When reward is 0, the temporal-difference error is 0 + 7.2 - 8 = -0.8. When reward is 2, it is 2 + 7.2 - 8 = 1.2. Their expected value is 0.5 times (-0.8) + 0.5 times 1.2 = 0.2, matching the residual.
The squared residual is 0.2 squared = 0.04. The mean squared temporal-difference error is instead 0.5 times 0.64 + 0.5 times 1.44 = 1.04. The difference, 1, is the variance of the sampled error here. Treating 1.04 as the squared Bellman residual would mix inconsistency with reward noise.
If the value estimate becomes the exact value 10, the Bellman residual becomes zero. Individual temporal-difference errors remain -1 or 1 because the reward is still random. Their mean is zero while their mean square is 1. Perfect expected-value consistency therefore does not require every observed error to vanish.
They illustrate the distinction among residual, sample error and value error; they do not describe a trained system or prove that a particular learning method will converge.

Illustrative case study

Fictional company Willow Simulation reviews a value-prediction component. Its analysts initially monitor squared errors from individual transitions. They then separate the variability of those observations from the average discrepancy at each modeled state.
The team checks whether a declining metric represents improved consistency or merely a change in sampled situations. It also compares predicted values with independent return estimates. This fictional workflow reports no measured result; one diagnostic cannot capture overall decision quality.

Common misconceptions and mistakes

A Bellman residual is not automatically the realized reward prediction error from one transition. The expected and sampled quantities coincide only in special cases. Mean squared temporal-difference error and mean squared Bellman error should not be used interchangeably.
Another mistake is claiming that a zero residual for one policy proves that policy is optimal. It establishes consistency with that policy's equation, under the model and representation used. A small average residual can also hide large errors in lightly weighted states. Finally, function approximation may prevent exact consistency, and minimizing one error objective need not minimize every other notion of value or policy error.

Business use

Bellman residuals can help diagnose planning and value-learning systems. Reports should identify the policy or optimality operator, the states covered and the weighting scheme. Teams should interpret residuals alongside model accuracy and operational outcomes. Internal consistency with an inaccurate model can still yield poor decisions when the real environment behaves differently.

Check your understanding

Why was the example residual 0.2?

Answer: The expected backup was 8.2 while the estimate was 8.

Why was the mean squared sample error 1.04?

Answer: It included reward variability as well as squared mean discrepancy.

Why did exact value 10 still produce nonzero sample errors?

Answer: Individual rewards remained random.

Related terms

Learn next

Compute an expected backup and several sampled backups for the same state, then compare their means, variances and squared errors.

Sources

Sutton and Barto, Reinforcement Learning: An Introduction, second edition, Sections 11.4–11.5: https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf . The one-state calculations and company scenario are original illustrations.

Previous: Bellman pseudospectral method Next: Bellman update