What is being projected?

The Bellman-updated value target is projected into the model's representable value space.

Can changing state weights change the solution?

Yes, as the example's weighted mean illustrates.

Does zero MSPBE prove optimal control?

No. The definition here evaluates a fixed policy; policy quality is a separate question.

AI Glossary / Reinforcement learning and robotics / Advanced

Mean squared projected Bellman error

Definition

Mean squared projected Bellman error, abbreviated MSPBE, measures the squared distance between an approximate value function and its Bellman target after that target has been projected into the function class the model can represent. In linear policy evaluation, it describes how closely the current estimate satisfies the projected Bellman equation associated with its features and state weighting.

What it means

A limited model cannot assign arbitrary values to every state. For example, two states with identical features may have to receive the same prediction even when their future rewards differ. Applying a Bellman backup can produce a target outside that model's representable space.

MSPBE first finds the closest representable version of the target, then compares it with the current estimate. This distinguishes inconsistency that the representation can express from target differences that the projection discards. A zero score means the estimate is a projected fixed point, not necessarily that it equals the true value function or satisfies every original Bellman equation.

How it works

Fix a policy, feature matrix Phi and nonnegative state-weighting matrix D. The approximate values are V equal to Phi theta. The Bellman operator T gives expected rewards plus discounted continuation values under the policy.

For linearly independent features under D, the weighted projection is Pi equal to Phi times the inverse of Phi-transpose D Phi, followed by Phi-transpose D. MSPBE is the D-weighted squared norm of V minus Pi T V.

Equivalently, define C = E[phi phi-transpose], A = E[phi (phi minus gamma phi_next)-transpose], and b = E[reward times phi]. Then MSPBE equals (b minus A theta)-transpose C-inverse (b minus A theta).

These expectations must correspond to the stated policy and weighting. If transitions come from another behavior policy, appropriate corrections or sampling assumptions are needed. GTD2 and TDC use auxiliary estimates to work with this objective incrementally. Their convergence claims require mathematical conditions; the objective's existence alone does not guarantee a particular implementation converges.

Three practical examples

1. A researcher tests whether a linear critic has approached its projected fixed point, while separately measuring its true prediction error in a known simulator.

2. A feature-design study identifies states whose Bellman targets differ but whose shared representation forces them to receive one value.

3. An off-policy prediction experiment checks that the state weights and importance corrections used in its diagnostic match the intended evaluation problem.

Think of it this way

Think of asking a small paint palette to reproduce a complex image. First approximate the desired colors using the available palette, then check whether the current painting matches that approximation. A perfect match to the limited palette's target can still differ visibly from the original image. MSPBE concerns that projected consistency, not unrestricted representational accuracy.

Worked example

Consider an original two-state episodic example. From A, the process immediately terminates with reward 0. From B, it immediately terminates with reward 4. There is no continuation value after either transition, whatever discount below one is chosen.

Use one constant feature, phi(A) equal to phi(B) equal to 1. The model must predict the same scalar theta in both states. Set the evaluation weights to 0.75 for A and 0.25 for B.

The Bellman target vector is (0, 4). Its closest constant prediction under these weights is the weighted mean: 0.75 times 0 plus 0.25 times 4, which is 1. The projected target is therefore (1, 1).

At theta equal to 1, the current values already equal that projection. MSPBE is zero. However, the original Bellman residuals, target minus value, are -1 at A and +3 at B. MSBE is 0.75 times 1 squared plus 0.25 times 3 squared, giving 3. The actual values are also (0, 4), so the weighted squared value error is 3 in this special terminal example.

At theta equal to 0, MSPBE is 1 because both projected differences are 1. MSBE is 4 because only B contributes a nonzero residual: 0.25 times 4 squared. Moving to theta equal to 1 removes projected inconsistency but cannot remove the representation's inability to distinguish the states.

The matrix form confirms the result: C is 1, A is 1 because successor features are zero, and b is 1. Hence MSPBE is (1 minus theta) squared. With equal state weights, b and the projected target become 2.

Illustrative case study

In a fictional project, Hazel Analytics evaluates a queue-management policy using compact state features. Its projected-error diagnostic approaches zero, yet individual predictions remain poor for two operationally different situations.

The team discovers that the representation maps both situations to identical features. It adds an informative distinction and repeats evaluation with held-out simulated episodes. The project report retains both projected consistency and prediction-quality measures. This prevents a mathematically successful fit within the old representation from being presented as accurate forecasting of every state.

Common misconceptions and mistakes

Zero MSPBE is not zero MSBE. Projection can remove components of the Bellman residual that lie outside the feature space.

MSPBE is also not simply the squared norm of the average TD update: the inverse feature second-moment matrix changes the geometry. An implementation must handle singular or poorly conditioned feature matrices appropriately.

Ordinary TD can target a projected fixed point without being gradient descent on MSPBE at each update. Conversely, minimizing an objective in theory does not establish stability for arbitrary nonlinear approximators, learning rates or off-policy datasets.

Business use

The concept helps teams distinguish an optimization problem from a representation problem. If projected error is small but prediction quality is poor, further training with unchanged features may not address the main issue. Evaluation should expose state coverage, feature choices and the practical cost of mistaken predictions. Report the diagnostic alongside task performance rather than using it as a stand-alone quality certificate.

Check your understanding

Why is the projected target 1 rather than 2?

Answer: A receives three times B's weight.

What prevents exact predictions in the example?

Answer: The constant feature forces the two states to share one value.

Related terms

Mean squared Bellman error
GTD2
projected Bellman equation
TDC
linear function approximation
feature covariance and TD fixed point

Learn next

Next, give A and B separate indicator features and repeat the calculation. Then study how the auxiliary vector in gradient TD algorithms estimates the feature-weighted residual correction.

Sources

Primary source: Richard S. Sutton and colleagues, Fast Gradient-Descent Methods for Temporal-Difference Learning with Linear Function Approximation, Sections 3 and 4: https://icml.cc/2009/papers/546.pdf

The constant-feature calculation and fictional case are original teaching material. The entry uses the standard linear, weighted orthogonal-projection definition.

Previous: Mean squared logarithmic error Next: Mean squared temporal-difference error