AI Glossary / Reinforcement learning and robotics / Advanced

Differential value function

Definition

A differential value function describes expected future reward relative to a policy's long-run average reward in a continuing task. It captures how favorable a starting state or action is beyond that ongoing baseline. Unlike an ordinary discounted value, it is interpreted through average-reward equations and an appropriate reference or limiting convention.

What it means

Two starting states can lead to the same long-run reward rate while offering different short-term prospects. One may provide an immediate benefit before both settle into the same ongoing pattern. Differential values describe such relative effects; the average reward describes the sustained rate.
A differential value is therefore not the total reward earned over an infinite lifetime. The function assigns relative values under a specified policy; it is not simply a difference between successive states.

How it works

For a fixed policy in a suitable continuing setting, a differential value h satisfies an average-reward Bellman equation: h(s) = E[R - g + h(S_next) | S=s], where g is the policy's average reward per step. The expectation includes the policy's actions and the environment's transitions.
Adding a common constant to every h value leaves this equation unchanged. An implementation may choose a reference state with value zero or another normalization to fix the level. This freedom does not make the average reward arbitrary: g has a separate meaning determined by the policy and process.
The assumptions matter. In a communicating or otherwise suitably restricted process, a single policy average can provide the relevant baseline. In more general multichain settings, long-run rewards can depend on the starting class and require a more careful formulation. Formal differential values also should not be confused with an assumption that every realized infinite sum of centered random rewards converges.

Three practical examples

• Queueing research: two backlog states may have different transient costs even when the policy eventually reaches the same steady operating pattern.
• Continuing allocation: a learner distinguishes the value of temporarily available capacity from the long-run reward rate achieved by its allocation policy.
• Policy diagnostics: analysts examine differential values and the estimated average reward separately, checking whether a change reflects a better sustained rate, a different transient effect or merely a changed reference level.

Think of it this way

Imagine two starting positions on a repeating route. One begins beside a useful stop, while the other reaches that stop later. Both eventually follow the same routine at the same average pace. A relative starting advantage is meaningful even though neither position changes the long-run routine itself.

Worked example

Consider a two-state continuing process. From either state, the next state is A with probability one-half and B with probability one-half. The reward depends on the current state: it is 3 in A and 1 in B. There is only one action.
The stationary state probabilities are one-half each. The average reward is therefore g = 0.5 times 3 + 0.5 times 1 = 2. After the first transition, the next-state distribution is already this same mixture regardless of the starting state.
Choose the normalization h(A) = 1 and h(B) = -1, whose stationary average is zero. At A, the Bellman right-hand side is 3 - 2 + 0.5 times 1 + 0.5 times (-1) = 1. At B, it is 1 - 2 + 0.5 times 1 + 0.5 times (-1) = -1. Both equations are satisfied.
Starting in A gives an expected centered reward of 1 on the first step. Every later step has expected centered reward zero. Starting in B gives -1 first and zero in expectation thereafter. Thus the limiting expected centered sums give 1 and -1 under this convention. This statement concerns expectations; it does not claim that individual random infinite sums converge.
Now add 1 to both values. The alternative values are h(A) = 2 and h(B) = 0. At A, the equation becomes 3 - 2 + 0.5 times 2 + 0.5 times 0 = 2. At B, it becomes 1 - 2 + 0.5 times 2 + 0.5 times 0 = 0. The reference level changed, but the difference h(A)-h(B) remains 2 and the average reward remains 2.
Finally, if the current state is A and the sampled next state is B, the differential TD error using the zero-mean values is 3 - 2 + (-1) - 1 = -1. If the next state is A, it is 3 - 2 + 1 - 1 = 1. Their average is zero, as required by the exact Bellman equation, despite nonzero sample errors. These are stipulated calculations, not learned results.

Illustrative case study

Fictional company Birch Queues evaluates a continuing scheduling simulator. Its team anchors one reference state's differential value at zero and logs the estimated average reward separately. This supports comparisons across checkpoints.
When all reported differential values shift by nearly the same amount, the analysts inspect normalization before concluding that every state became better. They also evaluate long-run reward directly. This fictional workflow reports no results.

Common misconceptions and mistakes

A large absolute differential value is not meaningful without its normalization. Shifting the reference level does not change the relative value differences or the average-reward Bellman equation.
Another mistake is assuming that the highest differential value necessarily identifies a policy with the highest long-run reward. The policy baseline itself must be considered. Differential values are also not discounted values with the discount factor casually removed. The continuing-task assumptions, baseline subtraction and limiting definition remain essential. Finally, exact values can still produce noisy temporal-difference errors on individual transitions.

Business use

Differential values can help analyze transient advantages in ongoing systems while keeping sustained reward as a separate objective. Reports should state the policy, reward interval, average-reward baseline and normalization. This distinguishes reference shifts, transient conditions and sustained improvements.

People also ask

Why can all values be shifted by a constant?

The constant appears on both sides of the Bellman equation and cancels.

Does that also permit changing the average reward freely?

No. The average reward is a separate property of the policy and process.

Can exact differential values yield nonzero sample TD errors?

Yes, because individual transitions can be random.

Check your understanding

Why was the average reward 2?

Answer: The stationary distribution assigns equal weight to rewards 3 and 1.

Why did both normalizations describe the same relative values?

Answer: Each preserved the A-to-B difference of 2.

Why did sample TD errors average to zero?

Answer: The values satisfied the expected Bellman equations.

Related terms

• Differential return
• Average reward
• Bellman equation
• Stationary distribution
• Bias function
• Continuing task
• Relative value iteration

Learn next

Solve a small average-reward Bellman system with one reference value fixed, then verify both the average reward and relative value differences.

• Average reward

More in Reinforcement learning and robotics

• Curse of horizon in off-policy evaluation - The curse of horizon in off-policy evaluation refers to the difficulty of estimating a policy's performance from another policy's data as decision sequences...
• DAgger - DAgger, short for Dataset Aggregation, is an iterative imitation-learning algorithm.
• Deep deterministic policy gradient - Deep deterministic policy gradient, or DDPG, is an off-policy actor-critic reinforcement-learning algorithm for continuous action spaces.
• Discounted state-action occupancy measure - A discounted state-action occupancy measure describes how much a policy visits each state-action pair over time, with later visits receiving less weight.
• Dominating policy - A dominating policy, in the Markov decision-process sense used here, is a policy whose expected return is at least as high as that of every alternative...
• Double deep Q-network - A double deep Q-network, usually called Double DQN, is a reinforcement-learning method that separates choosing a promising next action from estimating that...

Sources

Sutton and Barto, Reinforcement Learning: An Introduction, second edition, Section 10.3 and Exercise 10.7: https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf . The two-state calculations and company scenario are original illustrations.

Previous: Differential validity Next: Differentially private stochastic gradient descent