Can differential return be negative?

Yes, when rewards fall below the baseline over the relevant calculation.

Is average reward estimated in practice?

Often; estimation error can affect learning and interpretation.

Does subtracting a baseline guarantee convergence?

No. The process and limiting definition still matter.

AI Glossary / Reinforcement learning and robotics / Advanced

Differential return

Definition

Differential return is a reward-based quantity used in continuing, average-reward reinforcement learning. It measures accumulated rewards relative to a policy's long-run average reward, rather than accumulating raw rewards or discounting each later reward geometrically. It supports differential value functions that describe relative future advantages beyond the ongoing reward baseline.

What it means

A continuing process may generate rewards indefinitely. Adding all positive rewards can produce an infinite total, even when the process has a meaningful average reward per step. Subtracting the policy's average reward focuses attention on departures from that ongoing rate.
The subtraction is performed for every step. It is not a one-time deduction from the final total, and it is not the difference between two consecutive rewards. Differential return also differs from discounted return: the average-reward formulation changes the objective and the corresponding value equations, rather than merely choosing a discount factor very close to one.

How it works

The usual expression is G_t = sum over future steps of (R - r_bar_pi), where r_bar_pi is the policy's long-run average reward. For a finite prefix of H steps, this is the ordinary reward total minus H times the baseline.
Care is required with an infinite sequence. Centering rewards does not automatically make every realized infinite series converge. Formal differential values use suitable expected-value limits and assumptions about the continuing process; in some settings, a more careful limiting definition is needed. A finite centered sum should therefore be identified as a prefix calculation rather than silently treated as a complete infinite return.
Differential temporal-difference learning uses errors of the form reward minus estimated average reward plus next differential value minus current differential value. There is no discount multiplier in this basic average-reward equation. Adding the same constant to all differential values leaves their differences unchanged, so a reference convention may be used when interpreting absolute value levels.

Three practical examples

Queueing simulation: a learner compares service decisions by how they affect long-run performance and relative future rewards, rather than stopping after an arbitrary short episode.
Continuing resource allocation: researchers use an average-reward formulation when a repeated process has no natural terminal event.
Learning diagnostics: a team tracks both its estimated average reward and its differential values. A change in one should not automatically be interpreted as the same change in the other.

Think of it this way

Imagine measuring a machine's output relative to its normal long-run rate. A temporarily productive period contributes a positive deviation, while a slow period contributes a negative one. You are tracking performance above or below the baseline, not counting the machine's entire lifetime output. The baseline and the time interval must both be clear.

Worked example

Consider an original deterministic teaching sequence of rewards: 5, 1, 4, 2, followed by reward 2 forever. Its long-run average reward is 2 because the finite initial variation has no effect on the limiting average.
For the first four steps, subtract 2 from each reward. The centered rewards are 3, -1, 2 and 0. Their sum is 4. Equivalently, the raw reward total is 5 + 1 + 4 + 2 = 12, and four steps at the baseline account for 4 times 2 = 8. The difference is 12 - 8 = 4.
Every later centered reward is 2 - 2 = 0. Consequently, the full differential sum in this specially constructed sequence is also 4. Its convergence is straightforward because only finitely many terms are nonzero. This example does not establish convergence for arbitrary stochastic reward sequences.
Now suppose the baseline is incorrectly set to 1.5. The first four centered rewards sum to 12 - 4 times 1.5 = 6. Every later step adds 2 - 1.5 = 0.5. The centered sum then grows without bound. A small persistent baseline error can accumulate over a long horizon rather than remaining a small one-time discrepancy.
Finally, consider a separate differential TD calculation with observed reward 5, estimated average reward 2, current differential value 4 and next differential value 3. The error is 5 - 2 + 3 - 4 = 2. If both value estimates are increased by 10, the error remains 5 - 2 + 13 - 14 = 2. This demonstrates the role of value differences. It is a stipulated arithmetic example, not evidence that these value estimates are correct for the preceding sequence.

Illustrative case study

Fictional company Willow Scheduling studies a simulator that runs continuously. Its analysts define the reward per decision step and examine whether an average-reward objective matches the task. They track baseline estimates separately from the relative values used to choose actions.
When centered totals drift, they check baseline accuracy and the assumptions behind their calculation before concluding that the policy has a growing advantage. They report finite-window results as finite-window results. This fictional workflow reports no measured outcome.

Common misconceptions and mistakes

Differential return is not simply discounted return with gamma set to one while leaving rewards unchanged. Its per-step baseline subtraction is essential. It is also not a numerical derivative or a comparison with the immediately previous reward.
A zero mean centered reward does not guarantee that every sample-path sum converges. Infinite sums and expected limiting values need appropriate definitions. Another error is comparing differential values computed under different policy baselines as though they were ordinary absolute lifetime returns. Finally, changing the reward interval changes the interpretation of an average reward per step, so timing conventions should remain consistent.

Business use

The concept is useful when investigating ongoing decision processes whose goals concern sustained performance. Teams should define what one step represents, what reward is measured and why a continuing objective fits the task. Average reward and relative transient effects answer different questions. Operational evaluation should retain both the long-run objective and the limitations of finite observed data.

Check your understanding

Why was the example baseline 2?

Answer: All but finitely many rewards were 2.

Why was the complete centered sum 4?

Answer: The first four terms summed to 4 and all later terms were zero.

Why did shifting both TD value estimates leave the error unchanged?

Answer: Only their difference entered the calculation.

Related terms

Differential value function
Temporal-difference error
Average-reward reinforcement learning
Continuing task
Bias function
Discounted return
Ergodicity

Learn next

Compare finite reward totals, average rewards and centered totals for the same sequence, then state the assumptions needed before taking an infinite-horizon limit.

Infinite horizon

Sources

Sutton and Barto, Reinforcement Learning: An Introduction, second edition, Section 10.3 and Exercise 10.7: https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf . The reward-sequence and TD calculations and company scenario are original illustrations.

Previous: Differential privacy Next: Differential validity