Does average reward require episodes?

No. It is particularly useful for continuing tasks.

Is it the same as greedy immediate reward?

No. Actions can affect future states and thus the long-run rate.

Can two policies have equal average reward but differ materially?

Yes. Transient rewards, variability, and constraints can still differ.

AI Glossary / Retrieval and knowledge systems / Intermediate

Average reward

Definition

Average reward is a reinforcement-learning objective that evaluates a policy by its long-run expected reward per time step. It divides expected cumulative reward over an increasingly long horizon by the number of steps and takes the limit when it exists. This continuing-task meaning differs from simply averaging the total scores of several completed episodes.

What it means

Some decision-making systems operate continuously rather than pursuing a single terminal outcome. For these systems, sustained performance can matter more than an undiscounted total that grows without bound. Average reward provides a rate with which to compare policies over long periods.

For example, a system might seek to maximize useful work completed per decision interval after accounting for operating costs. The reward must encode the intended objective, and the interval must have a clear interpretation.

Although the supplied worklist places this term among retrieval and knowledge systems, the definition here is the standard sequential-decision and reinforcement-learning concept. It is not an average relevance score, a mean customer rating, or a synonym for immediate reward.

How it works

Fix a policy and an initial state. For a horizon of T steps, compute the expected sum of rewards earned while following that policy, divide by T, and consider the limit as T grows. Finite stationary Markov decision processes provide a common setting for this definition.

When a policy has a suitable stationary state distribution, its reward rate can be calculated by weighting each state's expected immediate reward by the fraction of time spent there. If the induced process has multiple recurrent classes, the long-run rate can depend on the starting state. A single universal rate for a policy therefore requires additional conditions.

Average-reward learning methods often estimate both the reward rate and differential values. Differential values describe relative future advantage after accounting for the ongoing baseline rate. They help compare actions whose consequences extend beyond the next reward.

Do not obtain this objective merely by setting the discount factor to one in an ordinary infinite sum: that sum can diverge. Also distinguish reward per decision from reward per unit of clock time. If decisions take unequal durations, a time-based objective needs a corresponding model and denominator rather than an unqualified per-step average.

Three practical examples

A simulated service scheduler compares policies by sustained completed work minus operating costs per fixed time interval.
A maintenance-planning experiment evaluates long-run net production while accounting for recurring downtime.
A continuing resource-allocation agent optimizes a stable reward rate rather than repeatedly treating arbitrary reporting windows as terminal episodes.

Think of it this way

Think of comparing two machines by output per hour over extended operation. One may produce a large initial burst and then slow down; another may start modestly but maintain a higher rate. The long-run rate highlights sustained performance, while startup behavior still needs separate consideration.

Worked example

Consider an original deterministic continuing task with equal-duration steps. Policy A produces the repeating reward sequence 6, 0, 0. Each three-step cycle earns 6, so its long-run average reward is 6/3 = 2 per step.

Policy B produces 1, 4 repeatedly. Each two-step cycle earns 5, giving average reward 5/2 = 2.5. Over six steps, A earns 12 and B earns 15, which confirms the respective averages of 2 and 2.5 for complete cycles.

Now give policy A a one-time startup bonus of 100 before its ordinary sequence. Over 60 regular steps plus the bonus on the first step, its total is 220 and its measured average is 220/60, approximately 3.667. Policy B still averages 2.5 over those 60 steps.

Nevertheless, the bonus contributes only 100/T to A's average over T steps, which tends to zero. Its long-run average remains 2, below B's 2.5. This illustrates why a short evaluation can favor a different policy from the long-run objective. It also shows that a genuine startup benefit is ignored asymptotically, rather than being economically meaningless.

Illustrative case study

Fictional case: Meadow Operations tests two scheduling policies in a simulator. One clears an initial backlog quickly, while the other processes recurring arrivals more efficiently after the system settles.

The team reports both the startup performance and the sustained reward rate. It checks that time steps represent equal intervals and that recurring costs appear in the reward. It also evaluates queue lengths and service commitments rather than assuming one average captures every operational concern.

The fictional analysis supports choosing an objective suited to the operation's horizon. It does not claim that a higher simulated reward rate guarantees better real-world service or that transient behavior can safely be ignored.

Common misconceptions and mistakes

Common mistakes include confusing average reward per step with average episodic return, assuming the rate is always independent of the initial state, and treating a brief running average as the exact long-run value.

Another mistake is overlooking transient costs because they vanish in the asymptotic average. Policies with identical gain can behave very differently during startup. A nonstationary environment also complicates interpreting a historical average as future performance. Finally, optimizing a poorly specified reward more efficiently can worsen unmeasured outcomes rather than improve the underlying task.

Business use

Average reward can suit ongoing operations with a meaningful sustained-performance objective. It gives teams a way to compare long-run policies without selecting an arbitrary discount factor solely to make cumulative values finite.

Specify the time unit, reward components, evaluation horizon, and assumptions about stability. Report uncertainty and startup behavior alongside the rate. When deadlines, finite lifetimes, or strongly changing conditions dominate, a different objective or explicit constraints may better capture the business decision.

Check your understanding

Why does A's startup bonus not change its long-run average?

Answer: A fixed bonus divided by an increasingly long horizon tends to zero, while its repeating cycle continues to average 2 per step.

Related terms

Policy
Stationary distribution
R-learning
continuing task
gain
differential value
recurrent class
discounted return

Learn next

study differential value functions, compare gain and bias optimality, and distinguish continuing-task evaluation from episodic reporting metrics.

Sources

Sources: Dewanto and Gallagher, Examining Average and Discounted Reward Optimality Criteria in Reinforcement Learning (2021), https://arxiv.org/pdf/2107.01348; Dewanto and colleagues, Average-Reward Model-Free Reinforcement Learning: A Systematic Review and Literature Mapping, https://arxiv.org/pdf/2010.08920. These author manuscripts were consulted for objective and scope. The supplied AIMA glossary link could not be retrieved; the operational example is original.

Previous: Average precision at k Next: Average treatment effect