Does R-learning use a discount factor?

The stated average-reward update has no gamma term.

Can behavior be exploratory?

Yes; it is an off-policy control method.

Are its Q-values absolute accumulated rewards?

No. They are relative values associated with an average-reward formulation.

AI Glossary / Reinforcement learning and robotics / Advanced

R-learning

Definition

R-learning is an off-policy reinforcement-learning control method for continuing tasks with an average-reward objective. It learns relative action values together with an estimate of the long-run reward per time step. Its temporal-difference update subtracts that average reward instead of discounting future values, connecting immediate outcomes with their longer-term effect on sustained performance.

What it means

Some decision processes continue indefinitely: a service system admits requests, resources become available again, and new requests arrive. There may be no natural episode whose total reward should be maximized. Average-reward learning asks how much reward the policy produces per step over the long run.

Simply adding rewards forever can produce an infinite total. R-learning instead tracks an average reward rate and values describing relative advantages around that rate. These relative values are not ordinary discounted returns. Their common additive offset is not important for action comparisons or the corresponding TD differences.

How it works

Maintain an action-value table Q and an average-reward estimate rho. A behavior policy, such as an exploratory policy based on Q, chooses actions. After observing reward r and next state s_next, form delta as r minus rho plus the maximum next-state action value minus the current state-action value.

Update the selected Q entry by alpha times delta. In the textbook variant used here, test whether the selected action is greedy after that update. If so, update rho by beta times the previously computed delta. Otherwise leave rho unchanged. Other presentations use related update and gating conventions, so implementation details should be identified explicitly.

The maximum next-state value connects learning to a greedy target policy even when behavior explores. The selective average-reward update aims to avoid estimating merely the exploratory behavior's average reward.

The setting needs assumptions under which a meaningful long-run reward rate exists. A single state-independent rate may be inappropriate when different starting states lead to separate recurrent classes. Average-reward algorithms and their convergence conditions require specific analysis; resemblance to Q-learning does not supply a universal convergence guarantee.

Three practical examples

1. A continuing queue simulation compares admission policies by sustained reward per decision step.

2. A resource-allocation experiment examines whether accepting a modest immediate benefit prevents more valuable opportunities later.

3. A teaching model contrasts relative-value updates with discounted Q-learning to show how their objectives and reward baselines differ.

Think of it this way

Think of assessing a continuously operating process against its normal production rate. A temporary surplus or deficit matters because it changes performance relative to that background rate. The assessment is about sustained throughput over time, rather than repeatedly declaring that the process has ended and starting a fresh total.

Worked example

Consider an original numerical update using the stated textbook convention. At state S, action A has Q-value 3 and action B has Q-value 5. The current average-reward estimate rho is 1. Set alpha to 0.1 and beta to 0.05.

First consider an exploratory choice of A. It gives reward 4 and reaches a different state whose maximum action value is 6. The TD error, computed before updates, is 4 minus 1 plus 6 minus 3, or 6. The new Q(S, A) is 3 plus 0.1 times 6, or 3.6.

B still has value 5, so A is not greedy after the update. Under this variant, rho remains 1.

Now consider an independent alternative example starting from the original table, in which B is selected and the same reward and next-state maximum occur. Delta is 4 minus 1 plus 6 minus 5, or 4. Q(S, B) becomes 5.4. Since B remains greedy, rho becomes 1 plus 0.05 times 4, or 1.2. The average update uses the already computed delta; it is not recomputed from the updated Q entry.

Adding 100 to every Q entry would leave both TD errors unchanged because the common shift cancels between next and current values. That illustrates their relative-value character.

To distinguish the objective from immediate reward, compare two repeating three-step reward cycles: (9, 0, 0) averages 3 per step, while (4, 4, 4) averages 4. If a choice commits the system to either repeating cycle, the second has the better average despite its smaller first reward. This separate arithmetic illustration does not claim to solve the preceding update's environment.

Illustrative case study

In a fictional project, Cedar Operations builds a continuing service simulator. A policy that accepts every apparently valuable request performs poorly later because it leaves too little capacity for subsequent work.

The team studies average-reward control and reports reward per fixed-duration step over long evaluation runs. It checks whether results depend on initial conditions and separates the learned rho estimate from independently measured performance. It also verifies its greedy-action gate and update ordering, since changing those details produces a different algorithm.

Common misconceptions and mistakes

R-learning is not obtained merely by setting gamma to one in an otherwise unchanged discounted Q-learning implementation. Subtracting an average-reward estimate and learning relative values are central.

An average per decision is not automatically an average per unit of clock time. When actions take unequal durations, the task formulation needs appropriate treatment of time.

The learned rho is not necessarily a simple running mean of every observed reward. Its update is tied to the selected algorithm and target-policy logic. Do not claim universal convergence, especially under function approximation or poorly explored continuing processes.

Business use

The concept is relevant when sustained operational performance matters more than finite-episode totals. Teams should define the unit of time, resource constraints and evaluation horizon before selecting an average-reward method. Compare long-run outcomes and transient behavior: two policies with similar average reward can differ substantially during startup or disruption. Independent evaluation is essential because the internal average estimate is part of training, not an external performance measurement.

Check your understanding

Why does rho stay unchanged after choosing A?

Answer: The updated action is still nongreedy under the specified gate.

Why does adding a common constant to Q leave delta unchanged?

Answer: The constant cancels in the difference between next and current action values.

Related terms

Average reward
Q-learning
differential value
off-policy control
continuing task
gain and bias
and recurrent class

Learn next

Next, compare average reward per step with reward per elapsed time in a variable-duration task. Then examine alternative differential-control algorithms and their specific convergence conditions.

Average reward

Sources

Primary source: Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, author draft, Section 11.3 and Figure 11.2: https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf

The update calculations, reward cycles and fictional case are original teaching material. The example explicitly follows the draft's post-update greedy test and previously computed TD error convention.

Previous: R chart Next: R-precision