Does a scalar reward mean only one consideration matters?

No. Several considerations can contribute, but their combination specifies trade-offs.

Is every goal expressible by a reward on the current state?

No; the available representation and permitted reward dependence matter.

Does a higher reward prove a better real-world result?

Only to the extent that the reward and evaluation conditions faithfully capture the intended result.

AI Glossary / Reinforcement learning and robotics / Intermediate

Reward hypothesis

Definition

The reward hypothesis is the proposal that goals and purposes can be represented through maximizing the expected cumulative value of a scalar reward signal. It provides a conceptual foundation for reinforcement learning: an agent selects actions to improve a numerical objective over time. Whether a particular goal admits a particular reward representation depends on assumptions about preferences, information and time.

What it means

A scalar signal is one number received at a time, such as a score, benefit or cost expressed with the opposite sign. Cumulative reward combines signals across a sequence. Expected cumulative reward averages over uncertain outcomes rather than promising the best outcome on every run.

The hypothesis concerns how goals can be represented. It does not say that any convenient measurement captures the intended goal. A customer-service agent rewarded for closing tickets may close them prematurely if resolution quality is absent from its objective.

It also does not establish that people agree on one objective, that the objective is easy to measure, or that an algorithm can efficiently discover the best policy. Representation, specification and learning are separate questions.

How it works

Start by stating which complete outcomes or histories are preferred, including uncertain alternatives. Decide what information a reward rule can use and how consequences at different times are combined. Then examine whether optimizing that rule preserves the intended preferences.

A familiar discounted objective is the expected sum of reward at the first step, gamma times reward at the second, gamma squared times reward at the third, and so on. Episodic totals and continuing average-reward criteria capture different planning priorities. Selecting one is part of defining the problem.

Restrictions matter. A reward depending only on the current represented state and action may fail to express a goal that distinguishes histories merged into that state. Additional memory can change the representational possibilities.

Formal research relates reward representations to assumptions about preferences over uncertain histories and their temporal structure. These are conditional mathematical results, not an unconditional assertion that every conceivable human purpose fits every standard reinforcement-learning formulation.

Three practical examples

1. A warehouse team asks whether throughput rewards also represent its preferences about delays, damaged goods and repeated work.

2. A tutoring designer compares rewards for immediate correct answers with objectives that include later retention, making the time horizon explicit.

3. A researcher tests whether a navigation task requires memory of previous directions before attempting to encode it with a reward on current location alone.

Think of it this way

Think of designing the scoring rules for a competition. Players can optimize the score, but the rules determine which achievements count and how trade-offs work. A perfectly scored performance can still disappoint the organizer if the scoreboard omits something the organizer values. The existence of a scoreboard does not prove that its rules express the intended competition.

Worked example

Consider an original two-step, deterministic illustration. Policy A earns rewards 6 and 0. Policy B earns rewards 0 and 8. Both terminate after the second reward, with no further consequences included in this toy objective.

For gamma equal to 0.5, A has return 6 plus 0.5 times 0, or 6. B has return 0 plus 0.5 times 8, or 4. Maximizing discounted return selects A.

For gamma equal to 0.9, A still scores 6, while B scores 7.2. The preferred policy is now B. They tie when gamma equals 6 divided by 8, or 0.75. With an undiscounted two-step total, B scores 8 and wins.

The arithmetic demonstrates how the chosen objective encodes a timing preference. It does not establish that either discount factor reflects the designer's actual priorities. If the designer says B should always be preferred in this setting, the gamma equal to 0.5 specification contradicts that stated preference.

Because outcomes are deterministic, expected returns equal the listed returns. Introducing uncertainty would require probabilities and an explicit treatment of how the reward represents desirable and undesirable outcomes. A higher average monetary outcome, for example, need not mean higher expected utility under a different reward definition.

Illustrative case study

In a fictional project, Larch Learning builds a practice scheduler. Its first reward counts exercises completed during each session. Reviewers notice that the objective gives no credit for retention measured a week later.

The team writes preference examples before revising the reward: fewer exercises with sustained understanding may be preferable to more exercises quickly forgotten. It also specifies which delayed assessments can be observed and how missing assessments are handled.

The team keeps independent retention and student-experience measures alongside training reward. This is an illustrative design case, not a reported deployment or measured improvement.

Common misconceptions and mistakes

Confusing reward with immediate pleasure overlooks delayed and abstract outcomes. Confusing reward maximization with greediness overlooks the cumulative objective. Assuming a single number removes value disagreements merely hides the decisions used to construct that number.

Another mistake is treating a theorem about representation as proof that the reward is measurable or the optimal policy learnable. Finally, the reward hypothesis and the related reward-is-enough hypothesis are distinct claims; the latter concerns the emergence of intelligent abilities through reward maximization.

Business use

For businesses, the concept is useful during objective design and acceptance testing. Ask stakeholders to rank concrete outcome scenarios, then check whether the proposed reward produces those rankings. Document time horizons, excluded consequences and cases where preferences remain unresolved. Evaluate real outcomes separately from the training score, particularly when agents can influence how that score is recorded.

Check your understanding

In the worked example, what happens at gamma equal to 0.75?

Answer: Both policies score 6.

Why is that insufficient to validate the reward?

Answer: Numerical consistency does not establish agreement with the designer's preferences.

What should be checked when history matters?

Answer: Whether the state or reward mechanism retains the relevant information.

Related terms

Discount factor
Markov property
Reward function
Expected return
Reward specification
Multi-objective reinforcement learning
Reward hacking

Learn next

Study reward functions and return definitions first. Then explore preference representation, state design and reward misspecification. Read formal expressivity results with their assumptions about histories, lotteries and temporal preferences in view.

Sources

Bowling, Martin, Abel and Dabney, Settling the Reward Hypothesis, ICML 2023: https://proceedings.mlr.press/v202/bowling23a/bowling23a.pdf. The paper characterizes conditions rather than offering an unconditional affirmation.

Abel and colleagues, On the Expressivity of Markov Reward, primary research explanation, 2021: https://deepmind.google/blog/on-the-expressivity-of-markov-reward/. This concerns limitations with a fixed Markov reward representation. The numerical example and fictional case above are original teaching illustrations.

Previous: Reward Next: Reward shaping