Does gamma determine how long an episode lasts?

No. Episode termination and discounting are separate design choices.

Can a finite task use gamma = 1?

Yes, when its return is well-defined.

Is gamma a probability?

It can have a continuation-probability interpretation in particular models, but need not represent a real measured probability.

AI Glossary / Graphs and symbolic AI / Advanced

Discount factor

Definition

A discount factor is a number used to reduce the weight of later rewards relative to earlier rewards in a sequential decision problem. Usually written gamma, it determines how much each additional step discounts a reward. It helps define the return an agent maximizes.

What it means

A small discount factor places more emphasis on near-term rewards. A value closer to 1 preserves more of the weight of delayed rewards. This changes the objective being optimized, so it can change which action is preferred even when the possible outcomes remain identical.
Discounting is distinct from uncertainty. A reward can be delayed but certain, or immediate but uncertain. Transition probabilities describe uncertain outcomes, while discounting describes how their timing affects the objective. Some models connect discounting to a continuation probability, but that interpretation requires explicit assumptions.

How it works

Using a convention in which the next reward has weight 1, discounted return is the next reward plus gamma times the following reward, plus gamma squared times the third reward, and so on. The exponent counts how many additional steps separate a reward from the first one considered.
For the standard nonnegative setting, gamma lies between 0 and 1. At 0, only the next reward contributes. At 1, rewards are added without discounting. A fixed finite episode can use an undiscounted sum, but an infinite sequence of positive rewards may diverge.
If reward magnitudes are bounded by M and gamma is strictly below 1, the absolute infinite discounted return is at most M divided by (1 minus gamma). This condition concerns reward sums, not a universal guarantee that a learning algorithm converges or behaves appropriately.

Three practical examples

Robot planning: A robot compares a nearby modest reward with a larger reward requiring several additional moves. The factor affects the timing tradeoff, while obstacles and action uncertainty belong in the environment model.
Maintenance decisions: A controller may receive immediate production rewards while a maintenance interruption improves later performance. Too little weight on later consequences can make an attractive short-term policy look preferable under a poorly chosen objective.
Service optimization: A simulation rewards both immediate task completion and later customer outcomes. Changing gamma alters their relative importance. The team must check that the decision-step duration and reward timing match the intended evaluation period.

Think of it this way

Imagine reading a sequence of reward amounts through progressively darker panes of glass. Each extra step adds another pane with the same transparency. Increasing gamma makes distant amounts easier to see, but it does not determine whether those amounts will actually occur or whether the reward measure represents what matters.

Worked example

Consider two deterministic choices with three reward opportunities and no rewards afterward. Choice A produces rewards 6, 0, 0. Choice B produces 0, 0, 10. The first reward has weight 1, the second gamma, and the third gamma squared.
With gamma = 0.5, A has return 6. B has return 0 + 0.5 times 0 + 0.25 times 10 = 2.5. The discounted objective prefers A.
With gamma = 0.9, A still has return 6. B has return 0.81 times 10 = 8.1, so the preference reverses. At gamma = 1, B has return 10 and A has return 6. Both finite sums remain finite.
The choices tie when 10 times gamma squared equals 6, giving gamma = square root of 0.6, approximately 0.775. Above this threshold B is preferred; below it A is preferred. This assumes certain, complete rewards.
Now suppose B's final reward is 10 with probability 0.5 and zero otherwise. Its expected discounted return at gamma = 0.9 becomes 0.81 times 5 = 4.05. A is preferred again. Probability and discounting perform separate operations.
Finally, consider a different reward stream that pays 1 forever. With gamma = 0.9 its return is 1/(1 - 0.9) = 10. With gamma = 1, its partial sums keep growing without bound. Equal weighting of future rewards does not by itself make an infinite horizon mathematically finite.

Illustrative case study

Fictional Maple Control develops a simulated maintenance agent using one decision step per hour. Later, the team changes the simulator to ten-minute steps but leaves the same discount factor per step.
Reviewers notice that a reward one hour away is now discounted six times rather than once. With gamma = 0.9 per step, its relative weight changes from 0.9 to 0.9 raised to the sixth power, approximately 0.531. The timing preference has changed even though the calendar delay has not.
To preserve that particular hourly discount relationship, the team considers a ten-minute factor of 0.9 raised to the power 1/6, approximately 0.9826. It separately reviews how intermediate rewards and state transitions changed. Adjusting the factor alone does not prove the two simulators define equivalent tasks.

Common misconceptions and mistakes

“Gamma is the learning rate.” The learning rate controls parameter updates; gamma weights delayed rewards.
“A larger factor always improves the policy.” It changes the objective and may expose different modeling or estimation difficulties.
“Gamma equal to 1 keeps every infinite sum finite.” The perpetual unit-reward example disproves this.
“Discounting makes later harms disappear.” It only changes their mathematical weight; omitted consequences remain omitted.
“The same factor means the same time preference in every simulator.” Decision-step duration matters.

Business use

Document the factor together with reward definitions, timing conventions, and decision-step duration. Compare policies across plausible values and examine whether the resulting behavior fits the intended time horizon. Do not tune only for an attractive score under the very objective being changed.
Evaluate delayed benefits and costs explicitly. A discount factor cannot repair missing rewards, invalid constraints, or unrealistic transition assumptions. Where requirements must hold regardless of reward timing, represent and check those requirements directly.

Check your understanding

Why does B beat A at gamma = 0.9?

Answer: Its delayed reward has discounted value 8.1, exceeding 6.

Why does uncertainty reverse that decision in the modified example?

Answer: B's expected reward falls to 5 before discounting.

What changes when steps become shorter?

Answer: The same calendar delay includes more discount multiplications.

Related terms

Learn next

Plot the two deterministic returns as gamma varies. Then change reward delays and probabilities separately to see which assumptions drive each policy preference.

Sources

AIMA glossary, discount factor; infinite-horizon wording qualified in this entry: https://github.com/aimacode/aima-glossary/blob/master/glossary.md UC Berkeley CS188, finite horizons and discounting: https://inst.eecs.berkeley.edu/~cs188/textbook/mdp/markov-decision-processes.html Sutton and Barto, Reinforcement Learning, return definition and bounded rewards, indexed textbook text consulted: https://www.incompleteideas.net/book/bookdraft2018mar21.pdf

Previous: Discount Next: Discounted state-action occupancy measure