Must computation run forever?

No. Algorithms can approximate values to a tolerance or solve suitable recursive equations.

Is infinite horizon always preferable?

No. A fixed project deadline can make finite-horizon reasoning more suitable.

Does a larger gamma guarantee a better policy?

No. It changes the objective and can increase sensitivity to uncertain distant outcomes.

AI Glossary / Graphs and symbolic AI / Advanced

Infinite horizon

Definition

An infinite horizon is a sequential decision setting with no fixed final time step. An agent evaluates actions in relation to an ongoing future, rather than a predetermined number of remaining decisions. The horizon describes the mathematical problem; it does not mean a computer must simulate infinitely many steps or that a real organization will exist forever.

What it means

A delivery service must consider tomorrow's vehicle availability while assigning today's work. If the model ends at an arbitrary date, it may undervalue maintenance whose benefits occur later. An infinite-horizon formulation removes that particular cutoff. However, it still needs a clear rule for valuing outcomes over time. Otherwise, different policies can both accumulate infinite total reward, making a direct comparison of their totals uninformative.
An uncertain stopping time can also be modeled explicitly. Terminology varies: some accounts distinguish tasks that continue indefinitely from tasks that stop when a specified condition occurs.

How it works

A common formulation uses a Markov decision process with states, actions, transition probabilities, and rewards. A policy specifies how actions are selected. Under discounted evaluation, the return is R0 + gamma R1 + gamma squared R2 + ..., where gamma is at least zero and less than one.
If each reward's absolute value is bounded by M, the discounted return's absolute value is at most M divided by (1 minus gamma). This makes infinitely many future terms mathematically manageable. Value functions and Bellman equations allow suitable algorithms to reason recursively instead of listing the entire future.
For finite state and action spaces, time-invariant transitions and rewards, and this discounted objective, an optimal stationary policy exists. Stationary means the policy does not change merely because the clock advances. It can still select different actions in different states. Other objectives, including long-run average reward, require their own analysis.

Three practical examples

An inventory controller repeatedly orders stock. An ongoing model can value stock carried into subsequent periods instead of treating everything left after an arbitrary final day as worthless. Storage costs, demand changes, and stockouts still need explicit treatment.
A robot alternates between productive work and recharging. Its policy accounts for repeated future cycles. A high immediate reward for working cannot by itself establish that continuing with an almost empty battery is the best ongoing strategy.
A maintenance planner chooses between operating and servicing a machine. Its model can capture future deterioration and recurring repair needs. If a known retirement date matters, a finite-horizon model or an explicit retirement state may be more appropriate.

Think of it this way

Think of tending a garden you expect to keep using. You care about the next harvest and the soil that supports later harvests. An infinite horizon provides a way to express that continuing responsibility. The discount factor says how strongly later modeled rewards influence today's choice; it does not certify that the future has been predicted accurately.

Worked example

Consider a fictional service with one initial choice between two irreversible plans. Rewards use invented benefit units, and all outcomes are deterministic.
Plan A gives 3 units immediately and 3 every period thereafter. Plan B costs 8 units immediately, represented by reward minus 8, and gives 5 units per period starting one period later. Neither plan allows switching later.
At gamma = 0.9, Plan A has value 3 divided by 0.1 = 30. Plan B has value minus 8 + 0.9 × (5 divided by 0.1) = 37. The later stream makes B preferable under this objective.
At gamma = 0.8, A has value 3 divided by 0.2 = 15. B has value minus 8 + 0.8 × (5 divided by 0.2) = 12. A is now preferable. The horizon remains infinite in both calculations; the weighting of later rewards changes.
For gamma below one, B exceeds A exactly when minus 8 + 13 gamma exceeds 3, or gamma exceeds 11/13, approximately 0.8462. They tie at that threshold.
Looking only at the immediate reward would favor A, since 3 exceeds minus 8. Setting gamma to one and comparing undiscounted infinite sums would not recover these finite values: both positive streams diverge.

Illustrative case study

Fictional repair cooperative Linden Service compares recurring maintenance policies. Its initial evaluation ends immediately after a costly servicing period, making maintenance look unattractive. The team builds an ongoing model and documents its discount factor, deterioration assumptions, and reward definition.
Reviewers then ask whether equipment replacement changes those assumptions. The team models replacement explicitly and tests several discount factors instead of reporting one policy as universally best. It also compares predictions with observed maintenance records. This case illustrates model design and review, not evidence that infinite-horizon modeling guarantees better operating results.

Common misconceptions and mistakes

Infinite horizon does not mean all future rewards receive equal weight. Discounting is one common evaluation rule.
A discount factor below one needs appropriate reward bounds or other convergence conditions; it is not a blanket guarantee for arbitrary unbounded rewards.
A stationary policy is not a constant action. It can respond to changes in the modeled state.
A mathematically optimal policy can still perform poorly when transitions, rewards, or relevant state variables are wrong.

Business use

Use this formulation when ongoing consequences matter and an arbitrary endpoint would distort choices. Record why the objective and time step fit the decision. A weekly discount factor and a yearly discount factor represent different weighting unless deliberately related. Compare policies under plausible model changes, and include real deadlines, replacement decisions, or stopping conditions when they affect operations.

Check your understanding

Which plan wins at gamma 0.9?

Answer: B, with value 37 versus 30.

At gamma 0.8?

Answer: A, with value 15 versus 12.

Can a stationary policy recharge in one state and work in another?

Answer: Yes; stationarity concerns time dependence, not identical actions everywhere.

Related terms

Learn next

Recalculate the two plans at gamma 0.85. Explain why the ranking changes near the threshold. Then add a known retirement date and identify which future rewards should disappear from the model.

Sources

MIT, Introduction to Machine Learning, Markov decision processes: https://introml.mit.edu/notes/dev/mdp.html Poole and Mackworth, Artificial Intelligence lecture 12.3, horizons and discounting: https://artint.info/3e/slides/ch12/lect_12_3_h.pdf AIMA glossary, infinite horizon: https://github.com/aimacode/aima-glossary/blob/master/glossary.md

Previous: Infimum (or greatest lower bound) Next: Infix