Must the problem contain uncertainty?

No. Deterministic problems can still require reasoning over time.

A rule for selecting actions using the relevant available information.

Does a longer planning horizon always help?

No. It can reveal important consequences, but also increases computation and may compound model errors.

AI Glossary / Reinforcement learning and robotics / Intermediate

Sequential decision making

Definition

Sequential decision making concerns choosing actions over time when present choices influence future conditions, information, or opportunities. A decision maker considers both immediate consequences and what may happen after later decisions. Problems can be deterministic or uncertain, fully or partially observed, and finite or continuing. Reinforcement learning is one approach to some such problems; sequential decision making is the broader subject.

What it means

The best immediate action may not produce the best overall outcome. Spending time on maintenance can reduce output now but preserve future capacity. Gathering information can delay action while improving a later choice.
A sequence of decisions is not merely a list prepared in advance. Often the appropriate next action depends on what has happened. A policy describes how to choose actions from the available information, allowing behavior to adapt to different outcomes rather than following one fixed path.

How it works

Define the decision stages, available actions, relevant information, and outcomes being valued. Specify how actions affect subsequent conditions and whether uncertainties can be represented with a useful model.
Choose an objective, such as expected total reward, discounted return, or a constrained performance measure. The horizon and treatment of future outcomes matter. A short horizon can favor choices that look attractive immediately but have undesirable later consequences.
When a model is available, dynamic programming or planning can reason backward or forward through consequences. When suitable experience is available, reinforcement learning can estimate values or policies. Partial observation may require reasoning over beliefs about hidden conditions rather than treating the latest observation as complete state information.
Evaluate the policy under realistic sequences, not only isolated decisions. Check assumptions about transition dynamics, observations, and costs. Constraints, uncertainty, and consequences outside a scalar reward may need explicit treatment. A policy optimized for an inaccurate model or incomplete objective can systematically make the wrong trade-offs.

Three practical examples

A robot decides whether to continue a task or recharge, accounting for both current progress and the ability to finish later work.
An inventory system chooses replenishment amounts while considering demand, holding costs, and future stock availability.
An inspection system chooses which observation to obtain next, balancing measurement cost against the value of improved information.

Think of it this way

Think of playing a board game. A move that gains a small advantage now can leave a weak position next turn. Good play considers what choices remain after the move and how later actions depend on the opponent's response. Sequential decision making applies that structure to many engineering and business problems.

Worked example

Consider an invented two-stage maintenance decision. At the first stage, a machine can keep working or undergo service. Working gives an immediate reward of 6, but leaves the machine able to earn only 1 at the second stage. Service has an immediate reward of −2, representing its cost, and leaves the machine able to earn 12 at the second stage. Assume these outcomes are certain and there are no later rewards.
With an undiscounted total, working yields 6 + 1 = 7, while servicing yields −2 + 12 = 10. A decision based only on immediate reward would choose work, but the two-stage objective prefers service.
With discount factor 0.9, working yields 6 + 0.9 × 1 = 6.9. Servicing yields −2 + 0.9 × 12 = 8.8. Service still wins.
More generally, service is preferred when −2 + 12γ > 6 + γ, or 11γ > 8. Thus the threshold is γ > 8 / 11, approximately 0.7273. At equality the options tie. This sensitivity calculation shows how valuing future outcomes changes the decision. It is an invented decision model, not a maintenance recommendation, and assumes the stated rewards and future capabilities are accurate.

Illustrative case study

In this fictional case, Poplar Inspection operates a battery-powered research robot. An early controller always chooses the next nearby inspection because it offers immediate progress. During longer trials, the robot repeatedly reaches a point where it cannot complete the remaining route with its available energy.
The team revises the decision model to include future task opportunities and recharge choices. It evaluates complete missions with different starting conditions and compares results with a simple scheduled-recharge policy.
The report records completed work, interruptions, and model errors. It treats better average performance as evidence under the tested conditions, while continuing to examine difficult sequences and inaccurate energy estimates.

Common misconceptions and mistakes

Sequential decision making is not synonymous with reinforcement learning. A known model can support planning without learning through interaction, and many sequential problems are studied in control and operations research.
Another mistake is assuming a greedy choice is always wrong. It can be appropriate under particular conditions, but those conditions need justification. Conversely, looking further ahead does not guarantee improvement if the model is inaccurate.
A policy is also not the same as one planned action sequence. It may prescribe different actions after different observations. Ignoring this distinction can hide the value of adaptation.

Business use

Businesses encounter sequential decisions in scheduling, resource allocation, maintenance, logistics, and service operations. The practical value comes from representing important future consequences and adapting decisions to new information.
Start with a clear objective and a credible baseline. Examine sensitivity to the horizon, discounting, transition assumptions, and constraints. Measure outcomes across complete decision sequences and retain visibility into why the policy chooses costly or delayed-benefit actions.

Check your understanding

Which option maximizes the example's undiscounted total?

Answer: Service, with total reward 10.

Why does immediate-reward maximization choose differently?

Answer: It ignores the second-stage consequence.

What happens when γ equals 8 / 11?

Answer: The two modeled options have equal discounted return.

Related terms

Policy
Markov decision process
Dynamic programming
Reinforcement learning
partially observable Markov decision process
planning horizon
discounted return
value of information

Learn next

Learn next how states and policies are defined in a Markov decision process. Study Bellman equations, then examine how partial observation and model uncertainty change the decision problem.

Markov decision process

Sources

Sutton and Barto, Reinforcement Learning: An Introduction, second edition, Chapter 3 on finite Markov decision processes and returns, and Chapter 8 on planning and learning: https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf

Previous: Sequential analysis Next: Sequential exchangeability