Must every level learn from scratch?

No. Some behaviors or high-level rules can be fixed, pretrained or manually specified while other components learn.

Are options the only hierarchical method?

No. They provide one influential framework for temporally extended actions, while other methods organize goals or value decomposition differently.

Does a hierarchy guarantee better sample efficiency?

No. Benefits depend on the task, skill design and learning procedure, and an unsuitable hierarchy can make learning harder.

AI Glossary / Reinforcement learning and robotics / Intermediate

Hierarchical reinforcement learning

Definition

Hierarchical reinforcement learning is an approach to learning sequential decisions at more than one level of abstraction. A higher-level policy can choose subgoals or extended behaviors, while lower-level policies select the detailed actions needed to carry them out. The hierarchy can organize decisions over different time scales.

What it means

A complicated task often contains reusable activities. Reaching a doorway, grasping an object or navigating to a charging point can each involve many primitive actions. A hierarchy lets a decision-maker reason about such activities without choosing every motor command at the same level.

The hierarchy may be designed by people, learned from experience or partly both. It does not require a human-like management structure. The central issue is how decisions and learning are organized, not the names given to the levels.

How it works

Define or learn higher-level choices and the lower-level behaviors they invoke. The higher level selects what to pursue, and the lower level acts until a termination condition or interruption rule applies. Learning must connect rewards and outcomes across these levels.

In the options framework, an extended action includes an initiation set, an internal policy and a termination rule. Its duration can vary, so value updates account for rewards accumulated during execution and discount the continuation value by the elapsed number of steps. Other hierarchical approaches organize subgoals or value functions differently. A useful hierarchy must preserve enough flexibility to solve the overall task.

Three practical examples

1. A simulated mobile robot chooses a destination region at a high level, while a lower-level controller selects individual movement commands.

2. A game agent chooses an activity such as obtaining a key, whose internal policy handles several moves before returning control.

3. A research manipulation system reuses a learned reaching behavior inside different tasks. Evaluation checks whether the skill's assumptions still hold for the new objects and starting conditions.

Think of it this way

Think of planning a trip by choosing a station and then following the detailed steps needed to reach it. You do not normally decide the whole trip one footstep at a time. A higher-level choice can simplify planning, but choosing a poor intermediate destination can still make the overall trip inefficient. Useful structure helps; structure alone does not guarantee good decisions.

Worked example

Consider an invented option that takes three primitive steps to reach a doorway. It produces rewards -1, -1 and 5. The discount factor per primitive step is 0.9. After the option ends, the best estimated continuation value is 4.

The discounted reward accumulated during the option is -1 + 0.9 × (-1) + 0.9^2 × 5 = -1 - 0.9 + 4.05 = 2.15.

The continuation contribution is 0.9^3 × 4 = 2.916. A semi-Markov-style target for this completed option is therefore 2.15 + 2.916 = 5.066.

Using only one discount factor on the continuation would give 2.15 + 0.9 × 4 = 5.75, incorrectly ignoring the option's three-step duration. If the option ended the entire episode, the continuation term would instead be zero.

This calculation assumes the stated rewards, timing and discount convention. It illustrates temporal abstraction in an option-value target, not a complete hierarchical learning algorithm or evidence that the doorway option is the best available behavior.

Illustrative case study

Fictional case: Alder Automation builds a simulated fetch-and-deliver task. A high-level policy selects between reaching a shelf, acquiring an item and moving to a destination. Lower-level policies execute the corresponding movements.

The team finds that a grasping skill sometimes reports completion without securing the item, causing the high-level policy to proceed incorrectly. Engineers improve the success signal and evaluate complete tasks, not just each skill in isolation. They also compare against a flat baseline under the same training budget. This invented case reports no measured efficiency gain or real robotic deployment.

Common misconceptions and mistakes

A deeper hierarchy is not automatically better. Too many levels can complicate credit assignment and training. Poorly chosen subgoals can exclude good solutions or reward local behavior that harms the overall objective.

Reusable skills do not necessarily transfer to every environment; their initiation conditions and assumptions matter. Nor is every workflow with substeps hierarchical reinforcement learning: the learning and decision structure must actually involve reinforcement-learning components. Finally, optimality within a fixed hierarchy may differ from unrestricted optimality over all primitive-action policies.

Business use

The approach can be relevant when a business task contains recurring activities and long sequences of decisions. Reusable skills may reduce duplicated learning or make decisions easier to inspect, but these are possibilities to test rather than automatic benefits.

Define each skill's scope, start conditions, completion signal and failure handling. Evaluate both individual skills and the end-to-end objective, including interruptions and unexpected states. Compare training cost and task outcomes with simpler alternatives. A hierarchy that looks intuitive to a designer may still be difficult for the learning process to use effectively.

Check your understanding

Why does the worked example multiply the continuation value by 0.9 cubed rather than 0.9?

Answer: Three primitive time steps pass before the option finishes. With discounting defined per primitive step, the future value must be discounted for all three steps, not treated as if the extended action lasted only one.

Related terms

Transfer learning
Temporal abstraction
option
subgoal
skill
semi-Markov decision process
policy hierarchy
credit assignment
intrinsic reward

Learn next

Learn ordinary reinforcement-learning returns before adding a hierarchy. Specify one extended action with clear initiation and termination rules, then calculate its duration-aware value target. Evaluate whether the abstraction helps solve complete tasks and whether its restrictions exclude useful primitive-action solutions.

Reinforcement learning

Sources

Sutton, Precup and Singh, Between MDPs and semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning, 1999: https://www.sciencedirect.com/science/article/pii/S0004370299000521

Bacon, Harb and Precup, The Option-Critic Architecture, learning option policies and termination conditions: https://arxiv.org/abs/1609.05140

The option-return calculation and Alder Automation case are original hypothetical illustrations.

Previous: Hierarchical navigable small world Next: Hierarchical (sequential) sums of squares