Are the higher-level states directly observed?

Usually they are hidden model variables, though some applications use supervision or constraints to guide their interpretation.

It can represent sequences composed of reusable subpatterns at different time scales.

Does the model discover one uniquely correct activity structure?

No. Different structures and parameters may explain similar observations, so interpretation requires evidence and comparison.

AI Glossary / Computer vision and multimodal AI / Advanced

Hierarchical hidden Markov model

Definition

A hierarchical hidden Markov model, or HHMM, is a probabilistic sequence model whose hidden states are organized into nested levels. Higher-level states govern sequences of lower-level states, allowing the model to represent activities or patterns that contain meaningful subactivities over different time scales.

What it means

An ordinary hidden Markov model describes a sequence through transitions among hidden states that generate observations. Some sequences have a richer structure: a high-level activity such as preparing a drink contains lower-level steps such as reaching, pouring and stirring. An HHMM represents this nesting explicitly. The hierarchy is part of the model, not simply a diagram drawn over an ordinary state list. The hidden states are inferred statistical explanations of observations. Their labels do not automatically correspond to real-world concepts unless the model design, data and evaluation support that interpretation.

How it works

In the classical hierarchical formulation, entering a higher-level state activates a lower-level process. That process transitions among substates and eventually terminates, returning control to its parent level. At the observation-generating level, states emit data according to their probability distributions. Transition and termination behavior determine how sequences unfold across the hierarchy.
Learning estimates model parameters from observed sequences, while inference can evaluate sequence likelihood or estimate hidden-state activity. The exact algorithms depend on the formulation and assumptions. A hierarchy can capture reusable substructure, but additional states and levels increase modeling choices and estimation demands. Poorly supported structure can overfit or produce states that are difficult to interpret.

Three practical examples

A video-analysis researcher models a broad activity as a sequence of smaller movement patterns extracted from frames.
A speech or handwriting study represents patterns with nested temporal structure rather than treating every observation as an unrelated event.
A workflow-analysis team studies recurring subprocedures within longer process logs, using the model to describe sequence patterns rather than asserting causation.

Think of it this way

Think of a play organized into acts, scenes and individual lines. An act controls which scenes occur, and scenes contain the finer sequence. The analogy is limited because an HHMM uses probabilities and inferred hidden states, and its learned hierarchy need not match a human-authored script or a unique interpretation of the sequence.

Worked example

Consider a simplified model in which a parent activity selects subactivity A with probability 0.6 or B with probability 0.4. Suppose A generates a particular two-observation sequence with probability 0.5 and then terminates, while B generates that sequence with probability 0.2 and then terminates. Under these stated assumptions, the sequence probability is 0.6×0.5+0.4×0.2=0.38.
The contribution from A is 0.30, so its posterior probability given this sequence is 0.30/0.38, approximately 0.789. This tiny example illustrates marginalizing over hidden alternatives and updating their probabilities. It omits the detailed internal transitions of a full HHMM and assumes the branch probabilities already include the relevant termination behavior.

Illustrative case study

Fictional example: Rowan Lab analyses short videos of a repeated assembly task. It proposes a hierarchy with broad stages and finer hand-motion states. On held-out clips, some workers perform steps in a different order, so the team revises the model rather than treating every unfamiliar sequence as an error. It compares inferred stages with independently annotated examples and reports uncertainty. The case illustrates structured sequence modeling, not proof that the hidden hierarchy captures workers' intentions or the causal organization of the task.

Common misconceptions and mistakes

A hierarchical model is not automatically more accurate than a flat one. Hidden states are not directly observed facts, and assigning meaningful names does not validate them. The hierarchy does not remove all Markov-style conditional assumptions; it organizes them differently. More levels can increase parameter and inference demands. An HHMM is distinct from a hierarchy of independent classifiers and from a generic recurrent neural network. Finally, sequence likelihood does not by itself establish that a workflow is desirable, safe or causally explained.

Business use

Businesses considering hierarchical sequence models should identify the temporal structure they expect and how it will be validated. Compare with simpler baselines and evaluate unfamiliar but legitimate sequences. Record assumptions about state duration, termination and observation features. If the model flags anomalies, define a review process that distinguishes unusual behavior from actual problems. Interpretability claims should be supported by agreement with meaningful reference annotations, not merely by a neat state diagram. Deployment monitoring is needed when process order or observation quality changes.

Check your understanding

Two hidden branches can generate the same observed sequence. Should its total probability use only the more likely branch?

Answer: Not when computing the full sequence probability. Contributions from compatible hidden alternatives are summed; choosing one branch answers a different inference question.

Related terms

Hidden Markov model
Probabilistic inference
latent state
sequence modelling
hierarchical model
state transition
temporal segmentation

Learn next

Learn ordinary hidden Markov models and the distinction between likelihood and hidden-state inference. Then trace a small parent-and-substate model, including termination, before adding multiple levels or interpreting learned activities.

Sources

Fine, Singer and Tishby, The Hierarchical Hidden Markov Model: Analysis and Applications (1998), primary paper: https://www.cs.princeton.edu/courses/archive/spr06/cos598C/papers/FineSingerTishby1998.pdf Primary-paper search excerpts consulted; full recursive inference and editorial review remain pending. The branch calculation is an original simplified probabilistic illustration.

Previous: Hierarchical decomposition Next: Hierarchical Language Model