Does memoryless mean the states are unrelated?

No. The next state can depend strongly on the current state; earlier history adds no further information under the model.

Must transitions stay constant over time?

No. That is the additional time-homogeneity assumption.

Does a stationary distribution mean the chain stops moving?

No. The overall probability distribution can stay fixed while individual trajectories continue changing states.

AI Glossary / Mathematics and optimization / Intermediate

Markov Chain

Definition

A Markov chain is a sequence of random states in which the conditional distribution of the next state depends on the past only through the current state. This is the Markov property. The states can still be strongly dependent over time; the property concerns what information is needed to predict the next step.

What it means

The current state acts as a summary of the relevant history under the model. If earlier information changes the next-state probabilities after the current state is known, the chosen state representation does not satisfy the Markov property for that process.
This article focuses on discrete-time, finite-state chains. A time-homogeneous chain uses the same transition probabilities at every step. The Markov property alone does not require time homogeneity; a model can allow those probabilities to change with time.
States should be defined carefully. A simple label such as active or inactive may omit useful history, while a richer state can include recent events or duration. Adding that information changes the model rather than proving that every convenient state definition was already Markov.

How it works

For a finite homogeneous chain, construct a transition matrix P with entry Pᵢⱼ equal to the probability of moving from state i to state j in one step. Under the row convention used here, entries are nonnegative and each row sums to one.
If pₜ is a row vector containing the current state probabilities, then pₜ₊₁=pₜP. Repeating the multiplication gives multi-step distributions. A particular state path has probability equal to its initial-state probability multiplied by the appropriate transition probabilities.
A stationary distribution π satisfies πP=π. Starting in that distribution leaves the distribution unchanged at each step, but it does not mean individual states stop changing. Nor does the existence of a stationary distribution alone guarantee convergence to it from every starting state; structural conditions such as irreducibility and aperiodicity matter.

Three practical examples

An operational model can describe movement between a small set of equipment conditions under a stated time interval.
A user-behavior model can approximate transitions between activity categories, while testing whether those categories summarize enough history.
A simulation can generate state sequences from a specified transition matrix to explore the consequences of the model's assumptions.

Think of it this way

Think of a board game in which the next move's probabilities depend on the square currently occupied. The analogy works only if that square contains all relevant information. If the next move also depends on a hidden card or how long the player has waited, the square alone is not a sufficient state.

Worked example

Consider an original homogeneous chain with two states, A and B. Its transition matrix has rows (0.8,0.2) and (0.3,0.7). From A, the chain stays in A with probability 0.8; from B, it returns to A with probability 0.3.
Start in A with certainty, so p₀=(1,0). After one step, p₁=(0.8,0.2). After two steps, the probability of A is 0.8×0.8+0.2×0.3=0.70, and the probability of B is 0.8×0.2+0.2×0.7=0.30. The two possible intermediate states are both included.
The particular path A→B→A has probability 0.2×0.3=0.06. This is only one path ending in A after two steps; A→A→A contributes another 0.64. Their sum gives the same 0.70.
For a stationary distribution π=(a,1−a), the first-coordinate equation is a=0.8a+0.3(1−a). Therefore 0.5a=0.3 and a=0.6, giving π=(0.6,0.4). Checking the second coordinate gives 0.6×0.2+0.4×0.7=0.4.
Now contrast a different chain that always switches states, with rows (0,1) and (1,0). It has stationary distribution (0.5,0.5), yet a chain starting at A alternates deterministically between A and B. Its state distribution does not converge to the stationary distribution. This demonstrates why stationarity and convergence must not be treated as synonyms.

Illustrative case study

In a fictional product-analytics project, a team models monthly accounts as active or inactive. Review shows that accounts inactive for one month return at a different rate from accounts inactive for a year. The two-state representation loses predictive history.
The team considers a richer state definition and checks it against held-out sequences. It also documents whether transition estimates differ across seasons. This hypothetical workflow shows how the Markov assumption can guide model criticism; it does not claim that a revised finite-state model captures every aspect of customer behavior.

Common misconceptions and mistakes

Do not interpret the Markov property as independence of consecutive states. Do not assume all transition probabilities must be positive. Do not mix row-vector and column-vector conventions without adjusting matrix multiplication.
A stationary distribution need not be unique, and even a unique stationary distribution does not by itself establish convergence from every initial state. An absorbing state is a state that cannot be left once entered; it is not the same as a stationary probability distribution spread over several states. Finally, estimated transitions describe the fitted model and data conditions, not guaranteed future behavior.

Business use

A Markov chain can provide a transparent approximation when a manageable state representation is useful. Document the state definitions, time interval, transition-estimation method, and initial distribution. Check whether the Markov and homogeneity assumptions are plausible for the intended analysis.
For decisions, distinguish a passive transition model from a model of actions that change transitions. A Markov decision process adds actions and their consequences; an ordinary chain does not automatically answer which intervention is best. Evaluate sensitivity to uncertain transition estimates, especially when long-run conclusions depend on rarely observed movements.

Check your understanding

A chain starts in state A, and P(A→B)=0.25. Is the probability of being in B after two steps necessarily 0.25²?

Answer: No. Sum over all possible intermediate states using their transition probabilities; squaring one transition generally counts neither the right event nor every path.

Related terms

Markov property
Stochastic process
Stationary distribution
Markov decision process
transition matrix
absorbing state
irreducibility
aperiodicity
state representation

Learn next

Next, calculate two-step probabilities from a small transition matrix by both matrix multiplication and path enumeration. Then compare a convergent chain with a periodic alternating chain to distinguish stationarity from convergence.

Sources

Primary terminology reference: Alexander Jung and contributors, The Aalto Dictionary of Machine Learning, Markov chain and Markov property entries: https://aaltodictionaryofml.github.io/ADictML_Main.pdf . The two-state transition, stationary-distribution, and periodic-chain examples are original derivations.

Previous: Markov blanket Next: Markov chain Monte Carlo