Must occupancy values sum to one?

Under the normalized infinite continuing convention, yes; other conventions differ.

Can they recover a policy?

For a stationary policy, conditional action probabilities can be recovered at states with positive occupancy.

Why does the initial state matter?

Discounting gives early visits substantial weight.

AI Glossary / Reinforcement learning and robotics / Advanced

Discounted state-action occupancy measure

Definition

A discounted state-action occupancy measure describes how much a policy visits each state-action pair over time, with later visits receiving less weight. A common normalized definition multiplies the discounted visitation sum by one minus the discount factor, making the measure a probability distribution in an infinite continuing process. The policy, initial-state distribution and environment dynamics all help determine it.

What it means

A policy says what action to choose when a state is encountered. An occupancy measure also reflects how often that state is encountered under the policy. Two policies that differ only slightly in their action probabilities can nevertheless produce different visitation patterns when their actions change future states.
Discounting gives early visits more influence. Consequently, the measure is not generally the same as a long-run stationary distribution or an unweighted count from a log. Its normalization convention matters too: some sources omit the one-minus-gamma factor and use an unnormalized discounted visitation measure.

How it works

For discrete states and actions, the normalized definition is d_pi(s,a) = (1-gamma) sum from t=0 to infinity of gamma^t Pr(S_t=s, A_t=a), with 0 <= gamma < 1. The probabilities include the initial-state distribution, policy and transitions.
Summing over all state-action pairs gives (1-gamma) times the geometric series, which equals one for a continuing process. Summing over actions gives the corresponding state occupancy. Where a state's occupancy is positive, dividing its state-action occupancy by its state occupancy recovers the policy's action probabilities for a stationary policy. A zero-occupancy state provides no such information.
For expected reward r(s,a), the normalized occupancy-weighted reward equals (1-gamma) times the usual expected discounted return. This makes occupancy measures useful for expressing objectives and comparing behavior distributions. Finite horizons and terminated trajectories need explicit conventions; for example, absorbing-state treatment and omitted terminal mass affect normalization.

Three practical examples

Imitation learning: a learner compares its state-action distribution with the distribution induced by demonstrations, rather than only matching actions at isolated recorded states.
Policy analysis: researchers inspect which situations receive substantial discounted visitation under a candidate policy and which receive little or none.
Off-policy evaluation: methods can compare target and behavior state-action distributions, subject to coverage and estimation assumptions. A distribution ratio is meaningful only where its required denominator is supported.

Think of it this way

Imagine a travel log in which the first stop receives the most weight and each later stop receives a smaller weight. Add the weights for every place-and-choice combination, then normalize them. The result describes weighted exposure to the journey's situations, not just the traveler's preferences at any one location.

Worked example

Consider an original continuing process that starts in A and alternates deterministically between A and B. In A, the policy chooses left with probability one-quarter and right with probability three-quarters; either action moves to B. In B, there is one action, return, which moves to A. Set gamma = 0.5.
A occurs at times 0, 2, 4 and so on. Its normalized occupancy is (1-0.5)(1 + 0.5 squared + 0.5 to the fourth + ...), which equals 0.5/(1-0.25) = 2/3. B occurs at times 1, 3, 5 and so on, giving occupancy 0.5 times 0.5/(1-0.25) = 1/3.
The state-action occupancies are therefore d(A,left) = (2/3)(1/4) = 1/6, d(A,right) = (2/3)(3/4) = 1/2 and d(B,return) = 1/3. They sum to one. The ratio d(A,left)/d(A) = (1/6)/(2/3) = 1/4 recovers the policy's left-action probability.
Now assign rewards 0 for A-left, 2 for A-right and 1 for B-return. The occupancy-weighted reward is (1/6) times 0 + (1/2) times 2 + (1/3) times 1 = 4/3. The expected discounted return is therefore (4/3)/(1-0.5) = 8/3.
We can check this directly. Expected reward on every A visit is 1.5, and reward on every B visit is 1. The discounted total is 1.5 + 0.5 times 1 + 0.25 times 1.5 + 0.125 times 1 + ..., or (1.5+0.5)/(1-0.25) = 8/3.
Although raw long-run visits split equally between A and B, discounted occupancy favors A because the process starts there. Starting in B would reverse the state occupancies in this example. Without normalization, the discounted state occupancies would instead be 4/3 and 2/3, summing to 2. These are exact model calculations, not estimates from a finite dataset.

Illustrative case study

Fictional company Cedar Simulation compares two controllers using normalized discounted occupancies. Its analysts use the same initial-state distribution and discount factor for both, and document whether terminal states are represented explicitly.
When one controller appears to spend less weighted time in a certain region, the team investigates both its actions and the states those actions lead to. It does not interpret the measure as a direct count of real-world visits. This fictional workflow reports no operational performance result.

Common misconceptions and mistakes

Occupancy is not just the conditional policy probability pi(a|s). It combines that choice with visitation to the state. Nor does normalized discounted occupancy necessarily equal the stationary distribution: early visits and the initial conditions still matter.
Another mistake is comparing normalized and unnormalized formulas without accounting for their scale. An arbitrary collection of nonnegative numbers summing to one also need not be achievable as an occupancy measure under the environment's dynamics. Finally, matching an occupancy distribution under one setup does not establish equivalent behavior under different starting conditions or dynamics.

Business use

Occupancy measures help teams describe where a sequential system directs its activity and how that activity contributes to a reward objective. Reports should identify discounting, starting conditions and normalization. Estimated occupancies also depend on data quality and coverage. A low estimated occupancy may reflect genuinely rare behavior, limited observation or a mismatch between the data and the policy being analyzed.

Check your understanding

Why did A receive occupancy 2/3?

Answer: It appeared first and on every even step.

Why was discounted return twice the occupancy-weighted reward?

Answer: Dividing by one minus gamma here means dividing by 0.5.

Why were raw visit frequencies different?

Answer: They did not weight early visits more heavily.

Related terms

Discount factor
Stationary distribution
Occupancy distribution
State visitation
Initial-state distribution
Distribution matching
Off-policy evaluation

Learn next

Change the initial state or discount factor in the example, then recompute occupancy and verify the reward identity.

Sources

Kostrikov, Nachum and Tompson, Imitation Learning via Off-Policy Distribution Matching, Section 2.2, Equation 2: https://arxiv.org/pdf/1912.05032 . The alternating-state calculations and company scenario are original illustrations.

Previous: Discount factor Next: Discourse