Why enforce normalization?

Stationary flow equations alone can admit incorrect overall scales.

Are discounted and stationary targets identical?

Not generally; starting conditions and discounting can change the occupancy distribution.

AI Glossary / Reinforcement learning and robotics / Advanced

GenDICE

Definition

GenDICE, short for Generalized stationary Distribution Correction Estimation, is a method for estimating distribution corrections from offline transition data. These corrections support estimates of quantities associated with a target Markov process or policy. Its formulation covers both discounted visitation and undiscounted stationary settings under appropriate assumptions, with applications including off-policy evaluation and offline PageRank.

What it means

A dataset's frequency of observing a state or state-action pair may differ from its frequency under the target process. GenDICE estimates a ratio that compensates for that difference. It does not require the probabilities of the behavior policies that collected the data, but it still needs suitable transition data and coverage.
The ratio is a weight, not a probability by itself. Its interpretation also depends on whether the target is a discounted occupancy distribution or an undiscounted stationary distribution. Those distributions can differ even in the same environment.

How it works

The method uses the distribution's flow-consistency relationship: weighted probability mass should agree with the mass produced by transitions under the target process, with an initial-state contribution in the discounted setting. It formulates estimation through divergence minimization, normalization and a dual representation suitable for sample-based optimization.
Normalization matters especially in the undiscounted case, where flow consistency alone admits scaled versions of a stationary distribution, including zero mass. The desired nonnegative correction has dataset-weighted mean one. Theoretical consistency and practical estimation remain conditional on coverage, regularity, approximation and optimization assumptions.

Three practical examples

Policy evaluation: researchers estimate a target policy's reward average from existing transition records while correcting their distribution mismatch.
Offline graph analysis: analysts estimate stationary quantities from sampled graph transitions without directly traversing the entire graph during estimation.
Estimator checking: a team examines whether fitted weights satisfy normalization and approximate flow consistency before interpreting a weighted performance estimate.

Think of it this way

Imagine adjusting observations from a survey so they represent a different population. The weights must both match the population's structure and have the right overall scale. Multiplying all weights by zero would make many balance equations look trivial, but it would not produce a meaningful population estimate.

Worked example

Consider an original two-state Markov chain with states A and B. From A, the next state is A with probability 0.8 and B with probability 0.2. From B, the next state is A with probability 0.4 and B with probability 0.6.
Let the stationary probability of A be a. Flow consistency gives a = 0.8a + 0.4(1-a), so a = 0.4a + 0.4. Solving gives a = 2/3, leaving stationary probability 1/3 for B. Checking the incoming mass for B gives (2/3)(0.2) + (1/3)(0.6) = 1/3.
Suppose the dataset's distribution of starting states is p(A) = 1/4 and p(B) = 3/4, with successors sampled according to the stated transition probabilities. The ideal ratios are tau(A) = (2/3)/(1/4) = 8/3 and tau(B) = (1/3)/(3/4) = 4/9.
Their dataset-weighted mean is (1/4)(8/3) + (3/4)(4/9) = 2/3 + 1/3 = 1. Multiplying each dataset probability by its ratio reconstructs the stationary distribution. The ratio for A exceeds one because A is underrepresented in the dataset relative to the stationary target.
Assign reward 3 to A and reward 0 to B. The unweighted dataset reward average is (1/4) times 3 = 0.75. The corrected average is (1/4)(8/3) times 3 + (3/4)(4/9) times 0 = 2. Direct stationary averaging also gives (2/3) times 3 = 2.
Now multiply both ratios by two. The resulting weighted mass still satisfies the homogeneous stationary flow equation, because both sides double. However, its total mass becomes two and its weighted reward becomes four. The zero vector also satisfies that homogeneous equation but has total mass zero. These are not normalized stationary distributions. This illustrates why flow matching without a scale constraint is insufficient here.
For a separate discounted comparison, start the same chain in B and set gamma = 0.5. The normalized discounted occupancy of A satisfies a = 0.5[0.8a + 0.4(1-a)], since its initial mass is zero. Thus a = 0.2a + 0.2, giving a = 1/4 and occupancy 3/4 for B. In this case the dataset already matches the discounted target, so both ratios equal one. Its normalized discounted reward is 0.75, and its ordinary discounted return is 0.75/(1-0.5) = 1.5.
These exact model calculations illustrate the quantities GenDICE aims to estimate. They are not the result of fitting GenDICE to a finite sample. Knowing the transition matrix made direct calculation possible; the estimation problem is harder when only sampled transitions are available.

Illustrative case study

Fictional company Birch Evaluation studies a target controller using archived simulator records. Its analysts document whether their reported value means a long-run average or a normalized discounted average, and specify starting conditions when relevant.
They inspect estimated ratio normalization, coverage and sensitivity to data splits. If a weighted estimate changes substantially, they examine the weights and transition evidence rather than assuming the target controller itself changed. This fictional workflow reports no measured performance gain.

Common misconceptions and mistakes

Behavior-agnostic estimation does not mean that data quality or support is irrelevant. Missing observations cannot be repaired simply by assigning a large finite weight. Nor is a distribution ratio identical to a conditional action-probability ratio in a general sequential problem.
Another mistake is treating undiscounted stationary value as an infinite unnormalized sum of rewards. The average-reward and discounted conventions differ, as the example shows. Flow consistency alone also does not fix scale in the undiscounted setting. Finally, successful numerical optimization does not eliminate approximation error or guarantee that a finite dataset identifies the desired target quantity accurately.

Business use

GenDICE is relevant to research on extracting policy or stationary-process information from existing logs. Reports should state the target quantity, reward definition, normalization, coverage assumptions and data-generation setting. Comparing a raw dataset average with a corrected estimate can be informative, but the correction itself must be evaluated.

Check your understanding

Why was A upweighted?

Answer: Its stationary probability exceeded its dataset frequency.

Why were doubled weights invalid?

Answer: Their total weighted mass was two rather than one.

Why did the discounted example need no correction?

Answer: Its target occupancy already matched the dataset distribution.

Related terms

DualDICE
Stationary distribution
Average reward
Occupancy measure
Density ratio
Off-policy evaluation
Divergence minimization

Learn next

Change the dataset sampling proportions and verify that correct ratios still recover the same stationary reward average.

Sources

Zhang, Dai, Li and Schuurmans, GenDICE: Generalized Offline Estimation of Stationary Values, Sections 2–3: https://arxiv.org/html/2002.09072 . The two-state calculations and company scenario are original illustrations.

Previous: GELU Function Next: Gene expression programming