Must the behavior action probabilities be known?

The method is designed to avoid that requirement.

Can a correction exceed one?

Yes; the example assigns A a ratio of 3.

Does a weighted average automatically certify policy quality?

No; estimation quality and coverage still matter.

AI Glossary / Reinforcement learning and robotics / Advanced

DualDICE

Definition

DualDICE is a reinforcement-learning method for estimating corrections between a target policy's discounted state-action distribution and the distribution represented by an offline dataset. Its name expands to Dual stationary DIstribution Correction Estimation. The corrections can support off-policy evaluation without knowing the behavior policies' action probabilities. They do not remove the need for adequate data coverage.

What it means

A dataset may contain some state-action pairs more often, and others less often, than the policy being evaluated would encounter them. DualDICE estimates a ratio that corrects this mismatch. The ratio concerns joint state-action visitation, not merely the probability of choosing an action at a given state.
Here, discounted visitation is normalized by one minus the discount factor. Consequently, a corrected expected reward uses the paper's normalized value convention; recovering the usual discounted return requires undoing that normalization.

How it works

DualDICE uses a saddle-point formulation with an auxiliary function nu and a ratio function zeta. In the quadratic formulation, the optimal zeta equals the desired distribution ratio. Learning uses dataset transitions, initial-state samples and actions from the target policy. The formulation avoids explicit behavior-policy action ratios and the double-sampling difficulty of directly squaring a conditional Bellman expectation. Function approximation and optimization affect practical estimates.

Three practical examples

Logged simulations: a team evaluates a target controller using transitions collected by several earlier controllers, documenting the target's starting-state distribution separately.
Reward comparisons: analysts examine corrected reward averages for a fixed target policy. They keep the estimated visitation correction distinct from the reward values attached to recorded transitions.
Coverage diagnostics: researchers inspect where estimated ratios are large and investigate whether a small number of transitions dominates an evaluation. A finite estimate is not, by itself, proof that the dataset adequately represents the target behavior.

Think of it this way

Imagine adjusting a survey because its respondents do not represent the population of interest in the right proportions. An overrepresented group receives less weight and an underrepresented group receives more. For sequential decisions, the population of interest is itself shaped by actions and their consequences, making its proportions harder to discover.

Worked example

Consider an original continuing environment with one state S and two actions, A and B. Either action returns to S. The target policy chooses A with probability 3/4 and B with probability 1/4 at every step. Set gamma = 1/2. Because S is always occupied, its normalized discounted occupancy is one, and the target state-action occupancies are 3/4 and 1/4.
Suppose the dataset's sampling distribution assigns probability 1/4 to A and 3/4 to B. The exact correction for A is (3/4)/(1/4) = 3. For B it is (1/4)/(3/4) = 1/3. Thus, an A transition receives three times its original contribution, while a B transition receives one-third. Neither ratio is itself an action probability: A's ratio exceeds one.
As a normalization check, the dataset-weighted average ratio is (1/4) times 3 + (3/4) times (1/3) = 1. Multiplying each dataset probability by its ratio reconstructs the target probabilities, 3/4 and 1/4.
Give A a deterministic reward of 4 and B a reward of 0. The uncorrected dataset reward average is (1/4) times 4 = 1. The corrected average is (1/4) times 3 times 4 + (3/4) times (1/3) times 0 = 3. Direct evaluation under the target gives the same answer: (3/4) times 4 = 3 per step.
The ordinary expected discounted return is 3/(1-1/2) = 6. Reporting the corrected average 3 as the unnormalized return would therefore confuse two different conventions.
We can also check the auxiliary-function relationship in this small setting. Set nu(S,A) = 16/3 and nu(S,B) = 8/3. The target-policy average of nu at the next state is (3/4)(16/3) + (1/4)(8/3) = 14/3. Discounting this by one-half gives 7/3. Subtracting that quantity from each current nu value yields 16/3 - 7/3 = 3 for A and 8/3 - 7/3 = 1/3 for B, exactly the two corrections.
These auxiliary values are not the policy's reward-based action values. They were constructed to illustrate the correction relationship. The calculations supply the distributions directly; they demonstrate an ideal solution, not the result of running DualDICE on a finite dataset.
If the dataset never contained A, its denominator would be zero despite positive target occupancy. The above correction would then be unavailable. Renaming the method behavior-agnostic would not recover the missing evidence. Likewise, an estimated ratio of 3 from limited observations would need uncertainty assessment rather than being treated as exact.

Illustrative case study

Fictional company Willow Simulation maintains logs from several experimental controllers. Its analysts define a candidate target policy and the starting conditions relevant to their evaluation. They investigate whether the collected transitions cover the situations that policy could encounter.
The team compares an unweighted reward average with a distribution-corrected estimate and records the normalization used. It also examines how much the result changes across data splits and whether a few high-weight observations drive the estimate. A discrepancy triggers investigation of coverage, estimation error and data consistency. This fictional workflow reports no measured improvement or deployment result.

Common misconceptions and mistakes

Behavior-agnostic does not mean that the target policy is unknown, that transition information is unnecessary or that missing support can be ignored. The one-state example also does not generalize into a rule that joint occupancy ratios always equal action-probability ratios: in larger environments, policies can change which states are visited.
Another mistake is interpreting the auxiliary nu function as an ordinary reward-value predictor. Finally, a mathematically correct weighting identity does not establish that a fitted ratio model accurately estimates the required weights.

Business use

DualDICE is relevant to research on evaluating sequential decision policies from existing logs. A useful evaluation report identifies the target policy, initial conditions, dataset composition, reward definition and value normalization. It should separate exact identities from estimated quantities. Comparisons across candidate policies also need consistent evaluation conditions, especially when the available evidence covers their behavior unevenly.

Check your understanding

Why was A upweighted?

Answer: Its target occupancy exceeded its dataset frequency.

Why did the corrected average equal 3?

Answer: Weighting reconstructed the target's reward average.

Why was discounted return 6?

Answer: The normalized value was divided by one minus gamma.

Related terms

Off-policy evaluation
Occupancy measure
Distribution ratio
Saddle-point optimization
Behavior policy
Coverage
Double-sampling problem

Learn next

Change the dataset's action frequencies while keeping the target fixed, then recompute the exact ratios and corrected reward.

Sources

Nachum, Chow, Dai and Li, DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections, Sections 2 and 3: https://arxiv.org/pdf/1906.04733 . The one-state calculations and company scenario are original illustrations.

Previous: Dual total correlation Next: Duality gap