Must the old policy be randomized?

Randomization with known probabilities is particularly helpful. Observational logs require additional assumptions and cannot recover unsupported actions automatically.

Is lower variance guaranteed?

No. A useful outcome model can reduce variance, but poor overlap and model errors can still cause instability.

Does off-policy evaluation train the new policy?

Evaluation estimates a specified policy's value. Policy learning is a separate task that may use such estimates during optimization.

AI Glossary / Probabilistic machine learning / Advanced

Doubly robust off-policy evaluation

Definition

Doubly robust off-policy evaluation estimates the expected reward of a target decision policy from data collected by another policy, combining a reward or value model with an importance-weighted correction. Under the relevant assumptions, correctness of either the outcome model or the action-probability model can protect the estimate against misspecification of the other.

What it means

A website may want to assess a new recommendation rule using logs from an older rule. The logs reveal outcomes for recommendations actually shown, not for every alternative the new rule would have selected. Simply averaging recorded rewards therefore evaluates the old behavior, not necessarily the proposed policy.

A direct model predicts missing outcomes but can be wrong. Importance weighting adjusts for how often actions were selected, but can be highly variable. Doubly robust evaluation combines the approaches. The word “robust” names a particular statistical property; it is not a guarantee of reliable estimates with arbitrary logs, hidden confounding or unsupported actions.

How it works

For a contextual bandit, first predict each action's expected reward for a context. Average those predictions under the target policy to create a baseline. Then use the observed action's reward minus its predicted reward as a correction, weighted by the target action probability divided by the logging action probability. Average the resulting contributions across evaluation observations.

The correction compensates for outcome-model errors when the propensity information is correct. If the conditional reward model is correct, its residual has mean zero under appropriate conditions, which protects against certain propensity errors.

The target's actions need support in the logged data. Nuisance fitting and evaluation also require appropriate separation or statistical handling. Sequential reinforcement-learning versions account for trajectories and future value, rather than treating every logged step as an independent one-step bandit observation.

Three practical examples

1. A newsletter service estimates the reward of a new article-selection rule from randomized historical recommendations with recorded selection probabilities.

2. A customer-support platform compares routing rules using logged outcomes and agent-assignment probabilities. Unrecorded factors influencing assignment and resolution can undermine the evaluation.

3. A simulator-based control team evaluates a new multi-step policy from trajectories collected by an earlier controller. It uses a sequential estimator and checks whether long-horizon importance weights become unstable.

Think of it this way

Think of preparing a forecast and then correcting it using actual observations, giving extra attention to observations that represent situations the new policy would encounter more often. The analogy is limited: the weights must come from a defensible probability model. Assigning larger weights to whichever records seem persuasive is not doubly robust estimation.

Worked example

Consider a one-step problem with actions A and B. The target always chooses A. In every logged context, the old policy chose A with probability 0.5. Suppose a separately fitted reward model predicts A's expected reward as 0.6 for all four evaluation contexts.

The four logged actions are A, B, A and B, with observed rewards 1, 0, 0 and 1. For a deterministic target, each contribution is 0.6 plus an indicator of choosing A, divided by 0.5, times the observed reward minus 0.6.

The contributions are therefore 0.6 + 2 × 0.4 = 1.4; 0.6; 0.6 + 2 × (−0.6) = −0.6; and 0.6. Their average is 2/4 = 0.5.

The direct model alone would report 0.6. The correction changes it to 0.5 in this tiny sample. Individual contributions can lie outside the reward range; they are estimating-equation contributions, not probabilities. This example checks arithmetic and does not establish that the true target reward is 0.5 or provide a meaningful confidence interval.

Illustrative case study

Fictional case: Reading Harbour wants to replace its article selector. Its analysts retain contexts, chosen articles, rewards and the probabilities used by its randomized logging policy. They fit reward predictions separately and evaluate a locked candidate using a doubly robust estimate.

The team discovers that some new-policy choices were almost never shown historically. Instead of treating the estimate as sufficient evidence, it reports unstable weights and restricts conclusions to better-supported choices. It plans a controlled follow-up for the uncertain region. This is an illustrative workflow, not a claimed experiment or business improvement.

Common misconceptions and mistakes

“Either model can be correct” does not mean both can be arbitrarily wrong. It also does not mean missing action support is repaired by a sophisticated reward predictor.

Estimated logging probabilities must describe how actions were actually selected, including relevant conditioning information. A score proportional to a model's preference is not automatically a historical propensity.

Weight clipping or normalization may help stability but can change bias and the exact guarantee. Searching many candidate policies on the same evaluation data can also create selection optimism. Finally, a one-step estimate ignores downstream consequences unless the reward and setup legitimately capture them.

Business use

Use this approach when evaluating a policy online is costly and historical data contain adequate action coverage and trustworthy context information. Define the reward horizon and the population to which the estimate applies before comparing candidates.

Report uncertainty, weight distributions, support gaps and results from alternative estimators. Investigate disagreement between direct, importance-weighted and doubly robust estimates rather than selecting the most favorable number. Keep final evaluation separate from extensive policy search, and regard a well-supported offline estimate as evidence for a decision, not a universal deployment certificate.

Check your understanding

If the old policy never chose A in a context where the target always chooses A, can the worked formula simply use a zero logging probability?

Answer: No. The ratio is undefined and support is missing. A model-based extrapolation would require additional assumptions rather than inheriting the stated protection.

Related terms

Importance sampling
Propensity scores
Policy value
off-policy evaluation
contextual bandit
propensity score
reward model
overlap
double robustness

Learn next

Recompute the worked example with a different reward prediction while keeping the logged propensities fixed. Then study why sequential evaluation requires trajectory-aware corrections and can become difficult over long horizons.

Sources

Primary sources: Dudik, Langford and Li, Doubly Robust Policy Evaluation and Learning, estimator and bandit setting: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/double_robust.pdf ; Jiang and Li, sequential extension: https://proceedings.mlr.press/v48/jiang16.pdf . Independent review checked bandit equation (1), bias theorem and sequential equation (10). The four-row example and fictional case are original.

Previous: Doubly block circulant Next: Downhill simplex