Must every variable be integrated out?

No. The practical benefit often comes from averaging only a tractable subset.

Is replacing an input with its mean sufficient?

Not generally. The required quantity is the conditional expectation of the estimator or function being estimated.

Can the result still vary between runs?

Yes. Randomness in the variables still being sampled remains.

AI Glossary / Graphs and symbolic AI / Advanced

Rao-Blackwellization

Definition

Rao - Blackwellization is a way to improve an estimator by replacing part of its random variation with an appropriate conditional expectation. In Monte Carlo computation, it often means sampling some variables while averaging analytically over others. Under the stated finite-variance conditions, this preserves the target expectation and cannot increase the variance of the individual estimator.

What it means

A simulation may draw several random quantities even though the average effect of some of them is already calculable. Sampling that calculable part adds noise that is unnecessary for estimating the mean. Rao - Blackwellization removes this avoidable component by using its conditional average instead.
The classical statistical theorem conditions an estimator on a sufficient statistic, with conditions ensuring the resulting estimator is usable without knowing the unknown parameter. AI and Monte Carlo literature often uses the name more broadly for conditional averaging. These meanings are related, but an explanation should identify which setting and guarantee it relies on.

How it works

Suppose Y is a random estimator and X is information retained from the simulation. Compute g(X) = E[Y given X]. The law of total expectation gives E[g(X)] = E[Y]. The variance decomposition gives Var(Y) = Var(g(X)) + E[Var(Y given X)], so Var(g(X)) cannot exceed Var(Y).
The practical task is finding a conditional expectation that is tractable and evaluating it under the correct conditional distribution. Replacing a variable by its most likely value or inserting its mean into an arbitrary nonlinear formula is not generally the same operation. For independent Monte Carlo replicates, averaging the conditional expectations carries the per-replicate variance reduction into the sample mean.

Three practical examples

A simulation can sample an operating condition but use the known probability of a binary event under that condition instead of drawing the event itself.
A probabilistic inference method can sample a subset of variables while summing exactly over a manageable discrete subset.
A state-estimation research system can exploit a tractable conditional model for part of its state while representing another part through particles.

Think of it this way

Think of evaluating many possible weather scenarios for an outdoor event. If the average attendance conditional on each scenario is known, repeatedly drawing individual attendance counts adds noise when the only goal is average attendance. The analogy is limited because real conditional averages may be unknown or expensive, and substituting an estimated average introduces additional considerations.

Worked example

Consider an original two-stage simulation. First choose condition X = A or X = B with probability 0.5 each. Then draw a binary event Y. Under A, the event probability is 0.2; under B, it is 0.8. The target is the overall event probability.
A direct simulation records Y as either 0 or 1. Its expectation is 0.5(0.2) + 0.5(0.8) = 0.5. Since Y is a Bernoulli variable with probability 0.5, its variance is 0.5(1 − 0.5) = 0.25.
The Rao - Blackwellized simulation draws only X and records g(A) = 0.2 or g(B) = 0.8. Its expectation remains 0.5. Its variance is 0.5(0.2 − 0.5)² + 0.5(0.8 − 0.5)² = 0.09. The removed variance is 0.25 − 0.09 = 0.16, which equals the average conditional Bernoulli variance because both 0.2(0.8) and 0.8(0.2) equal 0.16.
With 100 independent replicates, the direct sample mean has standard error sqrt(0.25/100) = 0.05. The conditional-average version has standard error sqrt(0.09/100) = 0.03. This comparison uses equal replicate counts, not necessarily equal runtime.
In this tiny example, the entire expectation can be calculated exactly as 0.5, so simulation is unnecessary. The example isolates the principle used in larger problems where averaging over every variable is infeasible but averaging over some variables is practical.

Illustrative case study

Fictional case: Cedar Simulation Lab estimates the frequency of a service event across many possible workload conditions. The first simulator samples both the condition and a yes/no event. The team notices that the event probability is already available from a tractable submodel once the workload condition is fixed.
It replaces the final event draw with that conditional probability and compares repeated estimates under an equal computation budget. It separately checks whether the submodel's probabilities are appropriate. This fictional case illustrates reducing simulation noise; it does not claim that the method corrects a misspecified model or guarantees a runtime improvement.

Common misconceptions and mistakes

Do not claim that Rao - Blackwellization always produces a strictly smaller variance. Equality occurs when there is no conditional variation left to remove. Do not claim that it eliminates bias merely by averaging: it preserves the original expectation, which may already differ from the intended target.
An approximate conditional expectation can introduce error, and a costly exact calculation can offset savings in sample count. For correlated MCMC samples, the variance of a long-run average also depends on autocorrelation; a per-state variance argument alone is not a universal proof of better asymptotic efficiency. Distinguish analytic marginalization from simply dropping variables without accounting for their distribution.

Business use

Businesses using simulation or probabilistic inference can benefit when part of a computation has a known conditional average. The opportunity is especially relevant when estimates fluctuate enough to obscure comparisons between proposed actions.
Assess both statistical precision and computational cost. Confirm the conditional calculation on small cases where full enumeration is possible, then measure uncertainty in realistic workloads. Preserve the assumptions of the underlying model in reporting: a more stable estimate of a flawed model remains flawed. The benefit is reduced estimation noise under the model, not automatic improvement in real-world validity.

Check your understanding

Why does the worked example preserve the mean while reducing variance?

Answer: It replaces each binary outcome by its correct condition-specific expectation. The remaining values 0.2 and 0.8 retain the same overall mean but remove within-condition Bernoulli variation.

Related terms

Conditional expectation
Sufficient statistic
Marginalization
variance reduction
Monte Carlo method
law of total variance
particle filter
estimator

Learn next

Change both condition-specific event probabilities to 0.5. The conditional-average estimator is then always 0.5 and has zero variance, while the direct binary draw still has variance 0.25. Explain why the simulation's uncertainty disappears only for this target expectation.

Sources

Sources: Goodman, original Monte Carlo methods course notes, Week 2, Rao–Blackwellization: https://math.nyu.edu/~goodman/teaching/MonteCarlo15/notes/Week2.pdf . Robert and Roberts, Rao-Blackwellization in the MCMC era, distinctions among uses: https://arxiv.org/abs/2101.01011 .

Previous: Rankrandomization tests Next: Rao-Blackwellized particle filter