Does self-normalization recover Z?

The normalized weights themselves do not retain that constant; estimating a normalizer is a separate calculation under suitable assumptions.

Is the estimator always biased?

It is generally biased, with special cases where bias vanishes.

Can all weights be zero?

Yes, a finite proposal sample can miss the target's positive region; then the ratio is undefined and more appropriate sampling is needed.

AI Glossary / Probabilistic machine learning / Advanced

Self-normalized importance sampling

Definition

Self-normalized importance sampling estimates an expectation under a target distribution by drawing from a proposal distribution, weighting the draws, and dividing the weighted sum by the sum of weights. This ratio cancels an unknown target normalization constant. The estimator is generally biased at finite sample sizes, although it is consistent under appropriate conditions.

What it means

Sometimes a distribution can be evaluated only up to proportionality, or sampling directly from it is inconvenient. Another distribution may be easier to sample. Importance weights correct for the difference between the target and the proposal.
Self-normalization makes the weights sum to one, producing a weighted average of the sampled outcomes. It does not turn the original draws into independent target-distributed samples. Nor does it guarantee that the proposal visited every important region: unobserved possibilities cannot contribute to the estimate.

How it works

Let the target density be p(x) = p_tilde(x)/Z, where Z is unknown. Draw independent X_i from a proposal q, and calculate w_i = p_tilde(X_i)/q(X_i). Define normalized weights W_i = w_i/sum_j w_j. The estimate of E_p[f(X)] is sum_i W_i f(X_i).
The unknown constant Z cancels between numerator and denominator. In ordinary settings, consistency follows from convergence of both sample averages and a nonzero finite target normalizer, provided the target expectation exists and q covers the target support.
The estimate requires a positive weight sum. Numerical implementations often work with log weights and subtract their maximum before exponentiating. This preserves their ratios while reducing overflow or underflow.
Weight concentration, proposal coverage, and the particular function being estimated all matter. Normalizing weights solves an unknown-constant problem; it does not repair inadequate sampling or guarantee finite-variance uncertainty estimates.

Three practical examples

Bayesian computation: samples from an accessible approximation are reweighted toward a posterior known through an unnormalized density.
Simulation analysis: a previously generated set of scenarios is reweighted to study a different distribution, provided the scenarios adequately cover its important regions.
Probabilistic model evaluation: expectations are approximated when evaluating a target is easier than drawing directly from it.

Think of it this way

Think of surveying a group whose selection probabilities differ from the population you want to understand. Weights adjust each response's contribution, and dividing by total weight produces an average. But a survey that never reaches an important group cannot recover that group's answers by rearranging the responses it did collect. The analogy highlights coverage, not permission to use arbitrary survey weights.

Worked example

Consider an original two-state example. The unnormalized target masses are p_tilde(A) = 1 and p_tilde(B) = 3. Let the proposal choose each state with probability 1/2, and let f(A) = 0 and f(B) = 10.
Suppose three draws are A, A, B. Their importance weights are 2, 2, and 6. The normalized weights are 0.2, 0.2, and 0.6, giving the estimate 0.2 times 0 + 0.2 times 0 + 0.6 times 10 = 6.
Multiplying both target masses by 100 changes the raw weights to 200, 200, and 600 but leaves the estimate at 6. That is the normalization cancellation in action.
Because this example is tiny, we can compute the actual target expectation: the normalized target probabilities are 1/4 and 3/4, so the answer is 7.5. The three-draw estimate is not exact.
Finite-sample bias is especially clear with one draw. Self-normalization then gives the sampled f value, whose proposal expectation is 5, rather than 7.5. With two independent draws, AA gives 0, either mixed pair gives 7.5, and BB gives 10. Their probabilities are 1/4, 1/2, and 1/4, so the estimator's mean is 6.25.
For the three-draw sample, the common weight-based effective sample size diagnostic is 1/(0.2 squared + 0.2 squared + 0.6 squared) = 25/11, approximately 2.27. This measures weight concentration; it is not an exact count of independent target samples or a guarantee of accuracy.

Illustrative case study

Fictional case: Beech Simulation estimates expected machine downtime under a revised operating scenario. Its analysts consider reweighting an existing collection of simulated conditions rather than immediately rerunning the full simulator.
They verify the density ratios, inspect weight concentration, and check whether the earlier simulation included the operating regimes emphasized by the new target. When an important high-load region has almost no coverage, they generate additional appropriate simulations instead of treating normalized weights as a remedy.
The team reports uncertainty and compares selected results with direct simulation. This fictional case illustrates a computational decision process, not a claim of measured savings or reliable performance in every scenario.

Common misconceptions and mistakes

A ratio of unbiased sample averages is not generally unbiased. Weights summing to one do not prove that the target is well represented. More draws from a proposal with missing support cannot correct the missing region.
Clipping or smoothing weights changes the procedure and requires its own justification. Resampling weighted draws also introduces additional randomness and does not erase estimation error. A large weight-based effective sample size can still be misleading if every draw missed an important mode.

Business use

The method can make posterior or simulation expectations affordable when direct target sampling is difficult. Teams should compare the cost of collecting a better proposal sample with the instability of reusing an inadequate one. Useful reporting includes the proposal definition, target evaluation, weight diagnostics, and sensitivity to additional sampling. Decisions should reflect uncertainty in the estimated quantity rather than the apparent neatness of normalized weights.

Check your understanding

Why does multiplying every raw weight by 100 leave the estimate unchanged?

Answer: The same factor appears in the weighted numerator and the weight-sum denominator, so it cancels. This does not remove sampling error.

Related terms

Importance sampling
Proposal distribution
Effective sample size
Sequential Monte Carlo
Monte Carlo estimation
Normalizing constant
Pareto-smoothed importance sampling

Learn next

Work through small discrete examples to separate sampling error from ratio-estimator bias. Then study proposal design, log-weight computation, and diagnostics for heavy-tailed importance ratios before using the method in high dimensions.

Sampling error

Sources

Art B. Owen, Monte Carlo Theory, Methods and Examples, Chapter 9, Sections 9.2–9.3: https://artowen.su.domains/mc/Ch-var-is.pdf Chapter access timed out. Owen, Zero variance self-normalized importance sampling via estimating equations (2025), Section 2.2, independently consulted for the ordinary ratio estimator, support, consistency and bias: https://arxiv.org/pdf/2510.00389

Previous: Self-management (computer science) Next: Self-selection bias