Does K = 1 recover the ELBO?

Yes, for the stated construction it is the expected log importance weight.

Is more sampling always better for training?

A tighter objective does not guarantee easier optimization or better use of a fixed computing budget.

Is the bound a probability?

No. It is on the log-evidence scale and can be negative. For continuous observations, evidence is a density, whose log need not always be negative.

AI Glossary / Probabilistic machine learning / Advanced

Importance-weighted evidence lower bound

Definition

An importance-weighted evidence lower bound is a variational objective formed by taking the expected logarithm of an average of importance weights. It lower-bounds a model's log marginal likelihood and can tighten the ordinary evidence lower bound by using multiple latent samples.

What it means

A latent-variable model explains an observation x through hidden possibilities z. Computing the evidence p(x) requires combining all those possibilities, which may be difficult. An approximate posterior q(z|x) provides samples that help estimate the evidence.

Each sample receives an importance weight p(x, z)/q(z|x). The numerator measures its joint contribution under the model; the denominator corrects for how often the proposal samples it. The method averages these weights before taking a logarithm.

That order matters. Averaging logarithms gives a different objective. It is also essential to distinguish the expected objective from one Monte Carlo result: the expectation is a lower bound, while a particular random estimate can exceed the true log evidence.

How it works

Draw K independent samples from q(z|x), compute their weights w1 through wK, and define L_K as the expectation of log[(w1 + ... + wK)/K]. For K = 1 this is the ordinary ELBO.

With the required proposal support and integrability conditions, Jensen's inequality gives L_K no greater than log p(x). For fixed model and proposal, the standard construction is nondecreasing in K. The original importance-weighted-autoencoder work also gives convergence to log evidence under a bounded-weight condition.

Training can optimize this objective using sampled gradients. More samples increase computation, and tighter evaluation bounds alone do not guarantee easier optimization or a better downstream model. Numerical implementations usually combine log weights with a stable log-sum-exp calculation.

Three practical examples

A latent-variable research team trains an importance-weighted autoencoder using several posterior-proposal samples per observation.
An evaluator estimates a likelihood-related bound with a stated sample count when comparing generative models.
A probabilistic modeler checks whether a loose single-sample objective is distorting comparisons between candidate latent representations.

Think of it this way

Think of estimating a total by collecting several weighted contributions, combining them, and then applying a nonlinear summary. Applying that summary separately to each contribution can give a different answer.

The analogy concerns the order of operations. Importance weights are defined by probability ratios, not subjective judgments about which sample looks convincing. Changing them informally would break the mathematical argument.

Worked example

A two-state example with an exact answer

Let the latent variable be a or b, with proposal probabilities q(a|x) = q(b|x) = 0.5. Suppose p(x, a) = 0.1 and p(x, b) = 0.3. Then p(x) = 0.4 and log p(x) is approximately −0.9163, using natural logarithms.

The importance weights are 0.1/0.5 = 0.2 and 0.3/0.5 = 0.6. With one sample, the expected log weight is 0.5 log(0.2) + 0.5 log(0.6), approximately −1.0601.

With two independent samples, the average weight is 0.2 with probability 0.25, 0.4 with probability 0.5, and 0.6 with probability 0.25. Thus L_2 = 0.25 log(0.2) + 0.5 log(0.4) + 0.25 log(0.6), approximately −0.9882.

The two-sample bound is tighter: −1.0601 is below −0.9882, which is below −0.9163. However, the individual outcome with average weight 0.6 gives log(0.6), approximately −0.5108, above the true log evidence. Only the expectation has the guaranteed lower-bound property.

Illustrative case study

Fictional case: comparing generative experiments

A fictional research group evaluates two latent-variable models. One report uses a single-sample bound while another uses a much larger importance-sample count. The group realizes that part of the apparent difference may reflect evaluation procedures rather than model quality.

It reruns the comparison with a documented common protocol and repeats estimates to examine Monte Carlo variability. It records both the proposal used and the sample count, alongside model settings and computation cost.

The group also keeps sample-quality and downstream evaluations separate. This fictional case demonstrates reporting discipline without assuming that tighter likelihood bounds imply more useful generated content.

Common misconceptions and mistakes

A realized log average is not guaranteed to lie below log evidence. Calling every single estimate a certified lower bound is incorrect.

The expected bound is nondecreasing with K for fixed distributions, but independently sampled estimates need not increase monotonically. Retraining the model also changes the distributions, so the fixed-model statement cannot be applied blindly across separate training runs.

Importance weights are not the same as normalized weights summing to one. Replacing the weights inside the evidence estimate with normalized weights would destroy the intended calculation. Proposal coverage also matters: missed support invalidates the standard unbiased-evidence identity.

Business use

Use this objective or evaluation method when latent-variable evidence is difficult to compute and suitable proposal samples are available. Require clear reporting of K, proposal distribution, repeated-estimation procedure and numerical precision.

For comparisons, keep evaluation conventions aligned and inspect variability rather than treating extra decimal places as certainty. Weigh computational cost against the value of a tighter bound. If one sample dominates the weights, investigate proposal mismatch and estimation reliability before interpreting a seemingly favorable number as strong evidence.

Check your understanding

In the worked example, one a sample and one b sample are drawn. What is the realized two-sample log estimate?

Answer: The average weight is (0.2 + 0.6)/2 = 0.4, so the estimate is log(0.4), approximately −0.9163. Its equality to the true log evidence in this particular draw does not make every draw exact.

Related terms

Importance sampling
Variational autoencoder
Log-sum-exp trick
Evidence lower bound
marginal likelihood
Jensen's inequality
Monte Carlo estimation

Learn next

Compare log of an average with average of logs on the same weights. Then examine how proposal mismatch changes weight variability and why repeated estimates help distinguish sampling noise from model differences.

Sources

Primary source: Burda, Grosse and Salakhutdinov, Importance Weighted Autoencoders, including the multi-sample bound and its monotonicity result: https://arxiv.org/pdf/1509.00519

The two-state calculation and research-group case are original teaching examples.

Previous: Importance sampling Next: Impulse (delta)