Can the bound be negative?

Yes. All the log-evidence values in the discrete worked example are negative.

Does a zero gap mean perfect real-world predictions?

No. It means the approximation matches this model's posterior for these observations.

Can a sampled estimate exceed the target?

Yes. The one-sample calculation above shows why the expectation and its random estimate must be distinguished.

AI Glossary / Graphs and symbolic AI / Advanced

Variational lower bound

Definition

A variational lower bound is an expression, controlled by adjustable auxiliary choices, that does not exceed a target quantity. In probabilistic machine learning, the term commonly refers to the evidence lower bound (ELBO), which bounds the log marginal likelihood of observed data from below.

What it means

The bound provides a quantity to improve when the exact log evidence is difficult to calculate. A higher bound is useful information, but it is not automatically the exact answer. The remaining gap matters. Two methods can report different bounds because their approximations differ, even when they evaluate the same fixed model and observations.

How it works

For observations x, latent variables z, and an approximate distribution q(z), define ELBO(q) = E_q[log p(x,z) - log q(z)]. Under the usual conditions, log p(x) = ELBO(q) + KL(q||p(z|x)). Nonnegative KL divergence establishes the lower bound. Holding the model fixed, maximizing the ELBO minimizes that divergence. Equality holds when q equals the posterior. Jensen's inequality supplies another derivation. Minimizing the negative ELBO reverses the sign: it minimizes an upper bound on negative log evidence.

Three practical examples

A fictional document-model developer monitors a variational fitting objective. An increase can show progress in fitting the approximation under a fixed model. It does not, by itself, establish improved search results for readers; those require a separate evaluation.
A fictional forecasting team compares two approximate inference routines using the same observations and probability model. Their bounds help assess the routines' different fits, provided both calculations use the same conventions and include the same constants.
A fictional equipment analyst uses an exact two-state calculation to check an implementation. Because the posterior is available in this miniature case, substituting it for q should make the bound meet the log evidence. This offers a concrete check before moving to a larger problem.

Think of it this way

Imagine knowing the height of a platform beneath a ceiling without knowing the ceiling's exact height. Raising the platform can reduce the gap, but the platform's height alone does not reveal the gap. In the ELBO setting, the mathematical identity specifies the gap as a divergence. This analogy does not imply that all independently computed lower bounds have equal accuracy.

Worked example

Take an invented discrete model and a particular observation x. Let p(x,z=0) = 0.12 and p(x,z=1) = 0.18. Probability for other observations accounts for the remaining 0.70. Therefore p(x) = 0.30, log p(x) is approximately -1.203973, and the posterior probabilities of z are 0.4 and 0.6.
Start with q = (0.5, 0.5). Using natural logarithms, the ELBO is 0.5 ln(0.12/0.5) + 0.5 ln(0.18/0.5), approximately -1.224384. This is below -1.203973. The gap is approximately 0.020411, matching 0.5 ln(0.5/0.4) + 0.5 ln(0.5/0.6), the KL divergence.
Next choose q = (0.4, 0.6). Both joint-to-q ratios become 0.30, so the bound is exactly ln(0.30). The gap is zero. The observation has not become more probable: p(x) remains 0.30. Only the approximation has improved.
For a poorer choice, q = (0.8, 0.2), the ELBO is approximately -1.538768 and the gap is 0.334795. Exponentiating the first bound gives approximately 0.293939, below the evidence 0.30. The negative number -1.224384 itself is a log-scale bound, not a probability.

Illustrative case study

In a fictional model-review exercise, analysts call the worked example Model A. They initially use q = (0.8, 0.2), reporting a bound of about -1.538768. Another invented Model B gives the same observation evidence 0.25 and uses its exact posterior, yielding a tight bound of ln(0.25), approximately -1.386294.
Ranking these two reported bounds favors B. Ranking the exact evidences favors A, because 0.30 exceeds 0.25. Both bounds are valid; their different gaps cause the reversal. Tightening A's approximation to its exact posterior resolves this particular discrepancy without changing either model.
The reviewers therefore avoid presenting a bound comparison as conclusive evidence ranking. They retain the approximation settings and distinguish model changes from inference changes. This is a deliberately small counterexample, not a claim that lower-bound comparisons are always useless or always reverse rankings in practice.

Common misconceptions and mistakes

The ELBO bounds log evidence, not every posterior probability or business outcome. Its sign and scale must be labeled correctly.
A single random estimate need not remain below the true log evidence. In the example, sampling z=1 under q=(0.5,0.5) gives ln(0.18/0.5), approximately -1.021651, above -1.203973. The exact expectation is still a lower bound.
Improving q with the model fixed does not change the model's evidence. Conversely, when model parameters also change, an improvement cannot be attributed solely to a tighter approximation.
Different valid lower bounds need not preserve the ordering of their targets, as the case study demonstrates.

Business use

Reports should identify the model, data, approximate distribution, logarithm convention, and whether the displayed value is an exact objective calculation or a sampling estimate. In a small validation case, check both the inequality and the equality obtained with the exact posterior. For operational comparisons, evaluate the actual predictive or decision outcomes alongside the fitting objective. A numerically higher bound alone does not specify customer value, forecast accuracy, or cost savings.

Check your understanding

What is the evidence in the example?

Answer: 0.30.

Which q makes the bound tight?

Answer: (0.4, 0.6).

Why does Model B initially have the higher bound despite lower evidence?

Answer: Its approximation has zero gap, while Model A's chosen approximation has a larger gap.

Related terms

Variational posterior
Approximate inference
Evidence lower bound
variational inference
KL divergence
Jensen's inequality
marginal likelihood
negative log likelihood
Monte Carlo estimate

Learn next

Recalculate the example with q=(0.3,0.7), then verify that the log evidence minus the bound equals the corresponding KL divergence. Next distinguish optimizing q for a fixed model from jointly fitting model and approximation parameters.

Sources

Blei, Kucukelbir, and McAuliffe, Variational Inference: A Review for Statisticians, Section 2.2: https://arxiv.org/pdf/1601.00670 Cornell CS4787/5777, Lecture 26, ELBO definition and Jensen derivation: https://www.cs.cornell.edu/courses/cs4787/2025fa/lectures/lecture26.html All numerical models, comparisons, and business scenarios here are original illustrations.

Previous: Variational filtering sequential Monte Carlo Next: Variational message passing