Is there one universal EUBO formula?

No. Several upper-bound families exist; the posterior-expectation formula here is one specific construction.

Can the upper bound be negative?

Yes. In the worked example, −0.7855 is above −0.9163 even though both are negative.

Does a finite-sample estimate always remain above the evidence?

No. The theoretical bound applies to the defined expectation; an approximate estimate needs its own error analysis.

AI Glossary / Probabilistic machine learning / Advanced

Evidence upper bound

Definition

An evidence upper bound is a quantity that is at least as large as a model's log evidence, or log marginal likelihood, under a stated mathematical construction. In variational inference, such bounds can complement evidence lower bounds and help assess how closely approximate inference brackets the model evidence.

What it means

Model evidence is the probability or density assigned to observed data after integrating out latent variables or parameters. That integration is often difficult. A bound replaces the inaccessible quantity with another expression whose position relative to it is known mathematically.

The name covers more than one construction. This entry focuses on a posterior-expectation EUBO whose gap is KL divergence from the exact posterior to the variational approximation. Chi-divergence and Renyi-based upper bounds are related alternatives, not identical formulas. “Upper” refers to numerical ordering of log evidence, which may be negative; it does not mean a probability must exceed one.

How it works

Let x be observed data, z a latent variable and q(z) a normalized approximation. The focused construction is U(q) = E under p(z|x) of log[p(x,z)/q(z)]. It satisfies U(q) = log p(x) + KL(p(z|x) || q(z)). Nonnegative KL gives the upper-bound property.

By contrast, the familiar ELBO averages the same log ratio under q and equals log p(x) minus KL(q || p(z|x)). The direction of the divergence matters. When q matches the posterior, both gaps vanish under appropriate finiteness conditions.

The catch is computational: the upper expression requires an expectation under the exact posterior, usually the distribution we cannot easily access. Numerical approximation is therefore necessary in difficult problems. A noisy estimate of a bound is not automatically itself a certified bound.

Three practical examples

1. An inference researcher uses upper and lower constructions to investigate whether an approximate posterior is close enough for a model-evidence calculation.

2. A generative-model experiment compares evidence estimates while tracking uncertainty from sampling, rather than treating a single Monte Carlo number as the exact likelihood.

3. A model-comparison study examines whether evidence intervals overlap before concluding that one model is better supported. Broad or unreliable brackets may leave the comparison unresolved.

Think of it this way

Think of estimating a building's height from measurements known to lie above and below its roof. If both limits are valid and close, the height is well constrained. The analogy has an important limit: computational estimates of probabilistic bounds can contain numerical and sampling error. A value labeled “upper” on a software chart does not by itself certify its position above the truth.

Worked example

Consider a discrete toy model with joint probabilities p(x,z=0) = 0.30 and p(x,z=1) = 0.10 for the observed x. Evidence is 0.40, so log evidence is log(0.40), approximately −0.9163. The exact posterior is (0.75, 0.25). Choose q = (0.50, 0.50), with natural logarithms.

The log ratios are log(0.30/0.50) = log(0.60) and log(0.10/0.50) = log(0.20). Averaging them under q gives ELBO = 0.5 log(0.60) + 0.5 log(0.20), approximately −1.0601.

Averaging under the exact posterior instead gives U = 0.75 log(0.60) + 0.25 log(0.20), approximately −0.7855. Hence −1.0601 ≤ −0.9163 ≤ −0.7855, as required.

The upper gap is approximately 0.1308 and the lower gap approximately 0.1438; they are different KL divergences. Exact enumeration makes this check possible. It is not a demonstration that the same expectations are easy to compute for a large latent-variable model.

Illustrative case study

Fictional case: Latent Harbour compares two probabilistic customer-behavior models. One has a higher approximate ELBO, but its inference procedure also appears looser. The researchers examine complementary upper-bound calculations and repeated numerical estimates before interpreting the difference as model evidence.

They discover that the uncertainty from their estimation procedure is large relative to the apparent advantage. The report keeps the comparison unresolved and proposes stronger evaluation rather than advertising a decisive winner. This example illustrates cautious inference reporting; it claims no real dataset or experimental result.

Common misconceptions and mistakes

An upper bound on log evidence is not an upper bound on predictive error. It does not establish that a model is useful, calibrated or causally correct.

Do not swap posterior and variational expectations without changing the meaning: doing so switches the relationship to evidence. Nor should one minimize this upper bound over model parameters as though that necessarily maximized evidence. Tightening an inference approximation for a fixed model and selecting a model are different tasks.

Support matters. If q assigns zero probability where the posterior has positive mass, the focused KL divergence can be infinite. Missing such a region in a finite simulation can misleadingly conceal that problem.

Business use

The practical value is strongest in technical evaluation of probabilistic models, where evidence calculations influence research conclusions or model comparison. A business stakeholder should ask what the bound certifies and what remains approximate.

Reports should identify the exact upper-bound construction, sampling method, support assumptions and numerical uncertainty. Present lower and upper quantities using the same log convention and data normalization. If the calculation only estimates an expectation, label it accordingly. Evaluate downstream predictions separately; a well-understood evidence bracket is not a replacement for task-specific validation.

Check your understanding

What happens to both bounds in the worked example if q is changed to the exact posterior (0.75, 0.25)?

Answer: Both equal log(0.40). For either latent value, p(x,z)/q(z) equals 0.40, and both KL gaps are zero.

Related terms

Importance sampling
model evidence
marginal likelihood
ELBO
variational inference
KL divergence
chi upper bound
posterior approximation

Learn next

Derive the two KL identities side by side and note which distribution supplies each expectation. Then study how numerical estimators differ from deterministic bounds before using them for model comparison.

Sources

Primary sources: Murphy, Probabilistic Machine Learning: Advanced Topics, Section 10.5.3, page 474, equation 10.171: https://github.com/probml/pml2-book/releases/latest/download/book2.pdf ; Ji and Shen, Section 2: https://arxiv.org/pdf/1912.00650 . Full sourcebook downloaded and relevant passage read independently, confirming the exact posterior-to-variational KL construction. Earlier book-access limitation is resolved. Discrete example and calculations are original.

Previous: Evidence reversal Next: Evolutionary acquisition of neural topologies