Can the approximation ever be exact?

Yes, if the family contains the target and the optimization reaches it.

Would more iterations fix the worked example?

No. Every permitted q assigns A = 1 zero probability.

Is the approximate distribution the original model?

No. The example keeps the target fixed while changing the distribution used to approximate it.

AI Glossary / Graphs and symbolic AI / Advanced

Variational approximation

Definition

A variational approximation replaces a difficult target distribution with a manageable distribution selected by optimization. In a common Bayesian formulation, it approximates the posterior by choosing a member of a specified family that minimizes KL divergence from the approximation to that posterior.

What it means

The result is the best distribution the procedure can find under its chosen restrictions, rather than an automatic recovery of the truth. Those restrictions determine what uncertainty the approximation can represent. A useful business question is therefore: which possibilities or relationships might this representation fail to retain, even after successful optimization?

How it works

Specify a family q with adjustable variational parameters. Define a comparison objective and optimize those parameters. For KL(q||posterior) inference, maximizing the evidence lower bound provides an equivalent objective without needing the posterior's normalizing constant. Use the fitted q to estimate probabilities or expectations. Optimization error and restrictions of the family are separate issues: improving the optimizer cannot recover distributions excluded from the family. This entry focuses on that common probabilistic formulation; other variational methods use different objectives.

Three practical examples

A fictional inventory model estimates two uncertain demand drivers. A simplified representation might produce useful average-demand estimates while losing relationships between the drivers. The replenishment decision should therefore be checked against the joint shortage event it actually depends on.
A fictional document service estimates uncertainty about topic proportions. A team could compare a compact approximation with a richer reference calculation on a small document collection before choosing a representation for a larger trial. The comparison concerns the required outputs, not merely whether the fitting routine terminates.
A fictional equipment analyst tracks two possible faults. An approximation that assigns zero probability to one fault state can report apparently confident conclusions while excluding that state by construction. Inspecting the representation is essential before interpreting the reported confidence as evidence about the machine.

Think of it this way

Think of choosing a map from a restricted collection. Careful selection can identify the most suitable available map, but it cannot add a road that every map in the collection omits. Likewise, a variational approximation can be optimized accurately and still leave out important possibilities. The map analogy concerns representation, not a claim that inference is merely visual matching.

Worked example

Consider an invented posterior over two binary variables A and B. The probabilities of states 00, 01, 10, and 11 are respectively 0.4, 0.1, 0.1, and 0.4. These four numbers sum to one.
Deliberately restrict the approximation to q_r = (r, 1-r, 0, 0), for 0 <= r <= 1. Every member assigns A = 1 zero probability. This is an intentionally severe teaching restriction, not a recommended production choice.
Using natural logarithms, the objective is KL(q_r||p) = r ln(r/0.4) + (1-r) ln((1-r)/0.1). Terms with q = 0 contribute zero. Differentiating inside the interval gives ln(r/[4(1-r)]). Setting this to zero yields r = 0.8. The second derivative, 1/r + 1/(1-r), is positive, confirming the minimum.
Thus q = (0.8, 0.2, 0, 0). Its divergence is ln(2), approximately 0.6931. By comparison, choosing r = 0.25 gives approximately 1.3937, so optimization substantially improves the stated objective.
Nevertheless, the exact expectation of A+B is 1, whereas the optimized approximation gives 0.2. The exact probability of A = 1 is 0.5; q reports zero. Successful optimization has not corrected the family restriction. If all four-state probability distributions were permitted, this target itself would be available and would give zero divergence.

Illustrative case study

In a fictional maintenance workshop, analysts use the four-state example as a review exercise. Their automated report initially emphasizes that the optimization converged and that the objective improved from about 1.3937 to 0.6931. A reviewer asks for the implied probability of each fault, rather than accepting the improvement as a complete quality assessment.
This reveals that the chosen family cannot express A = 1 at all. The team also finds a potentially misleading success: both the exact posterior and q assign probability 0.8 to the event A = B. Agreement on that one event therefore coexists with a large error in another relevant quantity.
The workshop records these observations separately. The optimizer solved its restricted problem correctly; the representation was unsuitable for questions about fault A. A revised experiment must change the family, not simply run the same optimizer for more iterations. No actual equipment performance is claimed by this illustration.

Common misconceptions and mistakes

Convergence does not mean that the approximate posterior equals the target. Here, the optimum still excludes half of the target's probability mass.
Matching one statistic does not validate every downstream use. Agreement on A = B does not repair the incorrect estimate for A = 1.
KL divergence is directional. Reversing the comparison in this example gives infinite KL(p||q), because p assigns positive probability to states that q excludes.
An approximation is not necessarily a single best guess: q here still distributes probability between two states. Nor should its zero probabilities automatically be interpreted as discoveries of impossibility.

Business use

Before relying on an approximate posterior, identify the probabilities and expected costs that drive the decision. In the toy fault example, a policy triggered by A = 1 would require a different validation check from one triggered by A = B. Keep a small tractable reference problem where exact answers are available, and record representation choices alongside optimization settings. This makes it possible to distinguish a calculation failure from an inadequate uncertainty model.

Check your understanding

What is the optimal r?

Answer: 0.8.

What is the approximate expected value of A+B?

Answer: 0.2, compared with the target's value of 1.

Why can one matching statistic be misleading?

Answer: Different distributions can agree on one event while disagreeing sharply on another.

Related terms

Variational posterior
Variational parameters
Approximate inference
Optimization
Mean field approximation
Variational inference
evidence lower bound
KL divergence
posterior distribution

Learn next

Allow q to assign positive probability to all four states and investigate which restrictions remain. Then study how factorized families restrict dependence without necessarily excluding any state outright. Keep family expressiveness and optimization success as separate questions throughout the comparison.

Optimization

Sources

Blei, Kucukelbir, and McAuliffe, Variational Inference: A Review for Statisticians, Sections 1–2: https://arxiv.org/pdf/1601.00670 The four-state posterior, restricted family, calculations, and business illustrations are original examples. They demonstrate representation error rather than a benchmark result reported in the paper.

Previous: Variance Sum Law Next: Variational autoencoder