Is VMP a sampling algorithm?

Standard VMP uses deterministic variational updates, although extensions may incorporate numerical estimates.

Does it work only on simple chains?

No. Its local calculations can apply to broad graphical structures when the required operators are available.

Does every model have closed-form messages?

No. Nonconjugate components need additional methods or approximations.

AI Glossary / Probabilistic machine learning / Advanced

Variational message passing

Definition

Variational message passing, or VMP, is a way to organize variational inference as local computations and messages in a probabilistic graphical model. Each update revises an approximate distribution using relevant information from neighboring factors or nodes. In its standard conjugate-exponential form, these operations implement coordinate optimization of an evidence lower bound for a factorized posterior approximation.

What it means

Complex Bayesian models often require many interconnected expectations. Deriving every update from scratch is repetitive and can introduce mistakes. VMP packages reusable calculations into messages, so the structure of the graph determines which quantities each update needs.

The messages carry mathematical information such as expected sufficient statistics or natural-parameter contributions. They are not communication between human users or autonomous conversational agents. Local computation also does not imply exact local beliefs: the chosen variational factorization can discard posterior dependencies, even when the graph itself accurately represents the original model.

How it works

Factor the model's joint density into its component terms and choose a variational distribution over unobserved quantities. In a mean-field form, this distribution is a product of factors q-i.

To update q-i, take the expected log joint density over all the other variational factors, exponentiate and normalize. Terms unrelated to the variable cancel into the normalization constant, leaving only the model factors that involve it. This locality is what enables message passing.

For conjugate exponential-family models, expectations and natural parameters have compatible forms, so messages can be summarized compactly. Update a factor, recompute its relevant moments, and pass those moments or parameter contributions to the connected calculations.

Sequential exact coordinate updates improve or preserve the same ELBO. This is not a guarantee of a global optimum, nor a guarantee for every synchronous, damped or approximate update schedule. Nonconjugate factors may require additional bounds, numerical integration or other extensions, whose approximation and convergence behavior must be assessed separately.

Three practical examples

1. A Bayesian mixture model passes expected membership information into component-parameter updates and passes updated parameter expectations back into membership calculations.

2. A hierarchical measurement model exchanges information between uncertain group means and shared noise parameters.

3. A probabilistic-programming system uses reusable inference operators to reduce the manual algebra required when a model graph is modified.

Think of it this way

Think of a network of accounting desks, each responsible for one part of a calculation. Desks exchange the summaries needed by their neighbors instead of copying every document. The process can be efficient and consistent, but the summaries may omit relationships that a complete joint account would retain. That omission is part of the approximation.

Worked example

Consider observations 2 and 4 from normal distributions with common unknown mean mu and precision tau. Give mu an independent Normal(0,1) prior and tau a Gamma prior with shape two and rate two. Precision is inverse variance. Use the factorization q(mu)q(tau).

Initialize q(tau) so its expected precision is one. The mean update combines prior precision one with two expected observation precisions, giving q(mu) precision three, variance 1/3 and mean (2 + 4)/3 = 2.

The precision update needs expected squared residuals, not just residuals at the mean. For observation two, the expectation is (2 minus 2) squared + 1/3 = 1/3. For observation four, it is (4 minus 2) squared + 1/3 = 13/3. Their sum is 14/3.

The updated q(tau) has shape 2 + 2/2 = 3 and rate 2 + one half times 14/3 = 13/3. Its expected precision is 3 divided by 13/3 = 9/13, approximately 0.6923.

Passing that expectation back gives a subsequent q(mu) precision 1 + 2 times 9/13 = 31/13, variance 13/31 and mean 54/31, approximately 1.7419. These are successive coordinate updates, not a converged posterior. Omitting the 1/3 uncertainty terms would produce a different and incorrect update for this variational calculation.

Illustrative case study

Fictional case: Hazel Measurement expands a calibration model from a fixed noise level to an uncertain shared precision. Its modeling software supports conjugate VMP operators, allowing the team to reuse existing normal and gamma calculations.

Analysts check a small example by hand, including uncertainty contributions to expected squared residuals. They then monitor the ELBO and compare predictions with held-out measurements. When a richer dependence model changes uncertainty estimates, the team documents that sensitivity. Automated messages reduce algebraic effort but do not remove responsibility for validating the model and approximation.

Common misconceptions and mistakes

VMP is not the same as belief propagation or expectation propagation. Although all use messages, their objectives, approximations and update rules differ. Substituting one algorithm's message formula into another is not generally valid.

Messages are not always normalized probability distributions. Natural parameters and moments can be the appropriate quantities to exchange. Replacing every expectation with evaluation at a posterior mean can also be wrong, especially for nonlinear sufficient statistics.

A converged factorized approximation does not recover dependencies it was forbidden to represent. Likewise, the ELBO's role as a bound should not be confused with exact model evidence or guaranteed predictive performance.

Business use

Business value lies in making structured Bayesian models easier to implement, extend and audit. Reusable operators can help teams maintain models with many interacting uncertain quantities. Evaluate the resulting predictions and uncertainty, document supported factor types and update schedules, and check important calculations against a small reference problem. Automation is most useful when the inference restrictions remain visible to model reviewers.

Check your understanding

Why does the precision update include variance of mu?

Answer: Expected squared error includes both squared error at the mean and uncertainty around that mean.

What does the factorization remove?

Answer: Posterior dependence between mu and tau, even though their update equations remain coupled through expectations.

Related terms

Mean field approximation
Exponential family
Belief propagation
variational inference
evidence lower bound
conjugacy
expectation propagation and probabilistic graphical model

Learn next

Next, derive the expected-log-joint update for a normal mean and gamma precision. Then compare messages in VMP and belief propagation, paying attention to what each message represents and what objective each algorithm optimizes.

Belief propagation

Sources

Primary source: John Winn and Christopher M. Bishop, Variational Message Passing, especially the factorized variational update and local message construction: https://www.jmlr.org/papers/volume6/winn05a/winn05a.pdf

The normal-gamma calculation and fictional case are original teaching material. Gamma parameters use the shape-rate convention throughout.

Independent review checked Equations 6–7 and local message construction.

Previous: Variational lower bound Next: Variational parameters