What does the E-step optimize?

The variational distribution, holding model parameters fixed.

What does the M-step optimize?

The model parameters, holding the variational distribution fixed.

Does a higher bound always mean higher likelihood?

Not across arbitrary restricted-family updates; the approximation gap can change.

AI Glossary / Probabilistic machine learning / Advanced

Variational expectation maximization

Definition

Variational expectation maximization, or variational EM, is an alternating optimization method for latent-variable models when exact posterior inference is difficult. It updates a tractable approximate distribution over latent variables and then updates model parameters to improve the same evidence lower bound. In the common formulation described here, model parameters are point estimates while latent variables have variational distributions.

What it means

Ordinary EM uses the current parameters to calculate an exact latent-variable posterior, then updates the parameters using its expected sufficient statistics. That posterior calculation may be computationally impractical in a complex model. Variational EM restricts inference to a simpler family of distributions and optimizes within that family.

The restriction changes the learning objective from exact likelihood maximization to optimization of a lower bound. This makes difficult models more manageable but introduces approximation error. The name is sometimes used more broadly, including Bayesian parameter distributions; the important distinction is which variables are represented probabilistically and which quantities are optimized as single values.

How it works

For observed data X, latent variables Z and parameters theta, define an ELBO as E-q[log p(X,Z given theta)] minus E-q[log q(Z)]. This equals the log likelihood minus the Kullback - Leibler divergence from q to the exact posterior at theta.

In the variational E-step, hold theta fixed and optimize the bound over the chosen q family. In the M-step, hold q fixed and optimize the bound over theta, usually using expected complete-data statistics. The entropy of q does not depend on theta during that M-step.

Repeat until an appropriate stopping condition is met. Exact coordinate improvements make the bound nondecreasing. They need not make the actual observed-data likelihood nondecreasing when the E-step cannot close the variational gap. Approximate optimization steps need their own monitoring.

If the variational family contains the exact posterior and the E-step finds it, the usual EM behavior is recovered. With restricted families, initialization, parameter constraints and local optima remain important. Predictive validation addresses a different question from whether the training bound has stabilized.

Three practical examples

1. A topic-modeling system alternates approximate document-level latent inference with updates to shared topic parameters.

2. A latent-factor researcher uses tractable approximate distributions where exact expectations would require expensive integration.

3. An engineering team compares richer and simpler variational families to determine whether cheaper inference changes learned parameters enough to affect prediction.

Think of it this way

Think of reconstructing an incomplete puzzle while also adjusting the rules used to interpret it. One step fills in a manageable approximation to the missing pieces; the next revises the rules using that approximation. Progress on this simplified puzzle is useful, but does not establish that the complete puzzle has been solved exactly.

Worked example

Consider three independent observations in a two-component mixture. Their specified likelihood values under A are 0.9, 0.8 and 0.4; under B they are 0.1, 0.2 and 0.6. Only the mixing proportion pi = P(A) is learned; all component likelihood functions are held fixed.

For an intentionally restrictive illustration, let the variational family contain only deterministic assignment distributions. This is a hard-assignment special case, not the usual rich soft variational family. Initialize pi = 0.5.

The restricted E-step selects A, A and B, since these assignments maximize each observation's joint contribution at equal mixing proportions. The assignment distribution has entropy zero. Holding it fixed, the parameter-dependent bound is two log(pi) plus log(1 minus pi), with additional fixed terms log(0.9), log(0.8) and log(0.6).

Differentiating gives 2/pi minus 1/(1 minus pi) = 0, so the M-step sets pi = 2/3. The full bound rises from log(0.054), approximately minus 2.9188, to log(0.064), approximately minus 2.7489.

The actual posterior at the initial parameters assigns probabilities 0.9, 0.8 and 0.4 to A, so ordinary soft EM would use expected count 2.1 and update pi to 0.7. The different updates expose the consequence of the chosen variational restriction. This is one cycle, not a claim of convergence or an implementation of the full topic model cited below.

Illustrative case study

Fictional case: Cedar Research fits a topic model to a collection of technical reports. A fast variational E-step enables repeated model fitting, but analysts observe that different initializations produce different topic descriptions.

They compare bound values, held-out predictive scores and representative documents. A more flexible approximation changes several topic parameters, so the team documents the inference family along with the model. It treats optimization convergence as one diagnostic rather than evidence that the extracted topics are uniquely determined or objectively meaningful categories.

Common misconceptions and mistakes

An approximate E-step does not automatically inherit every guarantee of exact EM. A bound can improve while its gap to the true likelihood changes.

Variational EM is not synonymous with fully Bayesian parameter inference. In the formulation here, theta remains a point estimate; VBEM commonly maintains a distribution over it.

Replacing latent variables by their means is not a universal E-step. Required expectations can involve logarithms, products and other sufficient statistics. Finally, a stable ELBO does not prove global optimality, accurate uncertainty or predictive success on new data.

Business use

Business value lies in making latent-variable modeling feasible when exact inference would be too costly. Teams should record the approximation family, initialization and stopping rules, then evaluate the learned model against the intended use. Compare computational savings with changes in predictions or interpretations. Where decisions depend on uncertainty, inspect whether the inference approximation removes dependencies that materially affect that uncertainty.

Check your understanding

Why is the example's update 2/3 rather than 0.7?

Answer: The restricted family uses two hard A assignments, whereas exact soft inference gives expected A count 2.1.

When does variational EM recover ordinary EM?

Answer: When its E-step can obtain the exact posterior and the remaining updates match ordinary EM.

Related terms

Expectation-maximization
Latent variable
variational inference
evidence lower bound
coordinate ascent
mean-field approximation and variational Bayesian EM

Learn next

Next, derive the ELBO decomposition into log likelihood minus posterior KL divergence. Then compare hard, factorized-soft and structured latent approximations on the same small model, recording both bound and predictive behavior.

Log likelihood

Sources

Primary source: David M. Blei, Andrew Y. Ng and Michael I. Jordan, Latent Dirichlet Allocation, Section 5.3 and Appendix A.4 on alternating variational EM: https://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf

The mixture calculation and fictional case are original teaching examples. The deterministic family is explicitly an extreme restricted illustration.

Independent review checked Section 5.3 and Appendix A.4.

Previous: Variational bound optimization Next: Variational filtering sequential Monte Carlo