Does ADVI require conjugate priors?

No. Avoiding model-specific conjugacy requirements is part of its appeal.

Can its approximation be non-Gaussian on the original scale?

Yes. Transforming an unconstrained Gaussian back to constrained coordinates can produce a non-Gaussian distribution.

Is ADVI the same as amortized variational inference?

No. ADVI automates differentiation and optimization; amortization shares an inference function across cases. These are different ideas even though both concern variational inference.

AI Glossary / Probabilistic machine learning / Advanced

Automatic differentiation variational inference

Definition

Automatic differentiation variational inference, or ADVI, is a method for automating approximate Bayesian inference in differentiable probabilistic models. It transforms continuous latent variables to an unconstrained space, fits a tractable variational distribution there, and uses automatic differentiation with stochastic optimization to improve an evidence lower bound. Its output approximates a posterior distribution; automation does not make that approximation exact.

What it means

A Bayesian model combines assumptions about unknown quantities with observed data. Computing its posterior can be difficult, and deriving a separate inference algorithm for every model can slow experimentation. ADVI supplies a reusable computational approach for a broad class of models.
The user still determines the model, data, and substantive interpretation. Automatic differentiation calculates derivatives of the implemented computation; it does not establish that the model is appropriate or that the variational family captures the posterior adequately. Reliable use requires checking the resulting approximation as well as whether optimization appears to have settled.

How it works

First, constrained continuous variables are mapped to real-valued coordinates. The transformed log density includes the appropriate log absolute Jacobian determinant so that the change of variables preserves probability mass. ADVI then uses a Gaussian variational family in the unconstrained space, commonly with diagonal or full covariance.
Reparameterizing Gaussian draws in terms of standard-normal noise makes the objective amenable to Monte Carlo gradient estimation. Automatic differentiation supplies derivatives, and stochastic optimization updates the variational parameters. Results can be transformed back to the original parameter space.
Standard ADVI requires differentiability with respect to continuous latent variables. Discrete observations are compatible with this requirement; discrete latent variables generally need appropriate marginalization or a different treatment rather than direct application of the same recipe.

Three practical examples

A researcher fits a Bayesian regression with continuous coefficients and uses ADVI to explore model revisions quickly. Selected results are compared with a stronger reference computation before drawing conclusions about uncertainty.
An engineering team models a strictly positive scale parameter. A log transformation gives an unconstrained coordinate, and the team checks that the density transformation is handled correctly.
A document-modeling project uses a model whose discrete latent assignments have been marginalized, leaving a differentiable continuous-parameter problem. The team checks that the marginalization and likelihood implementation match the intended model.

Think of it this way

Think of translating many different problems into a common workspace where the same tools can be used. The translation must preserve what the original problem means. A convenient workspace and an efficient tool do not guarantee that the available approximation can represent every feature of the original answer.

Worked example

Consider an original density-transformation example with positive theta and target density p(theta) = 2 exp(−2 theta). This is an exponential distribution with rate two, chosen because the exact density is easy to check. Let z = log(theta), so theta = exp(z).
The derivative of theta with respect to z is exp(z). Therefore the transformed log density is log p_z(z) = log(2) − 2 exp(z) + z. The final z term is the log Jacobian. Differentiating gives 1 − 2 exp(z), which equals −1 at z = 0. Omitting the Jacobian would instead give derivative −2 there and would describe the wrong density.
For illustration, suppose a variational approximation is q(z) = N(0,0.25), with 0.25 denoting variance. On the original scale, theta then has a lognormal approximation. Its median is exp(0) = 1, while its mean is exp(0 + 0.25/2) = exp(0.125), approximately 1.13315. Exponentiating the Gaussian mean therefore gives the median here, not the transformed distribution's mean.
These chosen variational parameters are not claimed to be optimal. The true exponential mean is 0.5, which already shows that this illustrative approximation is poor for that summary. The calculation explains transformation mechanics and interpretation; it does not simulate an ADVI training run or demonstrate convergence.

Illustrative case study

Fictional analytics company PosteriorWorks replaces a slow model-fitting step with ADVI during prototype development. The optimization finishes quickly, but a reviewer notices that several uncertainty intervals are narrower than those from a carefully checked reference analysis on a smaller dataset.
The team investigates the variational family, initialization, and model parameterization. It also compares decision-relevant posterior summaries rather than treating a stable objective value as sufficient evidence of accuracy. This fictional scenario illustrates approximation assessment; it reports no real deployment, timing measurement, or guarantee that a more flexible family will fix every discrepancy.

Common misconceptions and mistakes

ADVI is not simply maximum a posteriori estimation. It optimizes a distributional approximation, whose parameters can include variances and correlations. Nor is automatic differentiation the same as exact inference: accurate derivatives can still guide optimization within an inadequate family.
A diagonal Gaussian approximation imposes independence in its chosen unconstrained coordinates, not necessarily in every representation of the original variables. Transformations can substantially change shapes and dependencies. Another mistake is interpreting optimization termination as proof of posterior accuracy. Multimodality, skewness, difficult geometry, and local optimization behavior can all matter. More optimization iterations cannot necessarily remove restrictions imposed by the approximation family.

Business use

ADVI can support rapid exploration of probabilistic models when computation limits iteration. Record the model, transformations, approximation family, initialization, settings, and diagnostic results. Compare important conclusions with stronger calculations on representative cases where practical. Assess the consequences of approximation error for the intended decision, including tail probabilities and uncertainty intervals. Treat speed as one performance dimension alongside predictive quality, numerical stability, and the reliability of reported uncertainty.

Check your understanding

Why is there a plus-z term in the worked log density?

Answer: It accounts for the Jacobian of theta = exp(z).

Why does exp(E[z]) differ from E[exp(z)]?

Answer: Exponentiation is nonlinear; the Gaussian variance affects the latter.

Does a completed optimization prove the approximation is accurate?

Answer: No. Optimization and approximation quality require separate assessment.

Related terms

Automatic differentiation
Reparameterization trick
Mean field approximation
Probabilistic Programming
Variational inference
evidence lower bound
Jacobian determinant
full-rank Gaussian

Learn next

Work through change-of-variable formulas before studying the Gaussian reparameterization used for gradient estimation. Then compare optimization diagnostics with checks of posterior approximation quality on a model with a known answer.

Sources

Kucukelbir and colleagues, Automatic Differentiation Variational Inference, JMLR, 2017: https://jmlr.org/papers/volume18/16-107/16-107.pdf Stan Reference Manual, Variational Inference: https://mc-stan.org/docs/reference-manual/variational.html

Previous: Automatic differentiation Next: Automatic evaluation