AI Glossary / Probabilistic machine learning / Advanced
Hierarchical variational posterior
Definition
A hierarchical variational posterior is an approximate posterior distribution constructed with additional layers of random variables or randomized variational parameters. The hierarchy can express dependencies and distribution shapes that a simpler factorized approximation cannot capture.
What it means
Bayesian inference asks what hidden quantities are plausible after observing data. The exact posterior is often difficult to compute, so variational inference selects a tractable distribution from a chosen family and optimizes it to approximate that posterior.
A fully factorized, or mean-field, family treats its latent components as independent. This can be convenient but restrictive. A hierarchical family introduces an additional probabilistic construction that can couple those components after the extra variables are integrated out.
The hierarchy described here belongs to the approximation. It need not be a new layer in the original data-generating model. Confusing those two roles can lead a team to think it changed its scientific assumptions when it actually changed only its inference method.
How it works
One construction first draws variational parameters lambda from q(lambda; phi), then draws latent variables z from a conditional family q(z|lambda). The marginal approximation is q(z) = integral q(lambda; phi)q(z|lambda) d lambda.
Even if z components are conditionally independent given lambda, dependence in or through lambda can make them dependent marginally. The adjustable hyperparameters phi are fitted using a variational objective.
Greater expressiveness can make the marginal density or entropy difficult to evaluate. Hierarchical variational methods therefore need an appropriate inference objective or auxiliary construction, not just a sampling recipe. Ranganath, Tran and Blei's hierarchical variational models provide a specific research framework for this approach; other structured approximate posteriors may use different hierarchies.
Three practical examples
- A Bayesian modeler approximates correlated latent effects that a product of independent distributions would miss.
- A latent-variable research project explores richer marginal shapes without changing its original likelihood.
- An inference team compares factorized and hierarchical approximations to assess whether an important joint uncertainty is being distorted.
Think of it this way
Think of two readings that share an uncertain calibration offset. Once the offset is fixed, their remaining errors can be independent. If the offset is unknown and varies, the readings move together.
This analogy explains conditional versus marginal independence. It does not imply that every hierarchical posterior represents a physical calibration effect. The extra variable may be a computational device introduced solely to improve the approximation.
Worked example
How a shared variable creates dependence
Construct an illustrative variational family as follows: u has mean zero and variance 1; z1 = u + e1 and z2 = u + e2. Let e1 and e2 be independent zero-mean Gaussian noises with variance 0.25, also independent of u.
Given u, z1 and z2 are independent. After integrating over u, each has variance 1 + 0.25 = 1.25, and their covariance is Var(u) = 1. Their correlation is 1/1.25 = 0.8.
The difference z1 − z2 equals e1 − e2, so its variance is 0.25 + 0.25 = 0.5. An independent approximation with the same two marginal variances would instead give variance 1.25 + 1.25 = 2.5 for that difference.
Matching each marginal therefore does not guarantee a good approximation to a joint quantity. These numbers illustrate representational capacity. They do not establish that this family matches any particular true posterior, since no observed dataset or target model has been specified.
Illustrative case study
Fictional case: estimating related demand effects
A fictional analytics team uses a Bayesian model with two related demand effects. Its initial mean-field approximation produces plausible individual uncertainty summaries, but decisions depend on the difference between the effects.
The team compares that difference under a richer hierarchical approximation and under a trusted small-scale reference calculation. It keeps the original likelihood and prior unchanged so the comparison isolates the inference approximation.
The team records differences in uncertainty, runtime and optimization stability. It does not conclude that the richer method is automatically correct. This fictional case shows why approximation quality should be evaluated on the quantities that influence decisions, not only on separate parameter plots.
Common misconceptions and mistakes
Hierarchical does not mean exact. A flexible family can still miss important posterior structure, and optimization can settle at a poor solution.
The extra distribution over variational parameters is not automatically a scientific prior about the original data-generating process. Its role must be distinguished from the model's own prior.
Conditional independence does not imply marginal independence. The worked example demonstrates the difference directly. Conversely, adding separate independent random parameters to each factor need not create cross-variable dependence; the hierarchy must actually connect the relevant components.
Business use
Consider this approach when posterior dependence matters to the intended business calculation. For example, decisions about a contrast, ratio or joint threshold can be sensitive to dependence even when individual intervals appear reasonable.
Ask the analyst to identify the original model, the approximation family and the evidence that the approximation is adequate. Compare selected posterior quantities with a stronger reference method where feasible. Include computational cost and repeated-run stability in the evaluation. A more elaborate approximation should earn its complexity through useful, reproducible improvements on relevant quantities.
People also ask
Does this require changing the underlying model?
No. A hierarchical variational approximation can be introduced while retaining the original likelihood and prior.
Can conditionally independent variables become dependent?
Yes. Integrating over a shared random quantity can induce marginal dependence.
Is it simply a hierarchical VAE?
No. A hierarchical VAE concerns a generative architecture with latent layers. A hierarchical variational posterior concerns the family used for approximate inference, although the two can appear together.
Check your understanding
In the worked family, what goes wrong if the two marginal distributions are retained but their covariance is set to zero when estimating z1 − z2?
Answer: Its variance becomes 2.5 instead of 0.5. Ignoring the positive covariance loses the cancellation of the shared u term and substantially changes the uncertainty calculation.
Related terms
- Mean field approximation
- Conditional independence
- Covariance
- Hierarchical variational autoencoder
- Variational inference
- approximate posterior
- auxiliary variable
Learn next
Derive the variance of a difference using covariance. Then compare a mean-field family with a hierarchical family on a small target distribution where posterior moments can be calculated accurately.
More in Probabilistic machine learning
- Hidden semi-Markov model - A hidden semi-Markov model, or HSMM, is a probabilistic model for sequences in which unobserved states can have explicitly modeled durations.
- Hierarchical Bayesian model - A hierarchical Bayesian model describes related quantities through multiple connected levels of probability distributions.
- Hierarchical softmax - Hierarchical softmax represents a probability distribution over many output classes by arranging them as leaves of a tree and multiplying conditional branch...
- Higher-order Markov model - A higher-order Markov model predicts the next state of a sequence using more than one preceding state.
- Horseshoe distribution - The horseshoe distribution is a continuous probability distribution commonly used as a Bayesian shrinkage prior for sparse signals or coefficients.
- Hybrid generative-discriminative training - Hybrid generative-discriminative training combines learning about the distribution of data with learning to predict a target from inputs.
Sources
Primary source: Ranganath, Tran and Blei, Hierarchical Variational Models, ICML 2016: https://proceedings.mlr.press/v48/ranganath16.pdf
The shared-variable covariance calculation and demand-analysis case are original teaching examples. They illustrate the approximation's possible structure without claiming an empirically verified fit.
Previous: Hierarchical variational autoencoder Next: High-dimensional regime