Does every posterior component have to be normal?

No. Beta, gamma and other conjugate families can be used with their corresponding likelihoods.

Is this Bayesian model averaging?

It uses a closely related weighting calculation, but the components can simply be alternative priors within one likelihood model rather than entirely different models.

Can updates be done one observation at a time?

Yes, under the specified likelihood structure. Carry forward the whole posterior mixture so the next update uses revised weights and component parameters.

AI Glossary / Probabilistic machine learning / Advanced

Mixture of conjugate priors

Definition

A mixture of conjugate priors is a weighted combination of prior distributions that are individually conjugate to the same likelihood. Bayesian updating produces a mixture of the corresponding component posteriors, with weights adjusted by each component's marginal likelihood.

What it means

A single convenient prior may not represent all plausible views about an unknown parameter. For example, prior knowledge might support either a relatively low event probability or a relatively high one. A mixture can retain those possibilities without averaging them into one less representative distribution.

Conjugacy means each component updates within its own familiar family. The overall prior and posterior are mixtures, and need not belong to that single-component family.

This is uncertainty about a parameter before observing data. It does not necessarily mean each observation comes from a different population or switches between separate parameter values. That would require a different generative specification.

How it works

Write the prior as a sum of weights w_k times component densities p_k(theta), with nonnegative weights summing to one. For each component, update using the common likelihood to obtain p_k(theta given data).

Also calculate its marginal likelihood m_k: the likelihood averaged over that component's prior. The posterior weight is w_k m_k divided by the sum of w_j m_j across all components. Components that better anticipated the data gain relative weight.

Predictions and summaries then average over the updated mixture. A mixture mean is a weighted average of component means, while its variance includes both within-component uncertainty and differences between component means. Sequential updating works by carrying forward both the component parameters and their revised weights.

Three practical examples

A reliability analyst represents uncertainty between two plausible ranges for a single unknown event rate.
A forecasting experiment combines several conjugate normal priors for an unknown mean when prior information supports distinct scenarios.
A product-testing team uses a beta-mixture prior for a response probability and updates the relative credibility of its prior scenarios as outcomes arrive.

Think of it this way

Think of maintaining several alternative planning scenarios. New evidence changes the details within each scenario and also how much weight the team gives it.

The analogy has limits: mixture components are mathematical distributions, and their weights follow Bayes' rule rather than a vote or an informal management preference. Components also need not correspond to objectively distinct real-world regimes.

Worked example

Two beta priors updated by two outcomes

Let theta be a fixed Bernoulli success probability. Give equal prior weight to Beta(1,3), with mean 0.25, and Beta(3,1), with mean 0.75.

Observe one success. The probability of that outcome under each component is its prior mean, so the unnormalized weights become 0.5 times 0.25 = 0.125 and 0.5 times 0.75 = 0.375. The posterior weights are therefore 0.25 and 0.75.

Conjugate updating changes the components to Beta(2,3) and Beta(4,1), whose means are 0.4 and 0.8. The posterior mean of theta is 0.25 times 0.4 + 0.75 times 0.8 = 0.7. Leaving the weights unchanged would incorrectly give 0.6.

Now observe a failure. The component predictive probabilities of failure are 0.6 and 0.2. Multiplying by the current weights gives 0.15 and 0.15, so the revised weights return to one half each. The components become Beta(2,4) and Beta(4,2), with means one third and two thirds.

The combined mean is now 0.5, but the posterior remains a mixture. Averaging the component parameters into Beta(3,3) would generally change the distribution and its uncertainty. Here the mixture variance is 5/84, whereas Beta(3,3) has variance 1/28.

Illustrative case study

A fictional subscription-research team studies one new product's response probability. Prior interviews leave two plausible response ranges, so the analyst proposes a beta-mixture prior rather than forcing a single narrow range.

Before collecting test results, the team documents the component choices and checks the prior predictive distribution. As outcomes arrive, it updates both component parameters and weights, then reports the full predictive uncertainty rather than selecting whichever component currently has the larger weight.

The analyst also checks sensitivity to the initial mixture. The case does not claim that a mixture automatically removes subjective assumptions or that each customer belongs to one of the prior components.

Common misconceptions and mistakes

Conjugate components keep their original weights: the weights generally change according to marginal likelihoods.
Average the parameters to obtain the mixture: a mixture of densities is generally different from a density evaluated at averaged parameters.
Choose the best-fitting component and discard the others: that is selection, not the full Bayesian mixture update.
A flexible prior fixes a wrong likelihood: the assumed observation process still matters.

Business use

This approach can preserve meaningful prior alternatives while keeping calculations manageable. It is useful when a single conjugate prior is computationally convenient but substantively too restrictive.

Document how scenarios and weights were chosen, and examine sensitivity when data are limited. Use proper component priors when marginal-likelihood weights are required; arbitrary normalizing constants from improper priors can make those comparisons undefined. When reporting uncertainty, retain the between-component contribution rather than averaging only component variances. More elaborate mixtures also increase the burden of explaining what the prior represents.

Check your understanding

Two components have prior weights 0.4 and 0.6 and marginal likelihoods 0.3 and 0.1. What are their posterior weights?

Answer: The unnormalized values are 0.12 and 0.06. Dividing by their sum, 0.18, gives two thirds and one third.

Related terms

Conjugate prior
Mixture distribution
Posterior predictive distribution
Beta-binomial model
marginal likelihood
Bayesian updating
model averaging
prior sensitivity

Learn next

Review one conjugate update, such as beta with Bernoulli observations. Then repeat it for two components while separately tracking weights. Calculate mixture means and variances, and compare the result with the misleading shortcut of averaging component parameters.

Sources

Vadim Sokolov, Bayesian Learning, section 3.10 on mixtures of conjugate priors and component evidence weights: https://vsokolov.org/html/_book/03-bl.html

The beta-mixture arithmetic and fictional product study are original teaching illustrations. The general mixture identity applies beyond normal components; component conjugacy must be checked against the chosen likelihood.

Previous: Mixture experiment Next: Mixture of experts