Does it always select one model?

No. Full posterior prediction averages over model or parameter uncertainty.

Can it learn sequentially?

Yes, using the current posterior as the basis for the next valid update under the continuing model.

Does a prior have to be uniform?

No. Priors should be justified, and uniformity is not automatically noninformative under every parameterization.

AI Glossary / Statistics and probability / Intermediate

Bayesian learning

Definition

Bayesian learning is an approach to learning from data that updates a probability distribution over hypotheses or model parameters using Bayes' theorem. Predictions can then average over the remaining uncertainty rather than rely only on one selected model. The learning process depends on the hypothesis space, prior distribution, likelihood, and assumptions linking training data to future cases.

What it means

A learner may have several plausible explanations for observed examples. Bayesian learning changes their relative weights according to how well each explanation predicts the data and how plausible it was beforehand.

The hypotheses might be distinct models, possible parameter values within one model, or combinations of structure and parameters. A continuous parameter space requires integration rather than a finite sum.

This perspective treats learning as uncertainty revision. It differs from keeping only the hypothesis with the highest posterior probability, although maximum a posteriori estimation is often used as a practical approximation. Uncertainty within the chosen model family also does not account automatically for possibilities that the family excludes.

How it works

Assign prior probabilities or densities to hypotheses h. After observing training data D, compute posterior weights proportional to p(D given h) p(h). Normalize these weights over the hypothesis space.

When examples are conditionally independent given a hypothesis, their likelihood factors into a product of individual example likelihoods. Dependence, changing environments, and selective observation require a different likelihood or explicit additional modeling; merely multiplying all available observations can overstate evidence.

For a new input x, the posterior predictive distribution averages p(y given x,h) over p(h given D), under the model's conditional assumptions. For a finite set this is a weighted sum; for parameters it is an integral. The result preserves uncertainty among plausible explanations rather than acting as if one fitted parameter vector were known exactly.

Small models sometimes permit exact calculation. Larger models often use sampling, variational approximations, or other computational methods. Evaluate those approximations and the predictive system separately. Learning can occur in batches or sequentially, provided each update uses the appropriate new evidence and does not count observations repeatedly.

Three practical examples

Early product data: update predictions when only a small number of outcomes are available, while retaining uncertainty instead of reporting an extreme estimate too confidently.
Shared forecasting: use hierarchical models to learn related groups jointly and represent uncertainty about both group-level and population-level parameters.
Scientific machine learning: compare plausible model explanations and average predictions where several remain supported, rather than prematurely selecting one.

Think of it this way

Think of a panel of forecasters receiving new evidence. Their influence changes according to how well their models anticipated it, while the final prediction can combine their opinions using those updated weights. The analogy only works when the weights and likelihoods are specified consistently; popularity voting is not Bayesian learning.

Worked example

Consider two simple hypotheses about a binary outcome. H1 assigns success probability 0.20 and H2 assigns 0.80. Give each prior probability 0.50 and assume outcomes are conditionally independent given the hypothesis.

Observe two successes. The unnormalized posterior weights are 0.50 × 0.20 squared = 0.02 for H1 and 0.50 × 0.80 squared = 0.32 for H2. Dividing by their sum, 0.34, gives posterior probabilities 1/17 and 16/17.

The posterior predictive success probability is (1/17) × 0.20 + (16/17) × 0.80 = 13/17, approximately 0.7647. Selecting only the most probable hypothesis would instead predict 0.80. Both calculations use the same evidence, but only the weighted prediction retains uncertainty between these hypotheses.

If a failure arrives next, the original prior-times-likelihood weights become 0.016 and 0.064. The posterior probabilities become 0.20 and 0.80, and the next success prediction becomes 0.20 × 0.20 + 0.80 × 0.80 = 0.68. The new observation weakens support for H2 without making H1 certain. These calculations illustrate learning within a deliberately restricted two-hypothesis model.

Illustrative case study

Fictional case: Maple Signals predicts whether a newly introduced notification will receive a response. The team initially considers several plausible response-rate regimes rather than assuming one precisely known rate.

As outcomes arrive, the system updates their posterior weights and generates predictive probabilities by averaging across regimes. Analysts compare those predictions with later outcomes and monitor whether the candidate models can explain the observed pattern.

A change in user composition eventually produces behavior poorly captured by every regime. The team revises the modeling approach instead of treating a sharply concentrated posterior over inadequate options as proof of learning success. Operational decisions remain subject to the costs of unnecessary and missed notifications.

Common misconceptions and mistakes

Do not equate Bayesian learning with naive Bayes. Naive Bayes is one particular model family, whereas Bayesian learning is a broader inference approach.

Do not assume a posterior concentrated on one model means that model is true. It may simply be the least inadequate member of the considered set.

Do not confuse model averaging with averaging independently trained models using arbitrary weights. Bayesian averaging uses posterior weights derived from a specified joint model.

Do not treat each incoming record as independent by default, or repeatedly update with the same training set as if it were fresh evidence. Also distinguish uncertainty about parameters from irreducible variability in future outcomes. Useful predictions may need to represent both.

Business use

The approach can help organizations express uncertainty while learning from limited or structured data. It is useful when decisions should respond differently to well-supported predictions and uncertain ones, or when information can be shared across related groups.

Implementation costs include prior specification, inference computation, diagnostics, and monitoring for distribution changes. Compare predictive quality and decision value against simpler approaches. A Bayesian label does not guarantee calibration or superiority, and approximate inference may lose some of the uncertainty benefits that motivated the approach.

Check your understanding

Why is the first predictive probability below 0.80?

Answer: The lower-success hypothesis still has positive posterior weight.

What changes after the additional failure?

Answer: The relative hypothesis weights shift, reducing the posterior predictive success probability to 0.68.

Related terms

Learn next

Compare posterior averaging and single-model prediction on a finite hypothesis example. Then explore continuous-parameter learning, computational approximations, and predictive checks that can reveal an inadequate hypothesis space.

Sources

Poole and Mackworth, Artificial Intelligence: Foundations of Computational Agents, Bayesian Learning, for posterior model weights and predictive averaging: https://www.cs.ubc.ca/~poole/aibook/html1e/ArtInt_196.html Russell and Norvig, Artificial Intelligence: A Modern Approach, Chapter 20 teaching slides, statistical learning: https://aima.cs.berkeley.edu/slides-pdf/chapter20a.pdf

Previous: Bayesian lasso Next: Bayesian linear regression