Does the method require removing a parameter physically?

No. Restrictive priors can encode the reduction within the parent parameterization.

Can it compare any two models?

No. The standard identity requires the shared-likelihood relationship and appropriate support conditions.

Are reduced-model probabilities determined by evidence alone?

No. Prior model probabilities are also needed when converting evidence into posterior model probabilities.

AI Glossary / Statistics and probability / Advanced

Bayesian model reduction

Definition

Bayesian model reduction is a method for obtaining evidence and posterior quantities for alternative probabilistic models from an already fitted parent model when the alternatives differ through their priors while sharing the likelihood. More restrictive priors can constrain or remove parameters. Reusing the parent model's inference can make comparisons across many reduced models substantially more efficient.

What it means

Fitting every combination of model effects separately can be expensive. Bayesian model reduction asks what inference would imply if parameters had been constrained more tightly initially.

The defining feature is a relationship between priors, not simply fewer parameters in a software file. A parameter can be effectively switched off by concentrating its prior at a fixed value, while the likelihood is represented within the parent model's structure.

The method is different from generic model compression or pruning based only on small fitted weights. It evaluates alternatives through probabilistic evidence and posterior relationships. It is also not unrestricted model comparison: changing the observation model or functional likelihood generally falls outside the shared-likelihood reduction identity.

How it works

Let the full model have prior p_F(theta), posterior p_F(theta given y), and evidence p_F(y). A reduced model uses prior p_R(theta) with the same likelihood for the observed data y.

Where the required support and integration conditions hold, the evidence ratio p_R(y) / p_F(y) equals the expectation, under the full posterior, of p_R(theta) / p_F(theta). The reduced posterior is proportional to the full posterior multiplied by that prior ratio.

This follows by substituting Bayes' theorem and canceling the shared likelihood. It allows alternative priors to be evaluated from full-model quantities. The full prior must support the reduced model's relevant possibilities; a simple ratio cannot recover regions excluded by the parent specification. Continuous point restrictions require the appropriate limiting or specialized treatment rather than careless division of incompatible densities.

Some distributional families yield analytic expressions. Practical variational implementations may instead use approximate posteriors and free-energy approximations to log evidence. Their speed does not make the approximation exact. Evaluate model priors, numerical reliability, and sensitivity before translating evidence comparisons into model probabilities or selecting a structure.

Three practical examples

Regression structure: compare models that constrain selected coefficients while retaining the parent model's likelihood representation.
Connectivity modeling: evaluate hypotheses about absent or tightly constrained connections in a probabilistic network model fitted to observations.
Hierarchical analysis: explore alternative prior restrictions or group-level structures within a compatible family, reducing repeated computational fitting where the method applies.

Think of it this way

Think of fitting a flexible measuring template, then asking how evidence changes if selected parts had been fixed beforehand. The shortcut requires compatible versions of the same template.

Worked example

Consider a deliberately small discrete model. A parameter theta can be -1, 0, or 1, with full-model prior probability one-third for each. For a particular observed event y, the likelihoods at these values are 0.10, 0.60, and 0.30.

The full evidence is (0.10 + 0.60 + 0.30) / 3 = 1/3. The full posterior probabilities are consequently 0.10, 0.60, and 0.30.

Define a reduced model fixing theta at zero, so its prior probabilities are 0, 1, and 0. The reduced-to-full prior ratios are 0, 3, and 0. Averaging these under the full posterior gives 0.10 × 0 + 0.60 × 3 + 0.30 × 0 = 1.8.

The reduced evidence is therefore 1.8 × (1/3) = 0.60, agreeing with direct evaluation of the likelihood at zero. If these are the only two candidate models and each has prior model probability one-half, the reduced model's posterior probability is 1.8 / (1 + 1.8), approximately 64.3%. The evidence ratio favors the restriction modestly; it does not establish that the restricted model is true.

Illustrative case study

Fictional case: Ash Dynamics studies a simulator with several optional interaction coefficients. Its analysts fit a parent probabilistic model and consider a documented set of alternatives that constrain selected coefficients near zero.

They use Bayesian model reduction to screen those alternatives efficiently, then inspect important comparisons and the quality of the parent posterior approximation. A proposed alternative with a different noise mechanism is excluded from this shortcut and assessed separately because it changes the likelihood.

The team reports sensitivity to prior widths and retains uncertainty across competitive structures. It avoids presenting the smallest selected structure as uniquely established when several candidates have similar evidence.

Common misconceptions and mistakes

Do not confuse parameter priors within a model with prior probabilities assigned to competing models. Both can affect the analysis, but they play different roles.

Do not assume a smaller model always has higher evidence. A restriction can improve or worsen the balance between fit and the probability mass the model allocates to the observations.

Do not apply the basic reduction identity after changing the likelihood without establishing a valid alternative derivation. The cancellation is central to the method.

Do not label variational evidence approximations exact Bayes factors. A poor parent posterior approximation can affect every reduced comparison. Also avoid treating a data-driven search across many candidate restrictions as irrelevant to how model probabilities and final claims should be interpreted.

Business use

The business opportunity is faster structured exploration when fitting a probabilistic model is expensive and many relevant alternatives differ only by restrictions expressible through priors. It can make sensitivity studies and uncertainty across structures more feasible.

Teams should retain the parent model specification, inference diagnostics, candidate priors, and evidence calculations. Validate consequential reductions through additional checks where practical. The method is most persuasive when computational savings support better analysis, rather than replacing scrutiny of the likelihood, prior choices, or whether the model family captures the phenomenon of interest.

Check your understanding

What permits the shortcut in the worked example?

Answer: The likelihood is unchanged and the reduced prior is supported within the full prior.

Why is 1.8 not a posterior probability?

Answer: It is an evidence ratio; posterior model probability also uses prior model odds and normalization.

Related terms

Bayes factor
Prior distribution
Structure learning
Model evidence
variational inference
model averaging
marginal likelihood
Bayesian model selection

Learn next

Derive the prior-ratio identity for a finite model, then examine a continuous conjugate example. Study how posterior approximation quality and prior restrictions influence evidence comparisons before applying large-scale structural searches.

Sources

Friston, Parr, and Zeidman, Bayesian model reduction, especially the shared-likelihood derivation and examples of reduced priors: https://arxiv.org/pdf/1805.07092 The discrete example above is independently constructed and evaluated both by reduction and by direct evidence calculation.

Previous: Bayesian logistic regression Next: Bayesian multivariate linear regression