Is it ordinary Gibbs sampling?

Exact Gibbs sampling draws directly from each full conditional. Metropolis within Gibbs replaces one or more such draws with conditional Metropolis - Hastings transitions.

Can a block contain several parameters?

Yes. Blocks can be multivariate, and grouping correlated parameters may improve movement at additional computational cost.

Is a full deterministic sweep necessarily reversible?

Not necessarily. Each component can preserve the target, and their composition preserves it, without the complete ordered sweep satisfying detailed balance.

AI Glossary / Probabilistic machine learning / Advanced

Metropolis within Gibbs

Definition

Metropolis within Gibbs is a Markov chain Monte Carlo strategy that updates variables or blocks conditionally, using a Metropolis - Hastings step for conditionals that are inconvenient to sample exactly. Other blocks can still use direct Gibbs draws.

What it means

Some posterior conditionals are familiar distributions; others can only be evaluated up to a constant. This strategy lets each part use a suitable update without requiring a direct sampler for the entire joint distribution.

During one block update, all remaining variables are held at their current values. The conditional target therefore changes as other blocks change. A valid Metropolis - Hastings transition preserves the relevant conditional distribution, even though one transition is not an independent draw from it.

Combining such updates can preserve the desired joint distribution. Convergence from an arbitrary starting point still requires suitable conditions, and finite runs need diagnostic checks. Correct stationary behavior and efficient exploration are separate questions.

How it works

Choose a block and propose a replacement value. Calculate the conditional target-density ratio and, when needed, the reverse-to-forward proposal-density ratio. Accept with the smaller of one and their product; otherwise retain the current block value.

Proceed to the next block using the updated values already available in the current sweep. If a full conditional can be sampled directly, use that exact Gibbs update instead. Repeat sweeps and retain the resulting joint states for Monte Carlo summaries.

One valid Metropolis step per selected block can be enough to define the sampler; an inner chain does not need to be run to equilibrium at every update. Additional inner steps can change efficiency. Strong correlations between blocks may make coordinatewise movement slow, motivating blocking, reparameterization or another sampling method.

Three practical examples

A hierarchical model samples familiar variance conditionals directly while using Metropolis proposals for a nonconjugate coefficient block.
A spatial model updates groups of latent variables while holding neighboring groups at their current values.
A research implementation combines different transition kernels for parameters with different constraints, checking that each targets the appropriate conditional.

Think of it this way

Think of revising a group plan one section at a time while keeping the other sections fixed. Each revision must be assessed against the latest version of the rest of the plan.

The analogy explains conditional updating and sequencing. The sampler does not simply accept improvements: it sometimes accepts lower-density proposals so that its long-run behavior represents a distribution rather than only an optimum.

Worked example

A conditional update followed by an exact draw

Use an invented bivariate Gaussian target with zero means, unit marginal variances and correlation 0.5. The conditional distribution of X given Y = y is Gaussian with mean 0.5y and variance 0.75. The corresponding conditional for Y given X has the same form.

Start at X = 0, Y = 2. Propose X = 1 from a symmetric random-walk proposal. With Y fixed at 2, the conditional mean for X is 1. Moving from 0 to 1 gives density ratio exp(1/(2 times 0.75)) = exp(2/3), approximately 1.9477, so the proposal is accepted with probability one.

Next sample Y from its full conditional using the newly accepted X = 1. That conditional has mean 0.5 and variance 0.75. Suppose, illustratively, the draw is Y = 0.8; the sweep ends at (1, 0.8).

For comparison, proposing X = 3 at the original state would give density ratio exp(−(4 − 1)/1.5) = exp(−2), approximately 0.135335. A uniform acceptance draw of 0.2 would reject it, retaining X = 0 before the Y update.

This Gaussian example permits exact draws for both coordinates; replacing one illustrates the mechanics without claiming better efficiency.

Illustrative case study

A fictional forecasting team adds a nonconjugate parameter to an existing Bayesian model. Most conditional updates remain simple, but one no longer has a convenient direct sampling distribution.

The team replaces that update with a Metropolis - Hastings transition and verifies the sampler on a reduced example with a known posterior. It records rejected proposals as repeated states and ensures that subsequent blocks use current values.

It then examines multiple chains, autocorrelation and effective sample sizes for quantities used in forecasts. A high acceptance percentage alone is not taken as success, because very small accepted steps can still explore slowly. No forecasting improvement is assumed in this fictional case.

Common misconceptions and mistakes

Every Metropolis substep must produce an independent conditional draw: it need only be a valid transition preserving that conditional.
Rejections should be discarded: retained states encode the chain's residence times and belong in its history.
Condition on old values throughout a sequential sweep: later blocks normally use values already updated during that sweep.
Correct component kernels imply fast mixing: strong dependencies can still produce slow joint exploration.

Business use

The approach is practical when extending a structured Bayesian model and only a few conditional updates are difficult. It permits targeted implementation work, but the resulting chain must be assessed as a whole.

Track computational cost per useful effective sample, not just iterations per second. Check parameter constraints, proposal asymmetry and the exact factors that change under each update. Adaptive tuning needs a valid design rather than unrestricted changes throughout sampling. For reported decisions, distinguish Monte Carlo error from uncertainty in the model itself.

Check your understanding

A block proposal is rejected. Should the sampler retry until it gets an accepted replacement before recording anything?

Answer: Not under the ordinary algorithm. It retains the current block value and proceeds according to the specified schedule. Discarding rejections changes the transition behavior and generally biases unweighted samples.

Related terms

Learn next

Derive the two conditionals of a correlated Gaussian and implement one exact and one Metropolis update on paper. Check accepted and rejected cases separately. Then compare coordinatewise and blocked movement, distinguishing target invariance from the speed of convergence.

Sources

Faming Liang, Computational Statistics lecture notes, Metropolis-within-Gibbs construction: https://www.stat.purdue.edu/~fmliang/STAT546/Lecture3_25.pdf

University of York, Bayesian Statistics notes, section on Metropolis within Gibbs: https://www-users.york.ac.uk/~bp787/Bayes_notes.html

The Gaussian sweep and fictional forecasting case are original teaching illustrations. The illustrative conditional draw is chosen for explanation, not reported as a random simulation result.

Previous: Metropolis-Hastings Next: Michelson contrast