Does collapsed sampling ignore uncertainty?

No. Correct integration incorporates uncertainty rather than fixing the integrated variable at one value.

Can every Gibbs sampler be collapsed?

Not conveniently. The required marginalization or resulting conditional sampling may be intractable.

Are successive draws independent?

Usually not. Gibbs sampling produces a Markov chain, so dependence and convergence require attention.

AI Glossary / Probabilistic machine learning / Intermediate

Collapsed Gibbs sampling

Definition

Collapsed Gibbs sampling is a Markov chain Monte Carlo method that integrates some unknown variables out of a probability model, then repeatedly samples the remaining variables from their conditional distributions. “Collapsed” describes the smaller sampled state, not a loss of the uncertainty represented by the integrated variables.

What it means

Suppose a model contains group assignments and unknown group proportions. An ordinary sampler might alternate between drawing proportions and drawing assignments. A collapsed sampler can sometimes average over every possible proportion mathematically before drawing assignments.
The remaining variables still have a joint probability distribution. Removing a shared parameter can actually make their dependence more visible: learning one assignment changes what is plausible about others. The method changes how inference is performed; it does not automatically change the underlying statistical model or make its conclusions certain.

How it works

Begin with the joint distribution of the observed data and unknown quantities. Identify variables whose integration is tractable, often because the prior and likelihood have compatible mathematical forms. Derive the resulting marginal distribution and the conditional update for each retained variable.
Initialize those retained variables, then update them repeatedly using the collapsed conditionals. In a topic model, this can involve removing a word's current topic assignment from relevant counts, calculating candidate probabilities, drawing a new topic, and restoring the counts.
Check sampling behavior and estimate quantities from suitable retained draws. If integrated parameters are needed, their conditional distributions given those draws can be used to recover uncertainty. Marginalization itself can be exact while estimates based on a finite chain remain approximate.

Three practical examples

Document exploration: a topic model integrates document-topic proportions and topic-word probabilities, sampling word-topic assignments to explore recurring themes.
Customer grouping: a suitable Bayesian mixture model integrates tractable mixture weights while sampling assignments. The analyst still checks whether the chosen groups describe useful behavior.
Repeated binary outcomes: a beta-Bernoulli model integrates an uncertain success probability while sampling missing binary outcomes, allowing known and currently assigned outcomes to influence each update.

Think of it this way

Think of planning a route while accounting for every possible departure time through a weighted calculation, rather than explicitly choosing a departure time at each step. One decision disappears from the working notebook, but its consequences remain in the route calculation. The analogy stops at the mathematics: arbitrary uncertainty cannot be averaged away efficiently, and a collapsed sampler must use the correctly derived distribution.

Worked example

Consider an original toy model: theta has a Beta(2, 1) distribution, and three binary variables z1, z2, and z3 are conditionally independent Bernoulli(theta) draws. This deliberately samples a prior joint model, without an observed dataset, to isolate the mechanism.
Integrate theta out. Suppose the current other values are z2 = 1 and z3 = 0. Conditional on those two values, theta would have a Beta(3, 2) distribution. Therefore the collapsed probability that z1 becomes 1 is 3 / (3 + 2) = 0.6.
Draw z1 using that probability, then update other coordinates similarly with the latest values. If z1 becomes 1, all three current values contain two successes and one failure. A subsequent conditional draw of theta would use Beta(4, 2), whose mean is 2 / 3.
The 0.6 update averages over uncertainty about theta given the other coordinates. It does not permanently replace theta with its prior mean or with one previously sampled value.

Illustrative case study

Fictional case: Cedar Research is organizing a collection of product-feedback notes. Its analysts compare ordinary and collapsed samplers for the same small topic model. They first use a tiny corpus where they can inspect count updates manually.
One implementation accidentally includes the word being reassigned in its own counts. The team fixes that error before comparing results. It then examines predictive behavior, multiple chains, and computation time per useful sample. Cedar does not declare the collapsed method superior simply because each stored state contains fewer numbers. This case illustrates an evaluation process, not a reported company result.

Common misconceptions and mistakes

Collapsing is not deleting inconvenient variables from the model. Their contribution must be integrated correctly. It is also not the same as blocking: blocking samples several variables jointly, whereas collapsing removes some from the sampled state through marginalization.
Fewer sampled variables do not guarantee faster computation or better mixing in every model. Conditional updates may become more expensive or less parallel. A plausible-looking chain is not proof of convergence. In mixture and topic models, label switching also complicates averaging quantities attached to particular group labels.

Business use

For a business, the value is potentially more efficient uncertainty-aware analysis in models with tractable structure. Useful evaluation measures include effective sample size per unit time, predictive performance, memory use, and stability of decisions across chains.
Ask what was integrated, whether the collapsed conditionals were tested, and which uncertainty summaries reach the decision maker. The inference method cannot rescue an unsuitable model, missing observations, or a business interpretation that treats statistical groups as established customer categories.

Check your understanding

In the worked example, suppose the other two values are both 1. What is the collapsed probability that z1 is 1?

Answer: The conditional distribution for theta is Beta(4, 1), so the probability is 4 / 5 = 0.8. The current value of z1 is excluded when calculating its replacement.

Related terms

Learn next

Learn conjugate updating and ordinary Gibbs sampling first. Then derive the three-variable example by integrating theta directly, and compare the resulting update with an uncollapsed implementation. Study topic-model counts only after the distinction between a marginal distribution and a plug-in estimate is clear.

Sources

Griffiths and Steyvers, “Finding scientific topics,” original paper, especially the integration and Gibbs-update derivation: https://groups.seas.harvard.edu/courses/cs281/papers/griffiths-steyvers-2004.pdf Publisher identifier for the same paper: https://doi.org/10.1073/pnas.0307752101

Previous: Collaborative filtering Next: Collective operation