AI Glossary / Probabilistic machine learning / Intermediate
Posterior predictive check
Definition
A posterior predictive check compares observed data with replicated data generated from a fitted Bayesian model. By examining relevant features of both, it investigates whether the model can reproduce patterns that matter for its intended use. It is a diagnostic assessment of model fit, not proof that the model is true.
What it means
A model can produce a plausible average prediction while missing the shape, variability, or dependence structure of the data. Posterior predictive checking asks a broader question: if this fitted model generated another comparable dataset, would that dataset resemble what was observed in the ways we care about?
The answer depends on which features are examined. A model that reproduces the mean may still miss extreme observations, long runs, group differences, or excess zeros. Choosing meaningful comparisons makes the check useful for understanding a specific mismatch rather than simply awarding a pass or fail.
How it works
Fit the Bayesian model and obtain credible draws from its posterior parameter distribution. For each selected parameter draw, simulate a replicated dataset from the observation model. This includes observation variability as well as the variation between posterior parameter draws.
For an ordinary regression check, preserve the relevant observed predictor values and design when generating replications. Compare graphs or summary statistics of the replicated and observed data. In hierarchical or temporal models, specify which effects or histories are held fixed and which are regenerated; these choices define the question being checked.
A comparison statistic can be a mean, a maximum, a zero count, or a feature such as lagged dependence. More general discrepancies can depend on parameters as well as data. The check should follow adequate computational diagnostics, because unreliable posterior sampling can make the replicated reference misleading.
Three practical examples
- Request-count modeling: compare observed zero counts and unusually busy periods with model-generated equivalents, rather than checking only total volume.
- Sensor modeling: compare the lengths of runs above a threshold to investigate whether an assumed independence structure misses persistence.
- Grouped service analysis: compare within-group spreads and between-group differences to see which level of a hierarchical model needs attention.
Think of it this way
Think of asking a model to produce several imitation datasets and placing them beside the original. A distinctive feature of the original can reveal what the model has failed to imitate. The analogy is limited: visual resemblance depends on the chosen display, and an imitation that looks convincing can still fail on an unexamined feature.
Worked example
Consider original observed counts (0, 0, 0, 4), with mean 1 and three zeros. Four deliberately constructed illustrative comparison arrays are (1, 1, 1, 1), (0, 1, 1, 2), (0, 0, 2, 2), and (0, 0, 0, 4).
Every array has total 4 and mean 1. Their zero counts are respectively 0, 1, 2, and 3. Using variance with denominator 4, their spreads are 0, 0.5, 1, and 3. The observed variance is also 3: the squared deviations from mean 1 are 1, 1, 1, and 9, summing to 12.
Thus a mean-only display cannot distinguish these very different patterns. Zero counts and spread reveal the distinction. These arrays are teaching constructions, not claimed posterior simulations, so their proportions do not estimate a posterior predictive probability. In a real check, generate many comparable arrays from the fitted model and locate the observed summaries within those simulated distributions.
Illustrative case study
Fictional case: Orchard Support uses a Bayesian model for daily incoming requests. The model captures the overall mean, but staff notice alternating quiet and exceptionally busy days. The analyst plots replicated daily sequences, their zero counts, and their upper-tail summaries alongside the observed sequence.
The team investigates whether time-varying rates, dependence, or a more flexible count distribution could explain the discrepancy. It does not automatically choose the most complicated alternative: it revises a specific assumption and checks again. Separate future-period evaluation remains part of the project. This invented scenario describes an analysis process, not an observed performance improvement.
Common misconceptions and mistakes
A posterior predictive check is not simply plotting fitted conditional means against observations. Replicated outcomes include observation noise. It is also different from a prior predictive check, which simulates without conditioning the parameters on the observed outcomes.
Using the same data to fit and diagnose a model is useful for model criticism, but it is not an independent held-out evaluation. Finally, success on several selected summaries cannot establish that every relevant aspect of the model is adequate.
Business use
For business use, select checks tied to the decisions the model supports. If staffing depends on unusually busy days, a mean-only fit assessment is incomplete. If planning depends on simultaneous demand across locations, examine that dependence rather than just each location separately.
Keep the observed feature, replicated reference, and proposed explanation together in the review record. A useful finding identifies a specific modeling weakness and its practical consequence. Combine this diagnosis with appropriate external or held-out evaluation before relying on the model for future decisions.
People also ask
Does a check always need a p-value?
No. Graphs and carefully chosen numerical summaries can expose the nature of a mismatch more clearly than one number.
Should parameters be fixed at their posterior mean?
That generally omits parameter uncertainty. Standard posterior predictive simulation draws parameters from the posterior and then generates outcomes.
Does a mismatch identify the exact cause?
No. Several assumptions or data-quality issues can produce similar discrepancies, so further investigation is needed.
Check your understanding
The model reproduces the observed mean but rarely generates the observed number of zeros. Has the mean check established adequate fit?
Answer: No. It shows agreement on one feature. The zero-count mismatch indicates an aspect requiring further diagnosis.
Related terms
- Posterior predictive distribution
- Posterior predictive p-value
- prior predictive check
- model criticism
- discrepancy statistic
- observation noise
- hierarchical model
- held-out evaluation
Learn next
Simulate outcomes from a simple fitted Bayesian model, preserving a clear replication design. Compare the mean and one additional feature that the model does not automatically fit. Then distinguish diagnostic checking on fitted data from forecasting evaluation on data that was not used for fitting.
More in Probabilistic machine learning
- Poisson regression - Poisson regression models a nonnegative count outcome using a Poisson distribution whose mean depends on predictors.
- Policy gradient theorem - The policy gradient theorem expresses how an agent's expected cumulative reward changes when the parameters of its action-selection policy change.
- Polyak-Ruppert averaging - Polyak-Ruppert averaging is a stochastic-approximation technique that averages successive parameter estimates rather than returning only the last estimate.
- Potential scale reduction factor - The potential scale reduction factor, commonly associated with R-hat, is a family of diagnostics for comparing multiple Markov chain Monte Carlo runs.
- Potts model - A Potts model is a model of interacting variables that each take one of several categorical states.
- Preconditioned stochastic gradient descent - Preconditioned stochastic gradient descent modifies a stochastic gradient with a matrix before using it to update model parameters.
Sources
Stan User's Guide, Posterior and Prior Predictive Checks: https://mc-stan.org/docs/2_38/stan-users-guide/posterior-predictive-checks.html Gelman, Meng and Stern, Posterior Predictive Assessment of Model Fitness via Realized Discrepancies, original paper: https://stat.columbia.edu/~gelman/research/published/A6n41.pdf The Stan guide and Gelman-Meng-Stern Sections 2.2-2.3 were independently consulted for replication and parameter-dependent discrepancies.
Previous: Posterior (prediction) Next: Posterior predictive distribution