Is smaller always better?

Smaller is better on the deviance scale; larger is better for elpdWAIC. Always check the output convention.

Does WAIC require a separate test dataset?

Its standard computation uses the fitted posterior and training observations to estimate out-of-sample performance. This does not eliminate the value of external validation.

Can it prove one model is true?

No. It compares estimated predictive performance under a stated task.

AI Glossary / Statistics and probability / Advanced

Widely applicable information criterion

Definition

The widely applicable information criterion, or WAIC, estimates a Bayesian model's out-of-sample predictive performance using pointwise likelihood values across posterior draws. It adjusts apparent fit with a penalty for effective model complexity. On its common deviance scale, smaller WAIC values indicate better estimated predictive performance for comparable models and prediction tasks.

What it means

A model usually describes its training observations more favorably than genuinely new observations. WAIC tries to account for this optimism without refitting the model separately for every held-out observation. It uses uncertainty across the posterior rather than only one fitted parameter vector.

The score concerns predictive distributions, not just whether a point prediction is close. Assigning very little probability to an outcome that actually occurs is penalized by the logarithmic scoring rule. A favorable WAIC does not establish causal validity, fairness or usefulness under a different future population.

How it works

For each observation, average its likelihood over posterior draws and take the logarithm. Sum these contributions to obtain the log pointwise predictive density, lppd. In the common variance-based version, calculate each observation's posterior variance of its log likelihood and sum these variances to obtain pWAIC.

The estimated expected log predictive density is elpdWAIC = lppd − pWAIC. The deviance-scale criterion is WAIC = −2 × elpdWAIC. Thus higher elpdWAIC and lower deviance-scale WAIC express the same preference. The pointwise unit must match the prediction question; dependent observations and hierarchical groups require particular care in defining what is being predicted.

Three practical examples

1. Demand analysis: An analyst compares two Bayesian models for the same product-demand observations. WAIC offers an estimate of predictive performance, with uncertainty in the comparison considered before selecting a more complex specification.

2. Customer response: A team compares alternative probability models for a recorded response. It checks that both models score the same observations and outcomes, rather than comparing incompatible datasets merely because both outputs are labeled WAIC.

3. Service planning: A hierarchical model predicts workloads across branches. Analysts distinguish predicting another observation from an existing branch from predicting a wholly new branch, because these are different evaluation tasks.

Think of it this way

Think of judging a rehearsal while allowing for the advantage of having practiced the same material. A strong rehearsal matters, but some apparent excellence may reflect familiarity. WAIC supplies a statistical adjustment for that advantage. It remains an estimate, and a different real-world performance setting may still require a different rehearsal design.

Worked example

Consider an original arithmetic illustration with two observations and three supplied posterior draws. The likelihood values for the first observation are 0.6, 0.7 and 0.8; for the second they are 0.2, 0.3 and 0.4. These are probabilities of the observed outcomes, not probabilities assigned to the parameter draws.

Their means are 0.7 and 0.3, giving lppd = log(0.7) + log(0.3) ≈ −1.56065. Compute the sample variance of the three log likelihoods for each observation, using denominator 3 − 1. Adding those two variances gives pWAIC ≈ 0.141995.

Therefore elpdWAIC ≈ −1.56065 − 0.141995 = −1.70264, and deviance-scale WAIC ≈ 3.40529. This demonstrates the calculation only. Three draws and two observations are far too limited for a dependable practical model comparison. Also, taking the average of log likelihoods would not reproduce the lppd calculation, which requires logging the average likelihood.

Illustrative case study

In this fictional case, Linden Repairs compares two Bayesian workload models. A more detailed model has a slightly lower WAIC, but a few unusual observations contribute disproportionately to the difference. The analyst checks posterior sampling and examines the observation-level contributions.

The team then evaluates the models using an appropriate cross-validation design and checks the business consequences of forecasting errors. It records that the original WAIC difference was suggestive rather than decisive. The chosen model must support the intended scheduling decision; a small difference in one approximate statistical score is not treated as sufficient evidence on its own.

Common misconceptions and mistakes

Do not confuse the effective complexity penalty with the literal count of parameters. Do not compare deviance-scale WAIC with an expected-log-predictive-density output without converting the scale and direction correctly.

The word “widely” does not mean universally reliable. Influential observations, weak priors and finite-sample effects can make the approximation poor. Poor posterior sampling also undermines the calculation. A likelihood that includes information unavailable at prediction time can answer the wrong question. Comparing models requires a common outcome, dataset and predictive target, not just two numerical scores.

Business use

WAIC can support Bayesian model evaluation when repeated refitting would be expensive. A business report should explain which future prediction is being approximated and whether observed score differences are large enough to influence the decision.

Retain observation-level contributions, sampling diagnostics and comparison uncertainty. The loo documentation recommends PSIS-LOO in preference to WAIC because it offers useful diagnostics and often better finite-sample behavior. When approximations are questionable, use a more appropriate validation strategy rather than presenting an unstable ranking as an objective verdict on model quality.

Check your understanding

Why is a complexity penalty subtracted from lppd?

Answer: The fitted-data score is optimistic about future prediction, and the penalty adjusts that optimism.

In the example, should one average probabilities before taking logs?

Answer: Yes, for lppd. The penalty separately uses variances of the log likelihoods.

Does a lower score guarantee better performance on a shifted population?

Answer: No.

Related terms

Cross-validation
Posterior sampling
Posterior distribution
log predictive density
effective number of parameters
PSIS-LOO
predictive scoring
model comparison

Learn next

Next, compute pointwise log likelihoods for a small Bayesian model and check posterior convergence. Compare WAIC with PSIS-LOO and inspect their diagnostics. Then design validation around the actual future unit of prediction, particularly for grouped or time-dependent data.

Sources

Vehtari, Gelman and Gabry, “Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC,” provides the pointwise calculation and limitations: https://arxiv.org/pdf/1507.04544

The loo package's WAIC documentation describes expected inputs, output conventions and its preference for LOO: https://mc-stan.org/loo/reference/waic.html

The likelihood values and Linden Repairs case are original teaching illustrations, not a fitted empirical study.

Previous: Wide model Next: Wiener connector