Why do small likelihoods matter so much?

Their reciprocals are large and can dominate the average that is subsequently inverted.

Can log-scale computation fix the estimator?

It can improve numerical arithmetic, but it does not remove the underlying instability caused by rare influential draws.

Are all harmonic-mean-based methods equally problematic?

No. Modified methods can change the weighting and assumptions substantially; evaluate the specific estimator rather than transferring guarantees between them.

AI Glossary / Probabilistic machine learning / Advanced

Harmonic mean estimator

Definition

The harmonic mean estimator is a method for estimating a Bayesian model's marginal likelihood from posterior samples by taking the harmonic mean of their likelihood values. Its simple formula is attractive, but the original estimator can be extremely unstable because very small likelihood values create very large reciprocal contributions. Its theoretical consistency does not guarantee practical accuracy.

What it means

Marginal likelihood, also called Bayesian evidence, averages the likelihood over the parameter prior. It is used in Bayesian model comparison and differs from the likelihood evaluated at the best-fitting parameter. Computing the required integral can be difficult even when posterior sampling is feasible.

The harmonic mean estimator appears to offer a shortcut: reuse existing posterior draws. However, the posterior may visit the regions needed for a stable reciprocal-likelihood average only very rarely. Estimates can therefore appear reassuringly stable while missing influential contributions. This entry concerns the original estimator, not every modified or learned estimator that uses a related identity.

How it works

Let L(theta) be the likelihood, pi(theta) a normalized proper prior and Z the marginal likelihood. When the likelihood is positive on the relevant prior support and the identity's integrals are well defined, the posterior expectation of 1/L(theta) equals 1/Z. Bayes' rule gives this relationship by canceling the likelihood inside the integral.

For N posterior draws, calculate the average of their reciprocal likelihoods and then invert it: Z_hat = N divided by the sum of 1/L(theta_i). The draws must represent the posterior appropriately; arbitrary fitted parameter values or prior draws cannot simply be substituted.

Rare low-likelihood draws can dominate that sum. The reciprocal estimator can have infinite variance, making familiar standard-error arguments invalid. Even finite-variance cases can be impractical. Numerical stabilization using logarithms helps prevent overflow or underflow, but cannot solve this statistical tail problem. Proper prior normalization and likelihood constants also matter when estimating evidence.

Three practical examples

1. Legacy model-comparison review: identify reports that reused posterior likelihoods through the original harmonic mean formula. Check the method's reliability before interpreting resulting evidence ratios.

2. Bayesian computation teaching: compare the estimator with an exact marginal likelihood in a small conjugate model. This reveals why apparently adequate posterior summaries do not necessarily support accurate evidence estimation.

3. Estimator development: study generalized identities that introduce carefully chosen auxiliary densities. Such variants require their own conditions and validation; changing the name or adding a fitted density does not itself establish reliability.

Think of it this way

Think of estimating a total using measurements that are usually modest but occasionally enormous. A long run without an enormous contribution can look stable, then one rare observation changes everything. This analogy describes reciprocal-likelihood behavior. It does not mean the low-likelihood draw is necessarily erroneous or should be deleted as an outlier.

Worked example

Suppose four illustrative posterior draws have likelihood values 0.2, 0.1, 0.2 and 0.1. Their reciprocals are 5, 10, 5 and 10. The average reciprocal is 30/4=7.5, so the harmonic estimate of evidence is 1/7.5, approximately 0.13333.

Now append one draw whose likelihood is 0.0001. Its reciprocal is 10,000. The new estimate is 5/(30+10,000), approximately 0.0004985. A single additional value has reduced the estimate by a factor of about 267.47.

The arithmetic mean of the original four likelihoods would instead be 0.15, but that is not the harmonic mean estimator and is not an evidence estimator simply because the draws came from the posterior. The example does not specify a full Bayesian model or claim to simulate typical posterior draws. It isolates the formula's sensitivity; neither of its two harmonic estimates is declared to be the true marginal likelihood.

Illustrative case study

Fictional case: Sequoia Modeling compares two forecasting models using saved posterior samples. Its analysts report a large evidence ratio from harmonic means, but repeated runs occasionally produce sharp downward jumps when a low-likelihood draw appears. Removing those draws makes the report smoother without providing a valid statistical justification.

The team withdraws the evidence comparison pending an appropriate estimation method and checks a simpler model with known evidence. It preserves the original draws for investigation. This fictional case illustrates why numerical tidiness and stable-looking output are not evidence that the integral was estimated accurately.

Common misconceptions and mistakes

The harmonic mean is not the arithmetic mean of likelihoods and not the inverse of the mean log likelihood. Posterior parameter convergence diagnostics do not automatically diagnose the tail behavior of reciprocal likelihoods. A large nominal sample size or agreement across several runs can still be misleading.

Consistency is an asymptotic statement under appropriate conditions, not a useful finite-sample error bound. Improper priors make ordinary marginal-likelihood comparisons problematic because their normalization is unspecified. Deleting influential draws without deriving a valid alternative changes the estimator. Finally, good evidence computation cannot remove the substantive sensitivity of model comparison to the chosen priors.

Business use

For business analysis, treat the original harmonic mean estimator as a method requiring strong justification rather than a default shortcut. If model selection depends on evidence, use an approach appropriate to the model and validate its numerical behavior against reference problems where possible. Alternatives include bridge sampling or other dedicated evidence methods, each with its own requirements.

Clarify whether the decision really needs marginal likelihood or instead concerns predictive performance. Those are different questions. Document priors, retained likelihood constants, sampling procedures and evidence-estimation uncertainty so that a precise-looking model score does not obscure a fragile calculation.

Check your understanding

Why should the fifth likelihood in the worked example not simply be discarded?

Answer: Its influence follows from the estimator's definition. Without a justified alternative method, removing it hides the instability and changes the target calculation.

Related terms

Bayes factor
Posterior sampling
Importance sampling
Marginal likelihood
Bayesian evidence
proper prior
bridge sampling
Monte Carlo error

Learn next

Derive the reciprocal-likelihood identity from Bayes' rule and calculate the estimator on a toy sample. Then compare it with exact evidence in a conjugate model and study why tail behavior controls importance-sampling reliability.

Importance sampling

Sources

Harmonic authors, original harmonic mean estimator documentation: https://astro-informatics.github.io/harmonic/background/Harmonic_Estimator/original_harmonic_mean.html

Neal, author analysis of harmonic-mean evidence estimation: https://radfordneal.wordpress.com/2008/08/17/the-harmonic-mean-of-the-likelihood-worst-monte-carlo-method-ever/

Independent review consulted Harmonic’s reciprocal-evidence derivation and Neal’s original instability analysis. Original five-value sensitivity calculation verified; case fictional.

Previous: Harmonic mean Next: Harmonization