Is it exact for a Gaussian posterior?

Yes, with the correct mode and full curvature, because the Gaussian log density is already quadratic.

Can it represent several separated modes?

A single Gaussian around one mode generally cannot. Mixtures or other inference methods may be needed.

Does a large data set guarantee accuracy?

No. Regularity, identifiability, parameter dimension and model structure matter; more observations alone do not remove every difficulty.

AI Glossary / Probabilistic machine learning / Advanced

Laplace approximation

Definition

A Laplace approximation replaces a probability density near a smooth mode with a Gaussian whose center and curvature match that local peak. In Bayesian inference, it commonly approximates a posterior using its mode and the inverse Hessian of its negative log density.

What it means

A posterior distribution can be difficult to integrate or sample from directly. Yet finding its most likely parameter value may be relatively manageable with numerical optimization.

Laplace approximation uses that mode as a starting point for an uncertainty estimate. It examines how sharply the log density bends nearby: a narrow, steep peak implies a smaller local variance, while a flatter peak implies greater local uncertainty.

The approximation is local, so it can miss asymmetry, heavy tails, boundaries and other modes. It is not an assertion that the true posterior is Gaussian. The same underlying quadratic-expansion idea can also approximate integrals and normalizing constants, with the relevant scaling factors retained.

How it works

First find an interior stationary mode of the log target density. Expand the log density to second order around that point. The first-order term vanishes at an exact stationary point, leaving a constant and a quadratic term.

Let H be the Hessian of the negative log density at the mode. If H is positive definite, the corresponding Gaussian has that mode as its mean and covariance H inverse. Equivalently, use the negative inverse of the log-density Hessian.

For constrained parameters, an unconstrained transformation can improve the setup, but the transformed density must include the appropriate Jacobian adjustment. Transforming samples back produces a distribution on the original scale that need not be Gaussian. Approximation quality can depend on the chosen parameterization.

Three practical examples

A Bayesian regression analysis uses local curvature around a fitted posterior mode to approximate parameter uncertainty.
A latent-variable model uses a Laplace-based integral approximation when direct integration is expensive.
A modeling team compares a quick Gaussian approximation with more intensive posterior sampling on selected validation problems.

Think of it this way

Think of replacing the top of a hill with a smooth dome that matches its location and curvature. The dome can describe the immediate neighborhood while missing an uneven slope or a second hill farther away.

Here the hill is the log density, not the density itself. The resulting probability approximation is Gaussian because exponentiating a concave quadratic gives a Gaussian-shaped function. Matching a peak does not reproduce the entire landscape.

Worked example

Comparing local curvature with exact uncertainty

Consider an original calculation for a positive parameter theta whose posterior density is proportional to theta to the fourth power times exp(−2 theta). This is a Gamma distribution with shape 5 and rate 2.

Ignoring a constant, its log density is 4 log theta − 2 theta. The first derivative is 4/theta − 2, which is zero at theta = 2. The second derivative is −4/theta squared, equal to −1 at the mode.

The direct Laplace approximation on the theta scale is therefore Normal(mean 2, variance 1). The exact Gamma mean is 5/2 = 2.5 and its variance is 5/4 = 1.25. The approximation captures the local peak but does not match the exact mean or variance.

It also assigns some probability to negative theta values, which the true distribution forbids. A log transformation could produce a different approximation that respects positivity after transforming back, but it requires including the change-of-variables factor and finding the mode on that transformed scale. Simply taking the logarithm of the original mode is not that complete procedure.

Illustrative case study

Fictional case: checking a fast posterior approximation

A fictional analytics team models equipment measurements with uncertain parameters. It wants a quick preliminary uncertainty calculation before running a more expensive inference method.

The team finds a posterior mode, checks the curvature matrix and constructs a Laplace approximation. On smaller versions of the problem, it compares approximate quantiles and predictions with numerical integration or well-diagnosed sampling.

When one parameter has a strongly skewed posterior, the team reports that limitation and revisits the parameterization. It does not interpret a successful optimizer message as proof that the uncertainty approximation is accurate. The fictional case describes an evaluation process, not a measured speed or accuracy gain.

Common misconceptions and mistakes

The covariance is the inverse of the negative-log-density Hessian, not the Hessian itself. Confusing precision with covariance reverses the relationship between sharpness and uncertainty.

The Gaussian's mean equals the chosen mode; the true posterior mean generally does not. A mode is also parameterization-dependent, so transformed and untransformed approximations need not agree.

A singular or indefinite curvature matrix is not a usable full covariance under the basic formula. It may indicate a saddle point, non-identifiability, boundary behavior or insufficient numerical convergence. Adding an arbitrary diagonal adjustment changes the approximation and should not be presented as an exact consequence of the original posterior.

Business use

Laplace approximation can be a useful baseline when the posterior is dominated by one reasonably regular peak and derivative calculations are feasible. It is especially helpful when a fast approximate answer can guide further analysis.

Record the parameter scale, optimizer, convergence diagnostics, curvature approximation and any regularization of the matrix. Check predictions and uncertainty in regions relevant to the intended decision, particularly tails or constrained parameters. In large models, diagonal or structured curvature approximations introduce additional assumptions beyond the basic Laplace method and should be identified explicitly.

Check your understanding

If a one-dimensional negative log density has second derivative 25 at its mode, what variance and standard deviation does the basic Laplace approximation use?

Answer: Variance 1/25 = 0.04 and standard deviation 0.2, assuming the mode is suitable and the local approximation is justified.

Related terms

Hessian
Posterior distribution
maximum a posteriori estimation
Gaussian approximation
precision matrix
marginal likelihood
change of variables

Learn next

Compare a Laplace approximation with exact numerical integration for a skewed one-dimensional density. Then repeat on a transformed parameter scale, including the Jacobian, and examine how the approximations differ.

Sources

Primary implementation reference: Stan Reference Manual, Laplace Approximation: https://mc-stan.org/docs/reference-manual/laplace.html

Primary teaching discussion: Roger D. Peng, Advanced Statistical Computing, section 5.1: https://bookdown.org/rdpeng/advstatcomp/laplace-approximation.html

The specified Gamma curvature calculation and equipment-modelling case are original teaching examples.

Previous: Language resource Next: Laplace's equation