Does the test require one model to be true?

Its framework can compare misspecified models.

AI Glossary / Statistics and probability / Advanced

Vuong's closeness test

Definition

Vuong's closeness test compares statistical models by their expected log likelihood, which corresponds to relative closeness to the data-generating distribution in Kullback-Leibler terms. In a common regular nonnested setting, it tests whether two fitted models are equally close rather than whether either is exactly true. A standardized sum of observation-level log-likelihood differences provides the test statistic. Different model relationships require different theory, so the familiar normal-reference formula is not valid for every pair of models that happens to be described as nonnested.

What it means

Both competing models may be imperfect. The question is whether one has a higher expected log density for data from the target process. This is a relative comparison, not a goodness-of-fit certificate for the winner. A nonsignificant result does not prove that the models are identical or equally useful for every prediction. Distinguishability, nesting or overlap, parameter estimation and the assumptions about independent observations all affect which version of the test is appropriate.

How it works

Fit both models to the same observations using compatible likelihood definitions. For each observation, calculate mi = log f1(yi) minus log f2(yi), evaluated at the fitted parameters and conditioning on the same relevant information. In the regular distinguishable nonnested case, form the sample mean mbar and its observation-level standard deviation, then compute V = sqrt(n) times mbar divided by that standard deviation. Under the equal-closeness null and suitable regularity conditions, V has an asymptotic standard normal reference. Complexity-adjusted variants exist, but their penalties and inferential targets must be stated rather than mixed with the unadjusted formula.

Three practical examples

Example 1: An analyst compares two different probabilistic descriptions of the same outcome. Example 2: An AI researcher compares fitted density models using observation-level likelihood contributions. Example 3: A business study examines whether one nonnested response model is closer in expected log-likelihood terms. These examples require compatible data and a valid model-class comparison. Comparing likelihoods computed on different outcomes, different observations or inconsistent normalizing constants does not produce a meaningful Vuong statistic.

Think of it this way

Think of two imperfect maps scored by how well each describes the same sequence of locations. The test examines whether the average score advantage is large relative to its variability. It does not declare the better-scoring map exact, and it does not automatically answer which map is best for every route or operational objective.

Worked example

Suppose two appropriately fitted, distinguishable nonnested models are compared on 100 independent observations under the conditions for the standard asymptotic test. Their observation-level log-likelihood differences, model one minus model two, have mean 0.03 and standard deviation 0.15, using the specified n-divisor convention. The statistic is sqrt(100) times 0.03 divided by 0.15 = 2.0. Against a standard normal reference, the illustrative two-sided p-value is approximately 0.0455. The positive sign favors model one in expected log-likelihood terms. This calculation assumes the theoretical conditions and summaries; it does not verify them. It is unadjusted for differences in model complexity, and its asymptotic p-value is not an exact finite-sample guarantee or a probability that model one is true.

Illustrative case study

In this fictional case study, Linden Analytics compares two probabilistic service-demand models. Analysts retain each observation's likelihood contribution and first examine whether the models meet the relevant distinguishability and regularity conditions. They report the relative comparison alongside predictive checks and operational error measures. When one model is favored, they still inspect its calibration and tail behavior. The team avoids using a single relative test as proof that the selected model describes demand adequately in every deployment scenario.

Common misconceptions and mistakes

Common mistakes include treating the test as a generic replacement for all likelihood-ratio tests, applying the normal reference to overlapping or observationally indistinguishable models without checking theory, and interpreting a winner as true. Another mistake is comparing only total log likelihoods without their observation-level variability. Dependence, boundary parameters and misspecified likelihood contributions can invalidate the simple calculation. Data-driven model searching also changes the interpretation of a final pairwise comparison.

Business use

Businesses can use the test as one part of probabilistic model comparison when its conditions are appropriate. State which models, data, likelihood definitions and adjustments were used. Pair the relative result with validation of the quantities the decision needs, such as calibrated probabilities or relevant tail predictions. If observations are dependent or models overlap, seek a method justified for that setting. A statistically detectable log-score advantage may still be operationally small, while an inconclusive test can reflect limited information rather than practical equivalence.

Check your understanding

What statistic follows from n = 100, mean difference 0.03 and standard deviation 0.15?

Answer: Two.

Which model does its sign favor?

Answer: Model one.

What must be checked before using the normal reference?

Answer: The relevant model relationship, distinguishability, regularity and sampling assumptions.

Does the result certify absolute model adequacy?

Answer: No; it is a relative comparison.

Related terms

Kullback-Leibler divergence
Related terms include likelihood ratio
nonnested model
model distinguishability
AIC
BIC and predictive validation

Learn next

Learn next how observation-level log scores relate to expected predictive performance and KL distance. Study why overlapping and nested model families can have different limiting distributions. Then compare relative hypothesis tests with information criteria and held-out evaluation, keeping their assumptions and purposes distinct.

Sources

Source: Quang H. Vuong, Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses, Caltech author archive, https://authors.library.caltech.edu/records/0gmss-rn517. This primary paper develops relative-closeness tests and distinguishes model relationships. The summary-statistic calculation is original and independently checked; the demand-model case is fictional. The example explicitly assumes the regular unadjusted normal-reference setting. Supporting implementation study: Edgar C. Merkle, Dongjun You and Kristopher J. Preacher, Testing non-nested structural equation models, https://arxiv.org/abs/1402.6720, especially equations 9 and 13. The original archive abstract was accessible; its PDF could not be retrieved for this review.

Previous: VSO language Next: W-shingling