Does a large p-value mean the model is accurate?

No; the test may lack power or conceal local discrepancies.

Can it replace a calibration plot?

No. The plot helps show direction and size of error.

Is it a discrimination test?

No. It concerns agreement between predicted and observed outcomes after grouping.

AI Glossary / Statistics and probability / Intermediate

Hosmer - Lemeshow test

Definition

The Hosmer-Lemeshow test is a grouped goodness-of-fit test commonly used with binary logistic regression. It compares observed event and non-event counts with counts expected from predicted probabilities across groups of similar predictions. A small p-value indicates evidence of disagreement under the test's assumptions; a large p-value does not establish that predictions are well calibrated.

What it means

A model predicting a 20 percent event probability should produce roughly 20 events per 100 comparable cases over repeated observations. This test checks such agreement after collecting cases into probability groups.
It reduces a detailed calibration question to a single statistic and p-value. That reduction can be convenient, but it hides where errors occur and how large they are. Two models can receive similar test results while making very different practical errors.

How it works

Start with binary outcomes and corresponding predicted probabilities for the same observations. Confirm the event coding: reversing the outcome without reversing probabilities invalidates the comparison.
Sort observations by predicted probability and divide them into groups, often using ten approximately equal-sized groups. Quantile boundaries and tied predictions can change the actual number and composition of groups. Software warnings about collapsed groups matter.
Within group j, calculate its size n_j, observed event count O_j, expected event count E_j, and average predicted probability p_j = E_j/n_j. Expected non-events equal n_j minus E_j; observed non-events equal n_j minus O_j.
Add the Pearson discrepancies for both event and non-event counts. Equivalently, each group's contribution is (O_j - E_j)^2 divided by n_j*p_j*(1 - p_j), provided the denominator is positive. Predictions of exactly zero or one require particular care.
The familiar implementation for fitted logistic regression compares the statistic with an approximate chi-squared distribution using g minus 2 degrees of freedom. This is an approximation associated with a particular testing procedure, not a universal rule for every grouped probability comparison.
Do not automatically transfer that reference distribution to arbitrary machine-learning models, heavily penalized fits, dependent observations, or externally supplied predictions. Establish which testing assumptions apply and whether a more suitable validation method is needed.
Small expected counts can undermine the approximation. With small samples, important discrepancies can go undetected; with very large samples, minor discrepancies can become statistically detectable. Group boundaries can also change the result.
Inspect observed and predicted rates alongside the test. Calibration plots, calibration intercept and slope where appropriate, and useful probability-error measures help describe the magnitude and direction of disagreement. Evaluate generalization on appropriately separated data rather than relying only on the training sample.

Three practical examples

An analytics team checks whether a logistic model's predicted subscription cancellations agree with observed cancellations across probability groups.
An AI evaluator compares predicted binary task-failure probabilities with recorded outcomes, first checking whether the chosen test applies to the model and sample.
A quality team supplements a grouped fit test with a calibration plot to locate systematic underprediction.

Think of it this way

Think of sorting weather forecasts into probability baskets and comparing the rain expected in each basket with the rain observed. One alarm can flag disagreement, but opening the baskets shows where it occurred. Quiet baskets do not prove every forecast was accurate.

Worked example

Consider four groups of 20 observations each. Within each group, predictions are constant at 0.1, 0.3, 0.6, and 0.8 respectively. Expected event counts are therefore 2, 6, 12, and 16. Suppose observed counts are 1, 8, 11, and 17.
The contributions are (1 - 2)^2/(20*0.1*0.9) = 5/9; (8 - 6)^2/(20*0.3*0.7) = 20/21; (11 - 12)^2/(20*0.6*0.4) = 5/24; and (17 - 16)^2/(20*0.8*0.2) = 5/16.
Their sum is 2045/1008, approximately 2.02877. The second group has observed rate 8/20 = 0.4 against predicted rate 0.3, showing underprediction there.
This deliberately small example illustrates the arithmetic only. It has a low expected event count and an illustrative grouping scheme, so no inferential p-value or model-acceptance decision is claimed.

Illustrative case study

In this fictional case, Cedar Subscriptions initially describes a cancellation model as validated because its grouped test is not significant. An analyst plots observed against predicted rates and finds a commercially important gap in a small high-probability segment. The team reviews uncertainty and gathers more observations before recalibrating. Its report includes the group counts and calibration pattern, avoiding a blanket claim that a non-significant test proves reliable predictions.

Common misconceptions and mistakes

A non-significant result is not proof of good fit. Statistical significance does not measure the business cost of an error. A high discrimination score does not guarantee calibration: ranking cases and assigning accurate probabilities are different tasks.
Ten groups are a convention, not a remedy for sparse data or dependence. Changing groups after seeing results to obtain a favorable p-value undermines interpretation. A significant result also does not identify which predictors or modeling choices caused the discrepancy.

Business use

Use the test as one diagnostic when its assumptions and implementation are appropriate. For operational decisions, report probability discrepancies in understandable units, such as extra expected cancellations per thousand customers, together with uncertainty. Record the validation population, grouping rule, sample size, event count, and software settings so future checks are comparable.

Check your understanding

Why can changing the groups change the result?

Answer: Grouping changes which discrepancies are combined and which can cancel within a group.

In the worked example, which group underpredicts by ten percentage points?

Answer: The second: observed 0.4 versus predicted 0.3.

Related terms

Learn next

Learn calibration plots and the distinction between calibration and discrimination. Then study validation design, calibration intercept and slope, and the limitations of significance-based model assessment.

Sources

ResourceSelection documentation, hoslem.test: https://search.r-project.org/CRAN/refmans/ResourceSelection/html/hoslem.test.html BMJ, Evaluation of clinical prediction models (part 2), methodological discussion of grouped calibration testing: https://www.bmj.com/content/384/bmj-2023-074820

Previous: Horvitz - Thompson estimator Next: Hostile Attribution Bias