Does failure to reject validate the model?

No. The sample may have limited ability to detect important departures.

Is every goodness-of-fit test chi-square based?

No. Several methods compare distributions or model implications differently.

Can a tiny discrepancy be significant?

Yes, with enough independent data. Practical importance must be assessed separately.

AI Glossary / Statistics and probability / Intermediate

Goodnessoffit test

Definition

A goodness-of-fit test evaluates whether observed data are sufficiently inconsistent with a specified statistical model to provide evidence against it. In the frequency-based sense emphasized here, it compares observed category counts with counts expected under a proposed probability distribution. Pearson's chi-square goodness-of-fit test is a common example. A test result measures evidence about a particular model under stated assumptions; it does not prove that the model is true or useful for every purpose.

What it means

A model predicts a pattern, while a sample produces a noisy realization of that pattern. Even a correct model rarely predicts every observed count exactly. A goodness-of-fit test asks whether the discrepancy is unusually large relative to sampling variation under the model.

For example, a service team might compare observed request types with a prespecified expected mix. The relevant question is whether the count pattern is compatible with that proposed mix, not whether each request was individually classified correctly.

The phrase also covers other tests, including tests that compare a continuous empirical distribution with a theoretical distribution. Different tests emphasize different kinds of discrepancy and require different reference calculations.

How it works

For a categorical Pearson test, specify k mutually exclusive, exhaustive categories and null probabilities p_i that sum to one. With sample size n, expected counts are E_i = n*p_i. Compare them with observed counts O_i using X² = sum[(O_i - E_i)²/E_i]. Categories with positive expected counts must be handled consistently.

Under suitable independent multinomial sampling and adequate expected counts, the statistic has an approximate chi-square reference distribution. If the probabilities are fully specified in advance, the usual degrees of freedom are k - 1. Estimating parameters from the same data changes the reference calculation; subtracting the number of fitted parameters is justified only under appropriate estimation and regularity conditions.

Choose categories and methods deliberately, check sparse counts and dependence, and inspect the discrepancy itself. Exact or simulation-based methods may be preferable when the usual approximation is unsuitable. A continuous-distribution test likewise needs an adjustment if its reference parameters were estimated rather than known.

Three practical examples

Service monitoring: compare observed request-category counts with a documented expected distribution, investigating which categories contribute to a discrepancy.
Simulation assessment: check whether simulated outputs follow an intended categorical distribution, using separate checks for dependence and implementation correctness.
AI data review: evaluate a proposed probability model for a dataset as one diagnostic, without mistaking acceptable fit for proof of representative sampling or predictive accuracy.

Think of it this way

Think of a recipe that specifies a mixture of colored beads. A random scoop will not reproduce the recipe's proportions perfectly. A goodness-of-fit test asks whether the scoop differs more than chance variation would reasonably explain, provided the scooping process matches the assumed sampling model. It does not establish that the recipe is the best one for the intended product.

Worked example

Consider an original example with three request types and prespecified probabilities 0.40, 0.40, and 0.20. In 100 independent requests, the observed counts are 50, 30, and 20. Expected counts are 40, 40, and 20, all sufficiently large for this approximation.

The statistic is X² = (50 - 40)²/40 + (30 - 40)²/40 + (20 - 20)²/20 = 2.5 + 2.5 + 0 = 5. No probability parameters were estimated from these observations, so the reference degrees of freedom are 3 - 1 = 2.

For a chi-square distribution with two degrees of freedom, the upper-tail probability at five is exp(-5/2) = 0.08208. At a prespecified 5% significance level, the test does not reject the proposed distribution. It does not establish an 8.2% chance that the null model is true, and it does not prove the differences are operationally negligible.

As an independent sensitivity check, double every count while retaining the same proportions. The statistic becomes ten and its two-degree-of-freedom upper-tail probability becomes exp(-5), approximately 0.00674. The same proportional discrepancy can therefore provide stronger evidence with more independent observations. Direct summation and the analytic tail formula verify the calculations.

Illustrative case study

In a fictional support center, managers specify a request-type mix before reviewing a new month's records. A goodness-of-fit test suggests a discrepancy, with most of the statistic coming from an increase in technical requests.

Analysts inspect the category definitions and discover that a routing-policy change may have reclassified some messages. They compare records under consistent definitions before concluding that customer needs changed. They also check repeated requests from the same incidents. The fictional case illustrates why statistical disagreement prompts investigation; it does not identify the business cause or automatically validate a staffing response.

Common misconceptions and mistakes

Common mistakes include interpreting a large p-value as proof of fit, treating the p-value as the probability that the model is correct, and ignoring how sample size affects sensitivity.

For chi-square tests, do not substitute percentages for counts, overlook small expected frequencies, or use the unadjusted degrees of freedom after fitting model parameters. Binning continuous data differently can change the result, so opportunistic bin selection undermines interpretation.

A goodness-of-fit test is also distinct from a chi-square independence test, although both may use a Pearson statistic. One compares a distribution with specified model expectations; the other tests a relationship structure in a contingency table. Clarify the null hypothesis before interpreting either result.

Business use

Use the test as one part of model assessment. Report the null model, sample size, count table or discrepancy plot, statistic, reference method, and limitations. Relate detected deviations to practical tolerances and decision costs. For AI systems, combine distributional checks with task-specific evaluation, subgroup analysis, and monitoring of changes over time.

Check your understanding

Why does the worked example use two degrees of freedom rather than three?

Answer: The three counts must sum to the fixed total, leaving only two freely varying counts when the probabilities are specified. Estimating additional model parameters can impose further restrictions and requires an appropriate adjusted reference calculation.

Related terms

Learn next

Learn multinomial counts and chi-square distributions, then study parameter-estimation adjustments and simulation-based testing. Practice separating evidence against a model from the size and practical relevance of its discrepancies.

Sources

NIST/SEMATECH, Chi-Square Goodness-of-Fit Test: https://www.itl.nist.gov/div898/handbook/eda/section3/eda35f.htm — primary handbook formulas, binning, and fitted-parameter issues. SciPy, chisquare: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chisquare.html — primary implementation documentation for multinomial goodness-of-fit and degrees-of-freedom adjustments. The supplied term's soft-hyphen characters are retained in its title; the prose uses ordinary hyphenation. Numerical examples are original.

Previous: Goodness-of-fit Next: GOP (group of pictures)