Is goodness of fit the same as predictive accuracy?

No. Agreement with fitted data and performance on new data are related but distinct questions.

Does a large p-value prove a model?

No. It indicates a lack of strong evidence against the tested model under that test's assumptions.

No. Residual patterns and other diagnostics can reveal problems that it misses.

AI Glossary / Data science and analytics / Intermediate

Goodness-of-fit

Definition

Goodness of fit describes how well a statistical model agrees with the data being examined. It can be assessed with graphs, numerical summaries, and formal statistical tests. The appropriate assessment depends on the model and the question. A model that fits observed data well is not thereby proven true, useful for every decision, or accurate on new data.

What it means

A model describes an expected pattern, while actual observations usually differ from that pattern. Goodness-of-fit assessment examines those differences. For a proposed probability distribution, the question may be whether observed frequencies are compatible with the distribution. For a regression, it may involve checking the size and pattern of residuals, the differences between observations and fitted values. There is no single score that captures every kind of agreement or every important model assumption.

How it works

First state the model and identify what it predicts about the data. Compare those predictions with observations using diagnostics suitable for the data type and sampling process. For example, a chi-square goodness-of-fit test compares category counts with expected counts. Residual plots can reveal systematic patterns left unexplained by a regression. Formal tests require their own conditions, including a suitable treatment of estimated parameters and dependence. Their p-values describe how unusual a discrepancy would be under the tested model and assumptions; they are not probabilities that the model is true.

Three practical examples

Operations: an analyst checks whether a proposed distribution of incoming requests adequately describes recorded counts before using it in a simulation.
Forecast modeling: a team examines residuals over time to see whether a fitted model consistently misses seasonal patterns.
Quality analysis: observed category frequencies are compared with a specified set of expected proportions to investigate a possible mismatch.

Think of it this way

Think of checking how a pattern fits a piece of fabric. Some areas may line up while others pull away, and a single overall measurement can hide a local mismatch. Unlike tailoring, statistical data contain random variation. Small differences can occur even under a suitable model, so the assessment must distinguish expected variation from evidence of a meaningful problem.

Worked example

Suppose a process is specified to produce three categories in proportions 50%, 30%, and 20%. In 100 independent observations, the expected counts are therefore 50, 30, and 20. The observed counts are 45, 35, and 20. The chi-square discrepancy is the sum of (observed minus expected) squared divided by expected: 25/50 + 25/30 + 0/20 = approximately 1.33.
For this illustration, the category probabilities were specified before observing the sample, and none were estimated from it. With three categories, the conventional reference distribution has two degrees of freedom. The approximate upper-tail p-value is 0.51. At a prespecified 5% significance level, the test would not reject the stated proportions. That result does not establish that the proportions are exactly correct or that the process is stable over time. It means this test has not found sufficient evidence of disagreement in this sample under its assumptions. A different sample size or discrepancy could lead to another conclusion.

Illustrative case study

In a fictional delivery project, a fitted model has a high overall R-squared, so an initial report calls it accurate. A reviewer plots residuals against journey length and notices that long trips are systematically underestimated. The team investigates the model's functional form and checks performance on a separate period before deciding whether to revise it. The example shows why a favorable summary score can coexist with a practically important lack of fit. It reports no measured operational improvement or guarantee that a revised model will generalize.

Common misconceptions and mistakes

Failure to reject a goodness-of-fit test is not proof of the model. A small sample may have limited ability to reveal departures, while a large sample may detect departures too small to matter operationally. A high R-squared does not verify every regression assumption. Fitting and evaluating on the same data can also make agreement look more favorable than future predictive performance. For count tests, do not confuse observed counts with expected counts when assessing approximation conditions. Changing bins after inspecting results can affect conclusions and should be disclosed.

Business use

Ask which kinds of mismatch would change the business decision. A simulation may need realistic tail behavior even when the center of its fitted distribution looks satisfactory. A planning model may need acceptable errors during peak periods rather than only a favorable average. Document the fitted model, data window, diagnostics, and assumptions. Pair statistical evidence with the practical size of errors. If the purpose is prediction, add an appropriate evaluation on data not used for fitting, rather than treating goodness of fit as the complete validation process.

Check your understanding

The example produces a p-value of about 0.51. Can the analyst report a 51% probability that the specified proportions are true?

Answer: No. The p-value is calculated assuming the stated model and describes the probability of a discrepancy at least as large as the observed one under the test's conditions.

Related terms

Learn next

Practice calculating expected counts from stated category probabilities, then inspect residual plots from a simple regression. Compare statistical significance with practical error size. Learn why estimating distribution parameters from the same data can change the reference distribution of a goodness-of-fit statistic.

Sources

NIST, chi-square goodness-of-fit test and assumptions: https://itl.nist.gov/div898/handbook/eda/section3/eda35f.htm NIST, model fit and residual diagnostics: https://itl.nist.gov/div898/handbook/pmd/section4/pmd44.htm Philip B. Stark, original statistics teaching material on chi-square testing: https://www.stat.berkeley.edu/pub/users/stark/SticiGui/Text/chiSquare.htm Stark, hypothesis testing and p-value interpretation: https://statgrades.berkeley.edu/~stark/SticiGui/Text/testing.htm

Previous: Goodman and Kruskal's gamma Next: Goodnessoffit test