Does a large p-value prove there is no effect?

No. The data may be too imprecise to distinguish meaningful alternatives.

Are p-values always inappropriate?

No. Properly chosen tests can support inference when their assumptions and interpretation are respected.

Can repeated monitoring matter?

Yes. Unplanned stopping based on ordinary fixed-sample p-values can change the procedure's error rate.

AI Glossary / Statistics and probability / Intermediate

Misuse of p-values

Definition

Misuse of p-values means calculating, selecting, or interpreting p-values in ways that support conclusions the statistical procedure does not justify. Examples include treating a p-value as the probability that a hypothesis is true, reporting only favorable tests, or equating statistical significance with practical importance. Correct interpretation requires the study design, statistical assumptions, analysis choices, and substantive question.

What it means

A p-value describes how unusual a specified test statistic would be under a particular null model, counting outcomes at least as extreme according to the test's definition. That calculation assumes the model and testing procedure used to obtain it.
It does not reverse the conditioning to tell us the probability that the null hypothesis is true after observing the data. Nor does it tell a business how much an intervention is worth. Those questions require additional reasoning and information.

How it works

Misuse can begin before the number is calculated. An analyst may try many outcomes, subgroups, exclusions, transformations, or stopping times and report only the smallest p-value. The reported calculation then fails to represent the complete selection procedure unless that procedure is appropriately accounted for.
Misuse can also occur with a correctly calculated result. A small p-value may reflect a very precise estimate of a tiny effect. A large value may accompany an imprecise estimate that remains compatible with important effects in either direction. Neither result alone settles practical importance.
A sound workflow specifies the question, outcome, comparison, and analysis before inspecting the relevant results when possible. It distinguishes confirmatory analyses from exploration, records departures from the plan, and handles multiplicity or sequential monitoring using suitable methods.
Report the effect estimate, its uncertainty, relevant diagnostics, and the actual analysis choices alongside the p-value. Examine whether independence, measurement quality, sampling, and model assumptions are credible. A numerical threshold cannot repair confounding or biased data collection.
Threshold-based decisions can be useful when embedded in a justified procedure with understood error rates and consequences. The misuse is treating the threshold as a universal truth detector or concealing the choices that produced the result.

Three practical examples

Website experiments: a team searches dozens of engagement metrics and celebrates the sole unadjusted result below 0.05, without revealing the search.
AI evaluation: a tiny accuracy improvement becomes statistically significant in a huge test set but does not reduce consequential errors or deployment costs.
Operational studies: an underpowered pilot produces a large p-value, and managers incorrectly conclude that two workflows are equivalent.

Think of it this way

Think of a p-value as a warning light calibrated for a particular monitoring system. It can signal an unusual reading under that system's assumptions. It does not diagnose every possible cause, estimate the repair's value, or remain calibrated after undisclosed changes to the monitoring rules.

Worked example

Suppose a fictional analyst performs 20 independent, continuous, exactly calibrated tests. All 20 null hypotheses are true, and each test declares significance when p is below 0.05.
The probability that a single test does not cross the threshold is 0.95. Independence makes the probability that none crosses equal to 0.95 raised to the twentieth power, approximately 0.358486. Therefore, the probability of at least one false declaration is 1 - 0.95^20, approximately 0.641514, or 64.15%.
Selecting the smallest p-value and presenting it as the result of one prespecified test hides this family-level error risk. The 64.15% figure depends on the independence and exact calibration assumptions; dependent tests generally require another calculation.
For a separate, prespecified two-sided standard-normal test, an observed statistic of 2 gives p approximately 0.0455. This does not mean that the null has a 4.55% probability of being true. The two calculations answer different questions.

Illustrative case study

Fictional case: Cedar Commerce tests an AI shopping assistant. Its dashboard contains 20 outcomes, but the launch presentation includes only one favorable p-value. An analyst reconstructs the full analysis history and labels the finding exploratory.
The team reports all outcomes, estimated changes, uncertainty, and customer complaints. It then designs a new experiment around a prespecified business outcome and a justified monitoring plan. The revised process does not guarantee success; it makes the evidence and decision rules easier to assess.

Common misconceptions and mistakes

Do not interpret p = 0.04 as a 96% probability that the alternative is true. Do not describe a nonsignificant comparison as proof of equality. Equivalence requires an appropriate question, meaningful bounds, and a suitable analysis.
Avoid claiming that p = 0.049 and p = 0.051 establish sharply different realities. Also, one significant result and one nonsignificant result do not themselves demonstrate a significant difference between the underlying effects; that difference needs a direct assessment.
A smaller p-value is not automatically a larger or more useful effect. Changing to another statistical framework also does not eliminate selective reporting or poor study design.

Business use

Businesses can reduce misuse by retaining experiment plans and complete result histories, defining success measures before testing, and separating discovery from confirmation. Decision documents should explain expected benefits, costs, uncertainty, and possible harms. Review should focus on whether the evidence answers the decision question, rather than rewarding the number of significant findings.

Check your understanding

Why is a selected minimum from many tests different from one planned test?

Answer: Selection changes the relevant probability calculation and error risk.

What should accompany a p-value in a decision report?

Answer: The effect estimate, uncertainty, study context, assumptions, and complete account of analysis choices.

Related terms

Null hypothesis
Statistical significance
Effect size
Confidence interval
p-hacking
multiple testing
statistical power
equivalence testing

Learn next

compare a fixed-sample testing plan with a justified sequential plan, then practice interpreting effect estimates and uncertainty before looking at their p-values.

Fixed sample

Sources

Sources: American Statistical Association, March 2016 release summarizing its six principles: https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf . Original statement by Wasserstein and Lazar: https://doi.org/10.1080/00031305.2016.1154108 . The association's release was read in full; the journal PDF returned an access error. Numerical examples and fictional case are original.

Previous: Misuse detection Next: Mitchell - Netravali