Does p = 0.02 mean a 2% chance the null is true?

No. It is a tail probability under the specified null model.

Does p above 0.05 prove no difference?

No. The data may be too imprecise to distinguish meaningful alternatives.

Is a smaller p-value always a larger effect?

No. Sample size and variability also influence the value.

AI Glossary / Statistics and probability / Intermediate

P-value

Definition

A p-value is a probability calculated under a specified null model: the probability of obtaining a test statistic at least as extreme as the observed value, using the test's defined ordering of extremeness. It describes the compatibility of the data with that model and its assumptions. It is not the probability that the null hypothesis is true, nor the probability that a finding is merely due to chance.

What it means

A statistical test asks what outcomes would look like if the null model applied. The p-value locates the observed result within that reference distribution. A small value indicates that the chosen statistic is unusual under the specified model.
Interpreting this requires more than the null hypothesis alone. Sampling, independence, measurement, model form, and analysis choices can affect validity. A surprising result can reflect a difference of interest, a violated assumption, selective analysis, or other problems. The p-value does not identify which explanation is correct.

How it works

State the question, null model, test statistic, and whether the test is one-sided or two-sided before examining results where possible. Specify how outcomes are ordered from less to more extreme.
Use the null distribution of the statistic to calculate an exact or approximate tail probability. Some tests account for nuisance parameters through additional methods, and discrete tests may not achieve every possible p-value. Different legitimate tests can produce different values because they use different information or assumptions.
Interpret the result alongside the estimated effect, uncertainty, study design, and other evidence. A very small effect can produce a small p-value with enough data, while an important effect can remain uncertain in a small study.
Account for multiple comparisons and data-dependent decisions. Repeatedly trying analyses or stopping when a threshold is crossed can invalidate the interpretation of an ordinary fixed-design p-value unless the procedure addresses those choices. Transparent reporting should make the analysis path and relevant limitations visible.

Three practical examples

A randomized experiment uses a prespecified test to assess whether an observed outcome difference is compatible with a no-effect model.
A quality-control study tests a process parameter while also reporting the estimated deviation and its practical consequences.
An analyst evaluates a model assumption using a diagnostic statistic, recognizing that the test result does not replace examination of the data and context.

Think of it this way

Think of asking how unusual a measured result would be under a particular reference model. The answer describes the result relative to that model. It does not directly tell you how likely the model itself is, just as an unusual event under one explanation does not automatically establish another explanation.

Worked example

Consider ten independent coin tosses under the null model that the probability of heads is 0.5. Suppose nine heads are observed, and a one-sided test for excess heads was specified in advance.
There are 2^10 = 1,024 equally likely toss sequences under the null. Exactly nine heads occur in ten sequences, and ten heads occur in one. Therefore the one-sided p-value is P(H ≥ 9) = (10 + 1) / 1,024 = 11 / 1,024, approximately 0.01074.
For a two-sided test using distance from five heads as the extremeness rule, outcomes at least as extreme are zero, one, nine, or ten heads. Their counts are 1 + 10 + 10 + 1 = 22, giving p = 22 / 1,024, approximately 0.02148.
These are probabilities of specified tail events assuming the fair-coin model and independence. Neither number is the probability that the coin is fair. The one-sided value is also not the probability of exactly nine heads, which is 10 / 1,024. Choosing the favorable test direction only after observing the result would change the testing procedure and needs separate treatment.

Illustrative case study

In this fictional case, Maple Service Research evaluates a change to an appointment reminder. Its team initially focuses on whether a p-value is below a conventional threshold.
Reviewers ask for the estimated difference, uncertainty interval, number of outcomes examined, and the original analysis plan. They find that several exploratory comparisons were made after seeing the data. The report distinguishes those explorations from the prespecified comparison and avoids presenting all results as equally confirmatory.
The organization considers whether the plausible benefit is operationally meaningful and whether further evidence is needed. The final decision is not reduced to a single numerical cutoff.

Common misconceptions and mistakes

A p-value is not P(null hypothesis is true given the data). It also does not give the probability that a result will replicate or the probability of a false positive for this particular finding.
A large p-value does not prove no effect or equivalence. A small p-value does not measure effect size, practical importance, or freedom from bias. Values just above and below a threshold should not be treated as radically different evidence without context.
Finally, reporting only the smallest value from many unreported analyses can produce a misleading impression of compatibility with the null model.

Business use

P-values can contribute to evaluating experiments and model checks when the design and assumptions support their use. Businesses should pair them with effect estimates, uncertainty, operational costs, and decision consequences.
Define the analysis and decision criteria clearly. Distinguish exploratory work from confirmatory tests, account for repeated comparisons, and avoid equating statistical detectability with business value. Where evidence is inconclusive, report the remaining uncertainty rather than declaring that nothing matters.

Check your understanding

What is the example's one-sided p-value?

Answer: 11 / 1,024, about 0.01074.

Why is the two-sided value larger here?

Answer: Its tail event includes equally extreme outcomes in both directions.

What else should accompany a p-value?

Answer: Effect estimates, uncertainty, design information, assumptions, and relevant analysis choices.

Related terms

Learn next

Learn next how significance levels differ from observed p-values. Study power, interval estimation, and multiplicity, then compare frequentist testing with Bayesian probability statements without conflating their interpretations.

Bayesian probability

Sources

American Statistical Association, Statement on Statistical Significance and P-Values: https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf Wasserstein and Lazar, The ASA Statement on p-Values: Context, Process, and Purpose: https://www.tandfonline.com/doi/full/10.1080/00031305.2016.1154108 Harrell, Glossary of Statistical Terms: https://hbiostat.org/glossary/

Previous: P/T ratio Next: PAC learning