Is a p-value the probability that the null is true?

No; it is calculated assuming the null model and associated assumptions.

Does a significant result imply a large effect?

No. Effect size and uncertainty must be reported separately.

Can a non-significant result establish equivalence?

Not by itself. Equivalence requires a suitably designed analysis with justified margins.

AI Glossary / Statistics and probability / Beginner

Statistical hypothesis test

Definition

A statistical hypothesis test is a procedure for assessing data against a stated hypothesis about a population or probability model. It defines a test statistic and evaluates how unusual its observed value would be under the null hypothesis and other modeling assumptions. A testing framework may specify a rejection rule with a controlled long-run false-positive rate. The result provides evidence or a decision within that framework; it does not prove a hypothesis true or false or establish practical importance by itself.

What it means

The null hypothesis describes the reference claim being tested, such as a success probability of one half. The alternative describes the departures of interest. A one-sided alternative targets a direction, while a two-sided alternative targets departures in either direction.

A p-value summarizes how incompatible the observed statistic is with the null model, using a defined notion of results at least as extreme. It is calculated under that model, not as the probability that the null hypothesis is true. The interpretation also depends on assumptions about sampling, independence, measurement, and any selection or repeated testing.

How it works

Define the scientific or business question, target population, and parameter. State the null and alternative hypotheses before examining results when possible. Choose a statistic and a reference distribution justified by the design and assumptions, using an exact method, approximation, or appropriate resampling procedure.

If a formal decision is needed, specify the significance level alpha and rejection rule. A valid level-alpha procedure controls the probability of rejecting a true null at or below alpha over repetitions under its conditions. Power describes the probability of rejection under a specified alternative; it is not a single property independent of effect size and sample size.

Calculate the statistic and p-value, then report them with effect estimates, uncertainty, design details, and relevant limitations. Consider multiplicity, optional stopping, and analytical choices. A result just above a threshold is not fundamentally opposite to one just below it. Decisions should incorporate consequences, costs, and substantive evidence rather than relying entirely on a binary label.

Three practical examples

1. A quality team tests whether a defect probability exceeds a prespecified reference level under an appropriate sampling design.

2. A randomized website experiment assesses a difference in a defined conversion outcome while reporting the estimated improvement and uncertainty.

3. A laboratory tests a measurement procedure against a reference model, checking assumptions and practical tolerance requirements separately.

Think of it this way

Think of checking whether a set of observations is surprising under a proposed explanation. A surprising result can make that explanation less comfortable, but it does not uniquely identify the correct explanation. A familiar-looking result also does not prove that the proposed explanation is the only possible one.

Worked example

Consider an original fixed-size test with 12 independent binary trials and a common success probability p. The null is p = 0.5, and the alternative, chosen beforehand, is p > 0.5. Suppose 10 successes are observed. The test statistic is the success count X.

Under the null, X follows a binomial distribution with n = 12 and p = 0.5. The one-sided p-value is P(X ≥ 10) = [C(12,10) + C(12,11) + C(12,12)]/2 to the power 12. The numerator is 66 + 12 + 1 = 79, and the denominator is 4,096. Thus the p-value is 79/4,096, approximately 0.01929.

At a prespecified 0.05 threshold, this result falls in the rejection region for this one-sided test. It does not mean there is a 1.929% probability that p = 0.5, nor does it show how large the true improvement is. The observed success proportion is 10/12, approximately 0.833, and its uncertainty remains substantial with only 12 trials.

The calculation depends on fixed sample size, independent trials, and the common-probability model. A rule that stopped early after favorable results would require a test calibrated for that stopping plan. A two-sided question would require a correspondingly defined two-sided procedure.

Illustrative case study

In a fictional software team, a small experiment yields a p-value below the planned threshold. A manager proposes immediate rollout because the result is “proven.”

The analyst reports the estimated effect, its uncertainty, and the experiment's limited coverage. The team checks whether the possible benefit is meaningful relative to implementation cost and whether important secondary outcomes worsened. It plans an appropriate validation stage. The statistical test contributes to the decision without being treated as a guarantee of future performance or as a substitute for the business question.

Common misconceptions and mistakes

Failure to reject a null does not prove equivalence or no effect. A study may have low power for differences that matter. Conversely, a very large sample can detect a small effect of little practical consequence.

Do not interpret alpha as the probability that a particular rejected conclusion is wrong. Do not search through many outcomes, subgroups, or stopping points and report only the most favorable test as though it were the sole planned analysis. Assumption violations and biased data collection can invalidate an otherwise correct calculation. Statistical significance also does not establish causality unless the design and assumptions support that interpretation.

Business use

Hypothesis tests can help businesses evaluate claims and define repeatable decision rules under uncertainty. Their value increases when teams specify the question, effect scale, analysis, and decision consequences in advance. Use tests alongside estimation and operational judgment. Preserve unsuccessful and inconclusive results as well as favorable ones so that reporting reflects the evidence actually collected.

Check your understanding

A study reports p = 0.20. Does that establish a 20% probability that the null hypothesis is correct?

Answer: No. That interpretation reverses the conditioning. The p-value describes a tail probability under the null model, not a posterior probability of the hypothesis.

Related terms

Learn next

Learn sampling distributions and confidence intervals, then compare exact and approximate tests. Study multiplicity and study design before interpreting a collection of significance results.

Sources

American Statistical Association, press release summarizing its Statement on Statistical Significance and P-Values: https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf

NIST/SEMATECH, What are statistical tests?: https://www.itl.nist.gov/div898/handbook/prc/section1/prc13.htm

The primary guidance was consulted. The binomial calculation and fictional case are original teaching illustrations.

Previous: Statistical edit Next: Statistical interference