Does rejection prove the alternative?

It provides evidence against the null within the specified procedure and assumptions.

Should business decisions depend only on 0.05?

No; effect size, uncertainty, costs, and context also matter.

AI Glossary / Statistics and probability / Intermediate

Hypothesis testing

Definition

Hypothesis testing is a statistical procedure for assessing evidence against a specified null hypothesis using a test statistic and a reference distribution. It can support a decision to reject or fail to reject the null under stated assumptions. A test does not generally prove a hypothesis true or false, and its result must be interpreted alongside effect size, uncertainty, study design, and practical consequences.

What it means

Observed differences can arise from genuine population differences, sampling variation, measurement problems, or a flawed model. A hypothesis test asks whether a chosen summary of the data is sufficiently unusual under the null model to warrant rejection according to a defined procedure.
The conclusion is narrower than a business decision. Detecting a small improvement does not establish that deployment is worthwhile, while failing to detect a difference does not establish that two systems perform equally well.

How it works

Define the question, population, outcome, and parameter before choosing a test. State a null hypothesis and an alternative. For example, a null might specify equal mean error between two systems, while the alternative allows a difference. A one-sided alternative requires a justified direction chosen before examining the result.
Choose a test appropriate to the sampling design and data. Paired observations, clustered users, repeated measurements, and independent groups require different treatment. An impressive sample count does not make dependent observations independent.
Specify a significance level alpha if using a rejection rule. Under the test's assumptions, this controls the probability of rejecting a true null at or below the stated level. It is a long-run property of the procedure, not the probability that this particular rejection is mistaken.
Calculate the test statistic and its null reference distribution. The reference can come from a mathematical model, a randomization design, or a justified resampling procedure. Exact and approximate tests differ in their requirements and calibration.
A p-value is the probability, under the null model and relevant assumptions, of a test statistic at least as incompatible with the null as the observed one. The definition of extreme depends on the test and alternative. It is not the probability of the null given the data.
If the p-value crosses the preselected threshold, reject according to the procedure. Otherwise fail to reject. Report the result without translating non-rejection into proof of no effect. Demonstrating equivalence requires an appropriate design and predefined meaningful margins.
Consider power: the probability of rejection under a specified alternative. Power depends on effect size, variability, sample size, design, and the testing rule. A test can miss a practically important effect when it has insufficient information.
Account for multiple comparisons, exploratory selection, and repeated looks at accumulating data. Testing many outcomes and reporting only the smallest p-value changes the error rate. Sequential designs can support interim analyses, but ordinary fixed-sample thresholds should not be reused without justification.
Report effect estimates, compatible uncertainty intervals, assumptions, exclusions, and the analysis plan. A statistically significant finding can still be biased by confounding, missingness, measurement error, or selective reporting. Testing does not repair those problems automatically.

Three practical examples

An AI evaluation compares paired task-level performance between two systems using a test matched to the pairing.
A product experiment assesses evidence for a change in a predefined outcome while reporting the estimated effect and its uncertainty.
A quality team tests whether a process parameter departs from its stated target using a justified sampling model.

Think of it this way

Think of a test as an alarm calibrated for a particular background pattern. A signal says the observations are unusual under that pattern, not why they occurred. Silence can mean the pattern is adequate, or simply that the alarm lacks sensitivity to the change that matters.

Worked example

Suppose a preplanned experiment has ten independent binary trials with a common success probability p. Test H-zero: p = 0.5 against the one-sided alternative p greater than 0.5. The sample size is fixed in advance, and nine successes are observed.
Under the null, X follows a binomial distribution with ten trials and probability 0.5. Results at least as extreme in the specified direction are nine or ten successes. The p-value is [choose(10,9) + choose(10,10)]/2^10 = (10 + 1)/1,024 = 11/1,024, approximately 0.010742.
At a preselected alpha of 0.05, the procedure rejects this null. The observed success proportion is 0.9, which is 0.4 above the null value.
The p-value does not mean there is a 1.0742 percent chance that p equals 0.5. Nor does ten observations establish that the future success rate is exactly 0.9. A two-sided question, dependent trials, or a stopping rule based on interim outcomes would require reconsidering the calculation.

Illustrative case study

In this fictional case, Willow Analytics tests an AI workflow against a baseline on a predefined outcome. The result crosses its significance threshold, but the estimated improvement is smaller than the team's minimum worthwhile gain. It reports the statistical evidence and practical size separately, checks the uncertainty interval, and postpones deployment pending a broader operational assessment. The test informs the decision rather than serving as its sole criterion.

Common misconceptions and mistakes

Statistical significance is not practical importance. A p-value is not the probability that a result occurred by chance alone and not the probability that the null is true. Non-significance is not evidence of exact equality.
Changing a one-sided direction after observing the data, ignoring pairing, or searching many outcomes without accounting for selection can invalidate the intended interpretation. Increasing sample size can make a negligible difference detectable without making it useful.

Business use

Use tests to answer clearly specified inferential questions within a broader evidence process. Define the decision threshold in practical units as well as any statistical threshold. Ensure the evaluation population and sampling design match intended use, and preserve complete results so teams can assess uncertainty and selection effects.

Check your understanding

Why does the worked p-value include ten successes?

Answer: The alternative regards ten as at least as extreme as nine.

What would undermine its exact calculation?

Answer: Dependence, unequal success probabilities, or an unaccounted outcome-dependent stopping rule could invalidate the stated binomial model.

Related terms

Learn next

Learn sampling distributions and error rates. Then study effect estimation, multiple-testing procedures, equivalence testing, and evaluation designs that support the intended conclusions.

Sources

NIST/SEMATECH, What are statistical tests?: https://www.itl.nist.gov/div898/handbook/prc/section1/prc13.htm American Statistical Association, Statement on Statistical Significance and P-Values release: https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf NIST AIRC glossary landing page: https://airc.nist.gov/glossary/

Previous: Hypothesis space Next: I-frame