What is the significance level?

It is a prespecified error-control threshold associated with rejecting the null, interpreted under the test's assumptions over repeated use.

Does failure to reject mean accepting the null as true?

No. It means the test did not provide sufficient evidence for rejection under the chosen procedure; equivalence requires an appropriate design and analysis.

Can a small p-value prove causation?

No. Causal conclusions depend on study design and assumptions, not simply on statistical significance.

AI Glossary / Graphs and symbolic AI / Intermediate

Significance test

Definition

A significance test evaluates how compatible observed data are with a specified null hypothesis and statistical model. It uses a test statistic and its reference distribution, often reporting a p-value that describes how unusual an equally or more extreme result would be under those assumptions.

What it means

The null hypothesis might state that two systems have equal expected performance, that a population parameter equals a reference value, or that a particular association is absent. The test does not begin by assigning a probability that the null hypothesis is true.

Instead, it asks what kinds of observations would be expected if the null model and the other assumptions were correct. A small p-value indicates that the observed result is unusual under that model. It does not directly give the probability that the effect is real, that the result happened “by chance,” or that a business decision will succeed.

Statistical significance also differs from practical importance. A tiny difference can produce a small p-value with enough data, while an important effect can be estimated imprecisely in a small study.

How it works

Before examining results, specify the question, outcome measure, null and alternative hypotheses, test procedure and intended error control. The design determines whether observations are independent, paired, clustered or otherwise dependent. The test must respect that structure.

Calculate the test statistic and compare it with its distribution under the null model. A p-value summarizes the relevant tail probability. If a decision threshold alpha is used, a valid level-alpha test controls its long-run false-rejection probability under the null and the test's assumptions, typically at or below alpha rather than exactly equal for every setting.

Report the effect estimate and uncertainty alongside the p-value. Multiple comparisons, repeated unplanned looks at accumulating data, selective reporting and choices made after seeing results can invalidate a simple nominal interpretation. A test result is one part of an argument that also depends on study design, measurement quality and model assumptions.

Three practical examples

An AI evaluation compares two models on the same examples using a method that accounts for the paired observations.
A controlled product experiment tests a prespecified difference in a user outcome while separately assessing its business magnitude.
A quality-analysis exercise tests whether an observed process measure is compatible with a specified reference model, with assumptions checked for the measurement process.

Think of it this way

Think of a smoke alarm calibrated to react when a sensor reading is unusual under ordinary conditions. An alarm suggests that the ordinary-condition model deserves scrutiny, but it does not itself identify the cause or measure the size of a fire.

The analogy is only about detecting incompatibility. A p-value is not the probability that an alarm is false, and statistical testing needs explicit assumptions about how observations were generated.

Worked example

An exact small-sample calculation

Consider an invented experiment that fixes ten independent tosses in advance. The null hypothesis is that the coin has heads probability one-half on every toss. The two-sided alternative is that the probability differs from one-half. Nine heads are observed.

Under this null, all 2 to the power 10, or 1,024, ordered head-tail sequences are equally likely. Results at least as far from five heads as nine heads are 0, 1, 9 or 10 heads. Their counts are 1, 10, 10 and 1, giving p = 22/1,024, approximately 0.0215.

A prespecified 0.05 threshold would reject the null in this test. It would be incorrect to say there is only a 2.15% probability that the coin is fair. The calculation assumes the fixed sample size and independent, identically distributed tosses; stopping opportunistically or selecting this coin from many tested coins requires a different analysis.

Illustrative case study

Fictional case study: Cedar Evaluation compares several prompt variants for a language-processing task. One variant produces a p-value below the team's familiar threshold, and an early report calls it a definitive winner.

Review shows that the team tried many variants, inspected several metrics and reused the same test material during selection. The reported number does not account for those choices. The measured gain is also too small to resolve whether it matters operationally.

Cedar distinguishes exploratory findings from a planned follow-up evaluation, reports the full set of comparisons and includes effect estimates and uncertainty. The fictional lesson is that a threshold crossing cannot repair an evaluation design or replace judgment about usefulness.

Common misconceptions and mistakes

A p-value of 0.03 means a 97% probability that the alternative is true. It does not assign that posterior probability.
A nonsignificant result proves no difference. The study may lack precision or power to detect a relevant difference.
Statistical significance measures effect size. It combines the observed effect with variability and sample information; the magnitude must be reported separately.
Crossing 0.05 makes a result categorically different from 0.051. Nearby results should be interpreted in context, with assumptions and uncertainty visible.

Business use

Significance tests can support disciplined comparisons, but they should not become automatic release gates detached from context. Define a useful effect size, evaluation population and acceptable error trade-offs before collecting or selecting results.

For AI systems, consider whether examples share users, documents or sources, whether outputs were selected after repeated trials and whether evaluation labels are reliable. Report limitations that affect generalization. A statistically detectable benchmark difference does not by itself establish a meaningful improvement for the website's audience or an application's actual users.

Check your understanding

A model comparison reports p = 0.01 but omits the size of the performance difference. Can you conclude that the change is valuable to users?

Answer: No. The p-value does not measure practical value. You need the effect size, uncertainty, evaluation design and relevance to the intended use.

Related terms

Null hypothesis
P-value
Significance level
Type I error
Effect size
Confidence interval
statistical power
multiple testing
paired comparison

Learn next

Read p-value and effect size. Recalculate the ten-toss example, then explain why its tail probability differs from the probability that the hypothesis is true. Draft a model-comparison report that includes magnitude, uncertainty and all prespecified outcomes.

Sources

NIST/SEMATECH statistical handbook, Critical Values and p Values: https://itl.nist.gov/div898/handbook/prc/section1/prc131.htm. American Statistical Association's statement announcement and six interpretation principles: https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf. Relevant definitions and cautions were reviewed. The coin calculation is exact under its stated assumptions; the AI evaluation case is fictional.

Previous: Significance level Next: Silence compression