AI Glossary / Statistics and probability / Foundation
Significance level
Definition
A significance level is a prespecified probability threshold used to calibrate a statistical hypothesis test, commonly denoted alpha. A level-alpha procedure is designed so that its probability of rejecting a true null hypothesis is no greater than alpha under the stated assumptions. The nominal level may differ from the procedure's actual error probability, especially for discrete or approximate tests. It is not the probability that a particular rejected hypothesis is true or that an observed finding is wrong.
What it means
The significance level describes the behavior of a testing rule across hypothetical repetitions when the null model holds. Choosing 0.05 limits a particular kind of erroneous rejection for a valid test, but it does not say that 5% of reported discoveries must be false. That latter proportion also depends on which hypotheses are tested, power, selection and other factors. A significance level is chosen for a procedure; a p-value is calculated from the observed data under a specified model.
How it works
Specify the null, alternative, statistic and sampling or randomization model. Choose the significance level in advance, considering the purpose and consequences of errors. Calibrate a rejection region whose null probability meets the intended control. Equivalently, many procedures reject when a valid p-value is at most alpha. Account for planned multiple comparisons, interim looks and adaptive choices. Report the effect estimate and uncertainty as well as any threshold decision, because a binary label discards information about magnitude and compatibility with alternative values.
Three practical examples
Example 1: An A/B experiment prespecifies a test level before comparing a primary outcome. Example 2: A quality team defines an alarm rule with a stated probability of triggering under a stable null process. Example 3: An AI research team adjusts its testing plan when comparing many model configurations on the same evaluation data. These settings require different design details. Reusing a familiar threshold does not automatically control errors when dependence, selection or repeated monitoring changes the procedure.
Think of it this way
Think of setting the sensitivity of an alarm under a defined normal operating model. The calibration describes how often that rule would sound under those conditions. Once it sounds, the calibration alone does not tell you the probability that a real problem exists. That also depends on how common real problems are and how well the alarm detects them. Statistical testing has the same distinction between a rule's conditional behavior and the interpretation of one result.
Worked example
Suppose a two-sided test statistic is exactly standard normal under its null. A level 0.05 rule rejects when its absolute value exceeds approximately 1.95996, leaving total null tail probability 0.05. Across 1,000 such true-null tests, the expected number of rejections is 50, not a guaranteed count. If 20 tests are independent and all their nulls are true, each tested at 0.05 without adjustment, the probability of at least one rejection is 1 - 0.95 to the power 20, approximately 0.6415. Discreteness creates another distinction: a two-sided sign test with six independent nonzero signs rejects at nominal 0.05 only for all positive or all negative signs, giving actual null rejection probability 2/64 = 0.03125. The nominal threshold and actual size therefore need not coincide.
Illustrative case study
In this fictional case study, Pinecrest Analytics compares an AI routing system with its existing process. The team prespecifies one primary outcome and an analysis schedule. During exploration, analysts find several favorable subgroup comparisons, but keep them separate from the planned primary test and report the broader set of results. Managers consider the estimated improvement, uncertainty and operational cost instead of treating a threshold crossing as a complete adoption rule. The report explains how repeated comparisons were handled.
Common misconceptions and mistakes
Common mistakes include interpreting alpha as the probability a result is a false discovery, choosing the threshold after seeing the p-value and treating 0.049 and 0.051 as fundamentally different evidence. Another error is assuming a nominal 0.05 level remains valid after unplanned repeated looks or selective reporting. Lowering alpha generally reduces power for a fixed design, so error choices involve trade-offs. A nonsignificant result does not establish equality, equivalence or absence of a practically important effect.
Business use
Businesses use significance levels when they need a transparent, repeatable testing rule. The level should be part of a wider decision process that also considers effect size, uncertainty, implementation cost and the consequences of missed or false signals. AI evaluation plans should state the primary comparisons and any multiplicity or monitoring adjustments. Teams can report p-values continuously without turning every result into a categorical verdict, while still preserving any prespecified decision rule required by the study.
People also ask
Is 0.05 mandatory?
No; it is a common convention, not a universal requirement.
Is alpha the same as the p-value?
No; alpha calibrates the rule before observing the result, while the p-value summarizes a data-dependent discrepancy under the null.
Does a 1% level guarantee that only 1% of positive claims are false?
No; it controls a conditional error probability under stated assumptions.
Can actual error probability be smaller than alpha?
Yes, particularly for conservative or discrete tests.
Check your understanding
What is the expected number of rejections among 1,000 valid true-null tests at actual size 0.05?
Answer: Fifty, without guaranteeing that realized count.
Why is the chance of at least one rejection across 20 independent tests much larger than 0.05?
Answer: There are multiple opportunities for a false rejection.
What is the actual size of the six-sign rule in the example?
Answer: 0.03125.
Does nonrejection prove the null?
Answer: No.
Related terms
- P-value
- Rejection region
- Related terms include Type I error
- test size
- nominal level
- statistical power
- family-wise error rate and false discovery rate
Learn next
Learn next how test size and power vary with the null and alternative distributions. Compare family-wise error control with false discovery rate control, then study sequential testing if results will be monitored repeatedly. Practice interpreting estimates and intervals near a chosen threshold without converting a small numerical difference into an exaggerated substantive conclusion.
More in Statistics and probability
- Siegel - Tukey test - The Siegel-Tukey test is a nonparametric procedure for comparing the spread of two independent samples using specially assigned ranks.
- Sign test - The sign test is a nonparametric test based on whether observations lie above or below a specified value, or whether paired differences are positive or...
- Signal magnitude area - Signal magnitude area, commonly abbreviated SMA, is a feature that summarizes the absolute magnitude of a signal over a time window.
- Simalto - SIMALTO is a market-research approach that uses a grid of product or service attributes and levels to explore customer priorities under trade-offs.
- Simple effect - A simple effect is the effect of one factor evaluated at a specified level of another factor.
- Simple interaction - A simple interaction is the interaction between two factors evaluated at one specified level of a third factor.
Sources
Sources: Frank Harrell, Glossary of Statistical Terms, https://hbiostat.org/glossary/; NIST/SEMATECH e-Handbook, What are statistical tests?, https://itl.nist.gov/div898/handbook/prc/section1/prc13.htm. The normal-tail, multiple-test and discrete sign-test calculations are original illustrations under explicitly stated assumptions; the fictional routing case is not a reported deployment.
Previous: Signaling compression Next: Significance test