AI Glossary / Statistics and probability / Foundation
Type I error
Definition
A Type I error occurs when a statistical hypothesis test rejects a null hypothesis that is true. Its probability is evaluated under the null and depends on the testing procedure and its assumptions. The significance level is intended to control this probability, often at a chosen bound such as 0.05. It is not the probability that a particular significant result is false, nor the probability that the null hypothesis is true after observing the data.
What it means
A test can sometimes signal a departure simply because of random variation. Type I error describes that mistaken rejection within the specified testing framework. The null must be stated before the error can be interpreted: it might concern a mean difference, a distributional property or another parameter. The cost of rejection can differ by application, but the probability remains a conditional property of a procedure under a true null. A reported significance level is meaningful only when the procedure and assumptions support its intended control.
How it works
Specify the null, statistic, rejection rule and significance level in advance. Derive or justify the statistic's reference behavior under the null, then select a rejection region with an appropriate null rejection probability. For a composite null, control may need to hold across all parameter values allowed by it. Discrete tests can be conservative, so their actual probability may fall below the nominal level. Multiple testing, data-dependent selection and repeated monitoring require suitable adjustments or designs if the original error guarantee is to be maintained.
Three practical examples
Example 1: An experiment concludes that two population means differ when the relevant equality null is true. Example 2: An AI evaluation declares an improvement under a true no-improvement null using a prespecified directional test. Example 3: A process-monitoring test rejects an unchanged-baseline hypothesis when the baseline actually still applies. These examples are conditional on their formal nulls and procedures. A mistaken operational decision can arise for other reasons too, such as a wrong model or an irrelevant outcome.
Think of it this way
Think of an alarm that occasionally rings even when the condition it is designed to detect is absent. The alarm's false-alert behavior must be measured over occasions when that condition is absent. It is different from asking what fraction of all alarms correspond to real events, which also depends on how often those events occur. The same conditional-probability distinction matters for statistical tests.
Worked example
Suppose 25 independent normal observations have known standard deviation ten. Test H0: mu = 0 against mu greater than zero using the sample mean. Its null standard error is 10 divided by the square root of 25 = two. A one-sided 0.05 rule rejects when the sample mean exceeds 1.644854 times two, approximately 3.2897. Under the stipulated null distribution, that event has probability 0.05 and is a Type I error. If 100 independent studies all have true nulls and use this exact rule, the expected number of rejections is five. The probability of at least one is 1 minus 0.95 to the power 100, approximately 0.9941. Neither calculation says that five percent of significant results in an arbitrary research field are false; that is a different question requiring additional information.
Illustrative case study
In this fictional case study, Birch Experimentation runs many AI feature comparisons. A team initially treats each unadjusted 0.05 test as sufficient evidence for rollout. Analysts explain that the collection of tests has a different false-rejection behavior from a single prespecified test and adopt a suitable analysis plan. They report effect estimates and uncertainty, record all planned comparisons and investigate whether results replicate. Managers understand the error-control target without interpreting the significance level as the probability that each launched feature is ineffective.
Common misconceptions and mistakes
Common mistakes include reversing the conditional probability, treating alpha as the chance that a particular conclusion is wrong, and assuming the nominal level is guaranteed despite violated assumptions. Another error is choosing outcomes or stopping rules after examining significance without accounting for that selection. A Type I error is defined relative to a true null, which is usually unknown in practice. It cannot generally be diagnosed merely because a later estimate differs or a replication is nonsignificant.
Business use
Businesses should choose testing procedures and error-control targets that match the decision context, including the number of comparisons and how results are monitored. A low Type I error probability does not ensure useful power, practical value or freedom from bias. Consider the consequences of both false rejections and missed effects, and report meaningful effect sizes. Keep a record of planned and exploratory analyses so readers can distinguish formal error-controlled claims from findings that require further confirmation.
People also ask
Is a Type I error rejecting a false null?
No; rejecting a false null is a correct rejection.
Is alpha the probability that a significant result is false?
No.
Must an exact test attain its nominal level exactly?
No; discrete procedures can be conservative.
Can repeated testing change the overall chance of a false rejection?
Yes.
Does lowering alpha automatically improve every aspect of a study?
No; with other conditions fixed, it can reduce power.
Check your understanding
What event is the Type I error in the example?
Answer: Rejecting mu = 0 when that null is true.
What sample-mean cutoff is used?
Answer: Approximately 3.2897.
What is the expected number of false rejections across the specified 100 studies?
Answer: Five.
Does that imply exactly five will occur?
Answer: No; it is an expectation over repetitions.
Related terms
- Type II error
- Related terms include significance level
- test size
- false rejection
- statistical power
- familywise error rate and false discovery rate
Learn next
Learn next how a rejection region determines the null error probability and why composite nulls require a uniform bound. Compare per-test and familywise control, then examine procedures for repeated monitoring. Keep the probability of rejection given a true null separate from the probability of a true null given a rejection.
More in Statistics and probability
- Two-sample hypothesis testing - Two-sample hypothesis testing uses observations from two samples to assess a specified claim about their source populations or distributions.
- Two-sided test - A two-sided test evaluates departures from a null hypothesis in either direction.
- Twoway factorial design - A two-way factorial design studies two factors by including every combination of their selected levels.
- Type III error - Type III error is a statistical expression with several meanings, so its intended definition must be stated.
- Ultimate sampling unit - An ultimate sampling unit is a unit selected at the final stage of a sampling design.
- Uncomfortable science - Uncomfortable science is an informal methodological expression for research situations where limited available evidence is used both to develop explanations...
Sources
Sources: NIST, Level of Significance, https://csrc.nist.gov/glossary/term/level_of_significance_a; NIST/SEMATECH, Critical values and p values, https://itl.nist.gov/div898/handbook/prc/section1/prc131.htm; Harrell, Glossary of Statistical Terms, https://hbiostat.org/glossary/. These primary institutional and author sources define null rejection probabilities. The normal-test and repeated-study calculations are original and independently checked; the experimentation case is fictional.
Previous: Type constraint Next: Type II error