AI Glossary / Statistics and probability / Foundation
Type II error
Definition
A Type II error occurs when a statistical test fails to reject a null hypothesis that is false. Its probability is commonly denoted beta and is defined for a specified alternative situation. Statistical power is one minus beta at that alternative. Unlike a fixed significance-level target, the Type II error probability generally changes with the true effect size, variability, sample size and testing procedure. Failure to reject is not proof that the null is true.
What it means
A study can miss a real effect because its data are noisy or limited, the effect is small, or the chosen test is not sensitive to that departure. Type II error describes this missed rejection under a false null. To state its probability meaningfully, specify what the actual alternative is. A design may detect large effects reliably while often missing smaller ones. A single universal beta value without an effect definition can therefore give a misleading impression of the study's sensitivity.
How it works
Define the test, significance level, sample design and a scientifically or operationally meaningful alternative. Determine the sampling distribution under that alternative and calculate the probability that the statistic remains outside the rejection region. This is beta; the probability of entering the rejection region is power. Use appropriate analytical formulas or simulation that reflect the intended design. Planning calculations depend on assumed effects and variability, so sensitivity analysis is useful. After data collection, effect estimates and confidence intervals usually communicate uncertainty more directly than a power calculation based only on the observed effect.
Three practical examples
Example 1: A product experiment fails to detect a genuine but modest improvement in average completion time. Example 2: An AI comparison misses a relevant performance difference because too few independent tasks were sampled. Example 3: A quality study fails to reject a baseline mean even though a specified shift has occurred. These examples do not establish that every nonsignificant finding is a Type II error. The null's actual truth is usually unknown, and some failures to reject are correct.
Think of it this way
Think of an alarm that sometimes stays silent when a real change occurs. Its chance of detecting the change depends on how strong the signal is relative to background noise. A large change may be easy to detect while a small one is frequently missed. The silence tells you that the alarm did not trigger; it does not by itself prove that nothing changed.
Worked example
Suppose independent normal observations have known standard deviation ten. Test H0: mu = 0 against mu greater than zero using a one-sided 0.05 z test. With 100 observations, the standard error is one and the rejection cutoff for the sample mean is approximately 1.644854. If the true mean is three, a Type II error occurs when the sample mean does not exceed that cutoff. Its probability is Phi(1.644854 minus 3), approximately 0.0877, so power is approximately 0.9123. With only 25 observations, the standard error is two, the mean cutoff is approximately 3.2897, and beta at the same true mean three is Phi((3.2897 minus 3)/2), approximately 0.5576. These calculations assume the stated normal model, known variance and fixed design. They show that sensitivity depends on sample size at a specified effect.
Illustrative case study
In this fictional case study, Oak Product considers an AI workflow change. Its small pilot produces an uncertain estimate and does not reject the no-change null. Instead of announcing that the workflows are equivalent, analysts show the confidence interval and explain which operationally relevant improvements or deteriorations remain compatible with the data. They plan a larger study using a meaningful target difference and several plausible variability assumptions. Managers can then decide whether further evidence is worth collecting.
Common misconceptions and mistakes
Common mistakes include describing failure to reject as acceptance of the null, assigning beta without specifying an alternative, and claiming that a nonsignificant result proves practical equivalence. Another mistake is relying on observed power calculated from the same effect estimate as a new explanation of the p-value. Power planning should be tied to meaningful effects and design assumptions. More observations do not automatically solve selection bias, poor measurement or dependence that was ignored in the analysis.
Business use
Businesses use power and Type II error calculations to plan studies that can detect differences worth acting on. Identify a meaningful effect before choosing sample size, and consider uncertainty in the planning assumptions. If the decision is to demonstrate acceptable similarity, use an appropriate equivalence or noninferiority design with justified margins. Balance the costs of missed effects against false rejections and data collection. Report a nonsignificant result with its estimate and uncertainty rather than converting it into a definitive statement that nothing matters.
People also ask
Is beta always 0.20?
No; that is only a common planning choice in some contexts.
Is power one minus beta?
Yes, for the same test and specified alternative.
Does a nonsignificant result necessarily mean a Type II error occurred?
No; the null may be true.
Can tightening the significance threshold increase beta?
With other conditions fixed, often yes.
Can a study have high power for large effects but low power for small ones?
Yes.
Check your understanding
What is beta for 100 observations at the stated mean three?
Answer: Approximately 0.0877.
What is the corresponding power?
Answer: Approximately 0.9123.
Why is beta larger with 25 observations in the example?
Answer: The sample mean has more variability relative to the same target shift.
Does failing to reject establish equivalence?
Answer: No; equivalence requires a different inferential question and suitable margins.
Related terms
- Type I error
- Significance level
- Effect size
- Related terms include statistical power
- sample-size planning
- power curve and equivalence testing
Learn next
Learn next how the null and alternative sampling distributions overlap and how a rejection threshold trades different errors. Compare power curves under plausible sample sizes and variability. Then study interval estimation and equivalence methods so that an inconclusive result is not mistaken for evidence of practical sameness.
More in Statistics and probability
- Two-sample hypothesis testing - Two-sample hypothesis testing uses observations from two samples to assess a specified claim about their source populations or distributions.
- Two-sided test - A two-sided test evaluates departures from a null hypothesis in either direction.
- Twoway factorial design - A two-way factorial design studies two factors by including every combination of their selected levels.
- Type III error - Type III error is a statistical expression with several meanings, so its intended definition must be stated.
- Ultimate sampling unit - An ultimate sampling unit is a unit selected at the final stage of a sampling design.
- Uncomfortable science - Uncomfortable science is an informal methodological expression for research situations where limited available evidence is used both to develop explanations...
Sources
Sources: NIST/SEMATECH Engineering Statistics Handbook, Sample sizes required, https://itl.nist.gov/div898/handbook/prc/section2/prc222.htm; Penn State, Hypothesis Testing, Part 2, https://online.stat.psu.edu/stat200/Lesson06. These primary instructional sources explain Type II error, power and design dependence. The two normal-test planning calculations are original and independently checked; the pilot-study case is fictional.
Previous: Type I error Next: Type III error