AI Glossary / Statistics and probability / Advanced
Size (statistics)
Definition
In hypothesis testing, the size of a test is the largest, more precisely the supremum, of its rejection probability over all parameter values allowed by the null hypothesis. It describes the worst-case Type I error probability under the specified model. A test has level alpha when its size is no greater than alpha. This use of size is different from sample size and effect size, and it need not equal the nominal significance level printed in an analysis plan.
What it means
For a simple null hypothesis that specifies one distribution, size is just the probability of rejecting under that distribution. For a composite null, different null parameter values can produce different rejection probabilities, so the worst case matters. A procedure calibrated to a nominal 5% level may have actual size below 5%, exactly 5% or above 5% if its approximation or assumptions fail. The word size describes the test procedure's behavior, not the evidential strength of one observed result.
How it works
Define the full null parameter space, sampling model and rejection rule. Calculate the probability of rejection at each null value or establish a mathematical bound. Take the supremum over those values. Analytical distributions, exact enumeration or carefully designed simulations can help assess the result. Simulation estimates need Monte Carlo uncertainty and cannot establish a global bound merely by checking a few convenient null settings. Include nuisance parameters, selection rules and interim monitoring if they are part of the actual procedure.
Three practical examples
Example 1: A binomial test checks whether an event probability exceeds a specified ceiling, with the worst null rejection probability occurring at the boundary for a monotone rule. Example 2: Researchers study whether a regression test maintains its nominal level under different allowed nuisance-variance values. Example 3: An AI evaluation team simulates its entire model-comparison procedure under a null scenario, including any selection steps. Each exercise concerns operating characteristics, rather than the sample's observed effect magnitude.
Think of it this way
Think of testing an alarm across every condition labeled normal. It may sound rarely in some normal conditions and more often in others. Its size corresponds to the worst false-alarm probability across the specified normal set. Checking only the easiest condition could make the alarm appear better calibrated than it is. The definition also depends on what the normal set includes, so assumptions must be visible.
Worked example
Let X count successes in five independent Bernoulli trials. Test the null p at most 0.5 against p greater than 0.5, using the rule reject only when X = 5. The rejection probability is p to the power five, which increases with p. Its supremum under the null occurs at p = 0.5 and equals 1/32 = 0.03125. Thus the rule is a level-0.05 test with size 0.03125. At p = 0.4 its rejection probability is 0.01024; at the alternative p = 0.8 its power is 0.32768. For a theoretical randomized rule, reject always at X = 5 and with probability 0.12 at X = 4. At p = 0.5 this gives 1/32 + 0.12 times 5/32 = 0.05. Such randomization illustrates exact calibration but is not an automatic practical recommendation.
Illustrative case study
In this fictional case study, Cedar Evaluation compares several AI scoring methods. A proposed test is described as a 5% procedure, but analysts discover that the workflow first selects the most promising comparison on the same data. They evaluate the full selection-and-testing rule rather than only its final formula. The revised study documents which null settings were examined and the uncertainty of simulation results. It does not claim universal size control from a small set of favorable simulations.
Common misconceptions and mistakes
Common mistakes include confusing test size with the number of observations, assuming nominal and actual levels always match, and checking only one parameter value for a composite null. Another error is interpreting size as the probability that a particular rejection is mistaken. Asymptotic size describes limiting behavior and does not guarantee exact finite-sample control. A test can be conservative but have low power, so small size alone does not make it useful. The null model's assumptions remain part of every calibration claim.
Business use
Businesses rarely need the supremum notation in routine reporting, but they benefit from knowing whether a claimed error threshold is actually controlled. AI benchmarking procedures involving many candidates, repeated looks or adaptive selection are especially sensitive to this distinction. Validation should reflect the entire decision rule and realistic nuisance conditions. Report nominal targets separately from measured or theoretically established operating characteristics. A transparent conservative test may be preferable to an apparently more powerful procedure whose error control is unsupported.
People also ask
Is test size always equal to alpha?
No; a level-alpha test may have smaller size.
Can a test have different false-rejection probabilities at different null values?
Yes, which is why a composite-null definition uses a supremum.
Is power the same as size?
Power is rejection probability as a function of the parameter, especially under alternatives; size summarizes its worst value within the null.
Does a simulation prove exact control?
Usually not by itself; it estimates behavior for the simulated conditions with uncertainty.
Check your understanding
What is the size of the nonrandomized binomial rule in the example?
Answer: 0.03125.
Why is it still a level-0.05 test?
Answer: Its size is below 0.05.
At which null value is the maximum attained?
Answer: p = 0.5.
Is the probability 0.32768 at p = 0.8 part of the test's size?
Answer: No; it is power at an alternative value outside the null.
Related terms
- Type I error
- Related terms include significance level
- power function
- composite hypothesis
- nuisance parameter
- conservative test
- exact test and asymptotic validity
Learn next
Learn next how monotonicity can identify the least favorable null value in a one-parameter problem. Explore randomized tests to understand why discrete outcomes complicate exact calibration, then study simulation uncertainty and composite-null validation. Compare size and power together when assessing whether a testing procedure serves the intended decision.
More in Statistics and probability
- Simpson's paradox - Simpson's paradox is a statistical phenomenon in which an association observed within separate groups reverses when the groups are combined.
- Single-arm study design - A single-arm study design follows one group receiving an intervention without an internally assigned concurrent comparison arm.
- Single-subject design - A single-subject design, often called a single-case design, studies outcomes repeatedly within individual cases while systematically varying an intervention...
- Size invariance - Size invariance is an image-classification system's ability to preserve the correct class when the apparent size or resolution of the relevant image content...
- Skewness - Skewness describes asymmetry in a distribution.
- Sliced inverse regression - Sliced inverse regression, or SIR, is a supervised dimension-reduction method that seeks informative linear combinations of predictors for explaining a...
Sources
Sources: Berkeley STAT 210A, p-values, confidence regions, and interpreting tests, https://stat210a.berkeley.edu/fall-2025/reader/testing-interpretation.html; Jean-Paul Renne, Econometrics and Statistics, Statistical tests, https://jrenne.github.io/EcoStat/Tests.html. The binomial probabilities and randomisation calculation are original exact examples, and the fictional AI evaluation case illustrates procedure-level calibration rather than a measured error rate.
Previous: Size invariance Next: Skeletonization