AI Glossary / Statistics and probability / Advanced
Neyman - Pearson lemma
Definition
The Neyman - Pearson lemma identifies a most powerful statistical test for a fully specified null distribution against a fully specified alternative distribution, subject to a chosen Type I error limit. The test rejects the null for sufficiently large values of the alternative-to-null likelihood ratio, with possible randomization at the boundary.
What it means
A test must balance false alarms against its ability to detect a particular alternative. “Most powerful” means achieving the highest rejection probability under that specified alternative while respecting the false-alarm constraint under the null.
It does not mean proving which hypothesis is true, maximizing every business objective, or detecting every possible deviation equally well. The distributions and error constraint define the optimization problem.
How it works
For observed data x, form the ratio p1(x)/p0(x), where p0 and p1 are the null and alternative probability mass functions or densities. Set the threshold using the ratio's distribution under the null.
For discrete outcomes, rejecting all observations at a boundary may exceed the allowed error rate. A test can instead reject a specified fraction of boundary cases, using an independent random draw. This achieves an exact error level when appropriate. A nonrandomized rule can be more conservative. The lemma's guarantee assumes the stated distributions describe the data.
Three practical examples
1. Device monitoring: an engineering model specifies sensor behavior under normal operation and under one defined fault. The likelihood-ratio test provides a benchmark for detecting that fault at a chosen false-alarm rate.
2. AI evaluation: a team compares two fully specified distributions for a benchmark outcome. If it estimates unknown parameters from the same data, it must justify the resulting procedure separately rather than merely invoking the simple-hypothesis lemma.
3. Alert design: an analyst compares a proposed decision rule with the optimal rule for an illustrative probability table. This reveals whether weak detection comes from a poor threshold or from limited separation between the assumed distributions.
Think of it this way
Imagine a detector with a limited false-alarm allowance. Some observations contribute much more detection probability than false-alarm probability when included in the rejection rule. The likelihood ratio ranks those opportunities. A rare-looking observation is not necessarily a good choice: what matters is its relative probability under the two specified models.
Worked example
Consider an invented detector producing exactly one of three symbols, A, B or C. Under normal operation, H0, their probabilities are 0.50, 0.30 and 0.20. Under a specified fault, H1, they are 0.10, 0.30 and 0.60. Both rows sum to one.
The alternative-to-null ratios are 0.2 for A, one for B and three for C. Set the Type I error allowance to 0.25 for this teaching example. Always rejecting on C uses 0.20 of that allowance and gives power 0.60. The next-ranked outcome is B.
To use the remaining 0.05 allowance, reject on B with probability 0.05/0.30 = 1/6. Never reject on A. The resulting Type I error is 0.20 + 0.30 × 1/6 = 0.25. Its power is 0.60 + 0.30 × 1/6 = 0.65, so its Type II error under H1 is 0.35.
Why can no permitted rule do better? Let a, b and c be rejection probabilities on the three symbols. Power is 0.10a + 0.30b + 0.60c. Rewrite it as the false-alarm probability, 0.50a + 0.30b + 0.20c, plus 0.40c − 0.40a. Since the first quantity is at most 0.25, c is at most one and a is nonnegative, power cannot exceed 0.65. The constructed rule reaches that bound.
Without randomization, the only nonempty permissible rejection set is C alone. Its power is 0.60. Adding B would raise the false-alarm rate to 0.50. This discrete example explains why allowing randomized tests changes what is attainable.
Illustrative case study
Fictional Cedar Sensors uses the three-symbol example in an internal training exercise. Someone describes its rule as “optimal for all faults.” The team tests that claim against a different hypothetical fault distribution: A, B and C now have probabilities 0.80, 0.15 and 0.05.
The original rule has power 0.05 + 0.15/6 = 0.075 against this new fault. A rule that rejects on A with probability one-half has Type I error 0.50 × 0.50 = 0.25 and power 0.80 × 0.50 = 0.40. It is optimal for the new alternative because A now has the largest likelihood ratio and can absorb the entire allowance through randomization.
Cedar therefore records the alternative distribution beside every optimality claim. This is a fictional mathematical illustration, not a measured sensor deployment.
Common misconceptions and mistakes
“A larger likelihood ratio is a posterior probability.” Ratios can exceed one; a posterior probability requires additional probability assumptions about the hypotheses.
“Twenty-five percent of alerts must be false.” The example's 0.25 is the rejection probability conditional on normal operation. It is not the fraction of rejected cases that are normal.
“Randomisation changes the observed data.” It changes the decision for a boundary observation, not the symbol that was observed.
“Simple means easy.” A simple hypothesis fully specifies a distribution. A complicated but completely specified model can still be simple in this technical sense.
Business use
Use the lemma to clarify a detector's objective and to establish an ideal benchmark under explicit assumptions. Document the null, alternative, observation unit, error allowance and decision rule together.
The example deliberately uses a large error allowance to keep arithmetic transparent; it is not an operational recommendation. Real projects must assess the consequences of missed detections and false alarms. Estimated distributions, changing conditions and correlated repeated decisions can make actual performance differ from a theoretical calculation. Validate the full decision process on suitable independent evidence.
People also ask
Does the lemma cover composite alternatives automatically?
No. A uniformly most powerful test requires the same rule to be optimal across the relevant alternatives, which needs an additional argument.
Must the test use randomization?
No. Continuous settings often permit exact thresholds without it; discrete settings can use conservative rules.
Is failure to reject proof of normal operation?
No. In the example, the specified fault escapes detection with probability 0.35.
Check your understanding
Why select C before B?
Answer: Its likelihood ratio is three rather than one.
Why randomize on B with probability 1/6?
Answer: Its null probability 0.30 must contribute only the remaining 0.05 allowance.
What changes with the second fault distribution?
Answer: The likelihood-ratio ordering and optimal decision rule change.
Related terms
- Type I error
- Type II error
- Uniformly most powerful test
- Likelihood-ratio test
- simple hypothesis
- statistical power
- randomised test
Learn next
Repeat the three-symbol calculation with an allowance of 0.10. Determine which symbol lies at the decision boundary and verify both the false-alarm rate and power directly.
More in Statistics and probability
- Network sampling - Network sampling is a survey approach in which a target item can be reported through several linked sampling units.
- Networked model - A networked model assigns a local hypothesis space to each node of a network used for federated learning.
- Newman-Keuls test - The Newman-Keuls test, also called the Student-Newman-Keuls test, is a stepwise procedure for comparing group means.
- Noise factor - A noise factor is a process input or operating condition that is difficult, inconvenient or impractical to control in normal use.
- Noisy channel model - A noisy channel model explains an observed signal as the result of an underlying message passing through a process that can alter it.
- Nominal scale - A nominal scale classifies observations into categories without assigning those categories an inherent order or numerical distance.
Sources
University of California, Berkeley, Stat 210A course reader, Hypothesis Testing and the Neyman-Pearson Lemma, likelihood-ratio theorem, proof and boundary randomisation: https://stat210a.berkeley.edu/fall-2024/reader/hypothesis-testing.html
Previous: Next word index Next: Nguyen-Wainwright-Jordan lower bound