AI Glossary / Statistics and probability / Intermediate
False discovery rate
Definition
The false discovery rate, or FDR, is the expected proportion of rejected null hypotheses that are actually true. If V is the number of false rejections and R is the total number of rejections, define the false discovery proportion as V/R when R is positive and zero when there are no rejections. FDR is the expectation of that proportion across repeated applications of the testing procedure.
What it means
When many hypotheses are tested, some apparently promising findings can be false positives. FDR control limits their expected share among the selected findings under the procedure's assumptions. It is often useful when screening many candidates for further investigation.
An FDR target of 5% does not guarantee that at most 5% of this particular result list is false. It also does not assign each selected finding a 5% probability of being false. The guarantee concerns a repeated-sampling average of a random proportion.
How it works
Define the family of hypotheses and obtain valid p-values for its tests. The family should reflect the intended claims and analysis plan. Selective reporting, data-driven stopping, or choosing only favorable tests can invalidate the advertised error control if not accounted for.
Choose a procedure appropriate to the dependence structure. The Benjamini - Hochberg procedure, usually abbreviated BH, controls FDR under independence and specified forms of positive dependence. It is not automatically valid for every possible dependence pattern. Benjamini - Yekutieli, or BY, provides control under broader dependence with more conservative thresholds, assuming valid null p-values.
For BH with m tests and target q, sort the p-values from smallest to largest: p_(1), …, p_(m). Find the largest rank k satisfying p_(k) ≤ kq/m. Reject the hypotheses at ranks one through k. If no rank satisfies the condition, reject none.
This is a step-up rule: the largest passing rank determines the selected set. It is not a rule that stops permanently at the first failure when scanning upward. Software often returns adjusted p-values that can be compared with q to obtain the same rejection set.
For BY, replace q in those thresholds with q divided by the harmonic sum 1 + 1/2 + … + 1/m. This protects against more general dependence at the cost of fewer discoveries in many settings.
Report the method, family size, target, and adjusted p-values alongside effect estimates and uncertainty. FDR adjustment does not correct a flawed study design or make a tiny effect important. Follow-up validation may still be needed before using a selected finding in a consequential decision.
Distinguish FDR from the false-positive rate among true nulls and from familywise error rate, the probability of making any false rejection. If every null is true, FDR equals familywise error rate because every nonempty rejection set consists entirely of false discoveries.
Three practical examples
- A research team screens many candidate associations and prioritizes discoveries for independent replication.
- An AI team compares a predefined set of performance hypotheses across evaluation slices, documenting dependence and the chosen adjustment.
- A product analytics group uses an FDR-controlled exploratory screen before committing resources to confirmatory experiments.
Think of it this way
Think of screening a basket of candidates while controlling the expected share of false selections across repeated baskets. It does not label which individual items are false or promise that every basket has the same composition.
Worked example
Consider five invented, already sorted p-values: 0.006, 0.018, 0.029, 0.041, and 0.200. Assume the conditions for BH control hold and choose q = 0.05.
The rank thresholds are 0.01, 0.02, 0.03, 0.04, and 0.05. Rank three passes because 0.029 ≤ 0.03; ranks four and five fail. The largest passing rank is three, so the first three hypotheses are rejected.
The BH-adjusted p-values are 0.030, 0.045, approximately 0.04833, 0.05125, and 0.200. Comparing them with 0.05 gives the same result.
These numbers do not reveal V, the number of selected hypotheses whose nulls are actually true. Consequently they do not reveal the realized false discovery proportion. The FDR guarantee comes from the valid procedure and assumptions, not from knowing the truth of this particular list.
Illustrative case study
Fictional case: Horizon Analytics screens many potential predictors of service delays. It applies BH to a prespecified family and identifies candidates for a separate validation study. Managers initially interpret the target as a guarantee about each candidate. The team corrects that explanation and reports effect sizes, uncertainty, and replication results. This invented case separates a screening guarantee from certainty about individual findings.
Common misconceptions and mistakes
Do not confuse FDR with V/m, the proportion of all tests that are false rejections. Its denominator is the number of discoveries, with a zero-discovery convention.
Do not assume adjusted p-values are posterior probabilities that hypotheses are false or true. Do not ignore dependence conditions.
Applying an adjustment only to a hand-picked subset of significant results generally fails to represent the original testing family.
Business use
FDR control can make large exploratory analyses more useful by balancing discovery with a defined error criterion. Businesses should choose it according to the consequences of false selections and the availability of follow-up. When even one false claim is unacceptable within a family, a familywise criterion may be more aligned with the decision.
People also ask
Can one result list contain more false findings than the target fraction?
Yes.
What if there are no rejections?
The false discovery proportion is defined as zero.
Does BH always require independent tests?
Independence is sufficient, and specified positive-dependence conditions also support it; arbitrary dependence needs additional care.
Check your understanding
How many hypotheses are selected in the example?
Answer: Three.
Why is the fourth not selected?
Answer: Its adjusted p-value exceeds 0.05 and no higher rank passes.
Can we calculate the realized false discovery proportion from p-values alone?
Answer: No; the truth status of selected nulls is unknown.
Related terms
- Familywise error rate
- Multiple testing
- Benjamini–Hochberg
- Benjamini–Yekutieli
- false discovery proportion
- adjusted p-value
- statistical power
Learn next
Learn the difference between a realized error proportion and its expectation. Then practice BH step-up calculations and examine how dependence affects guarantees.
More in Statistics and probability
- Factorial design - A factorial design is an experimental arrangement that studies two or more factors together by combining their levels.
- Failure bias - Failure bias is an informal label for drawing general conclusions from visible failures while overlooking relevant successes or other cases that did not fail.
- Failure rate - Failure rate, in lifetime analysis, is the instantaneous rate at which units fail conditional on having survived to a given time.
- Fan chart (time series) - A fan chart is a time-series visualization that displays a central forecast together with nested probability bands for future outcomes.
- Fast probability integration - Fast probability integration, or FPI, refers to efficient numerical approaches for evaluating probabilities defined by uncertain model inputs and a...
- Field experiment - A field experiment evaluates an intervention in a setting that reflects real-world activity rather than only an artificial laboratory task.
Sources
SciPy, false_discovery_control documentation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.false_discovery_control.html R stats, p.adjust documentation: https://stat.ethz.ch/R-manual/R-devel/library/stats/html/p.adjust.html
Previous: Fairness metric Next: False negative