Does Holm require independent tests?

No, provided the individual p-values are valid.

Is it more conservative than ordinary Bonferroni?

It is no more conservative in terms of rejections for the same family and alpha.

Does it control FDR instead?

Its stated target is family-wise error.

AI Glossary / Statistics and probability / Intermediate

Holm - Bonferroni method

Definition

The Holm - Bonferroni method is a step-down multiple-testing procedure that controls the family-wise error rate: the probability of rejecting at least one true null hypothesis within a specified family. It orders valid individual p-values from smallest to largest and compares them with progressively less stringent Bonferroni thresholds, stopping at the first failure to reject.

What it means

Testing many claims separately at 5% can produce an overall false-positive risk greater than 5%. Holm's method addresses the family of claims together rather than treating each result in isolation.
It provides strong family-wise control, meaning the bound applies regardless of which null hypotheses are actually true. It is valid under arbitrary dependence among the tests when each input p-value is valid under its own null. It does not repair biased tests or poorly defined hypotheses.

How it works

Define the family and its error level alpha before inspecting the results. The family might consist of several planned comparisons supporting one product claim. What belongs in it depends on the inferential objective, not on which results happened to look promising.
For m hypotheses, sort the p-values so p_(1) <= ... <= p_(m), retaining their original labels. Compare the smallest with alpha/m. If it passes, reject that null and compare the next with alpha/(m - 1).
At rank i, the threshold is alpha/(m - i + 1). Continue only while every earlier comparison passes. At the first p-value above its threshold, stop rejecting: that hypothesis and all later hypotheses remain unrejected, even if a later p-value would pass its own threshold in isolation.
Adjusted p-values provide an equivalent reporting route. In sorted order, calculate the cumulative maximum of (m - j + 1)*p_(j) for j up to i, capped at one. Restore the original hypothesis order afterward. The cumulative maximum preserves the step-down stopping logic.
Compare adjusted p-values with alpha, or compare raw p-values with the sequential thresholds. Do not apply both adjustments simultaneously. Report which values are raw and which have been adjusted.
Holm rejects at least every hypothesis rejected by ordinary Bonferroni at the same alpha and family, and can reject more. This advantage does not mean the procedure is always powerful enough for a given study; effect sizes, variability, and sample size still govern detection.
The method controls family-wise error rather than the expected proportion of false discoveries among rejections. Procedures targeting false discovery rate answer a different error-control question and should not be substituted without considering the study objective.
Preserve the validity of the underlying tests. Dependence between tests is allowed, but invalid sampling assumptions, selective reporting, or unaccounted repeated looks can still undermine the analysis. Additional sequential monitoring requires an appropriate plan beyond sorting a single fixed set of p-values.

Three practical examples

An AI team compares a proposed model with a baseline on five prespecified evaluation outcomes.
A product experiment makes several planned treatment-versus-control comparisons and wants to control the chance of any false rejection in that family.
A quality team evaluates a small set of related process claims using valid tests with correlated outcomes.

Think of it this way

Think of a row of gates arranged from the strongest evidence to the weakest. The first gate is strict; later gates relax only after all earlier gates have been passed. Once a gate fails, you cannot skip it and resume farther down the row.

Worked example

Suppose five valid p-values, already sorted, are 0.004, 0.012, 0.018, 0.019, and 0.200. Set the family-wise level to alpha = 0.05.
The sequential thresholds are 0.0100, 0.0125, approximately 0.01667, 0.0250, and 0.0500. The first p-value passes because 0.004 <= 0.0100. The second passes because 0.012 <= 0.0125. The third fails because 0.018 > 0.01667.
Stop there: reject only the first two null hypotheses. The fourth is not rejected even though 0.019 is below its individual rank threshold of 0.025. Continuing after the third failure would not be Holm's procedure.
For adjusted p-values, the initial products are 5 x 0.004 = 0.020, 4 x 0.012 = 0.048, 3 x 0.018 = 0.054, 2 x 0.019 = 0.038, and 1 x 0.200 = 0.200. Taking cumulative maxima gives 0.020, 0.048, 0.054, 0.054, and 0.200.
Only the first two adjusted values are at most 0.05, matching the sequential result. Ordinary Bonferroni would use 0.01 for all five raw p-values and reject only the first. These decisions do not reveal which nulls are actually true.

Illustrative case study

Fictional case: Alder Evaluation plans five outcome comparisons for an AI assistant and chooses family-wise error control because any claimed improvement will appear in a combined performance statement. The analyst records the full family before examining results.
After computing valid tests, the team applies Holm adjustment and publishes all five estimates with clearly labeled raw and adjusted p-values. Two hypotheses pass the chosen threshold, while the remaining results are described with their uncertainty rather than called evidence of no effect.
Managers also examine whether the improvements are operationally meaningful. The procedure governs false-positive risk across claims; it does not decide whether a measured change is worth deployment.

Common misconceptions and mistakes

Common mistakes include continuing after the first failed threshold, multiplying each p-value without taking the cumulative maximum, and adjusting already adjusted p-values again.
Do not define the family using only significant findings, interpret an adjusted p-value as the probability that the null is true, or claim the method eliminates every chance of a false conclusion. Unrejected hypotheses remain uncertain rather than proven true.

Business use

Holm is useful for a defined set of confirmatory comparisons when controlling the chance of any false rejection is the priority. It is simple to communicate and does not require independence among tests. Reports should still include estimated effects, relevant uncertainty, and the complete comparison family.

Check your understanding

Where does the worked procedure stop?

Answer: At the third p-value.

Why is the fourth adjusted p-value 0.054 rather than 0.038?

Answer: The cumulative maximum enforces the earlier stopping point.

How many hypotheses are rejected?

Answer: Two.

Related terms

False discovery rate
Multiple testing
family-wise error rate
Bonferroni correction
adjusted p-value
step-down procedure
statistical power
confirmatory analysis

Learn next

Practice sorting and restoring labeled p-values, then compare Holm with ordinary Bonferroni. Study how to define a hypothesis family and why significance decisions should accompany effect-size interpretation.

Effect size

Sources

Sture Holm, A Simple Sequentially Rejective Multiple Test Procedure, original paper: https://www.ime.usp.br/~abe/lista/pdf4R8xPVzCnX.pdf R stats, adjusted p-values: https://stat.ethz.ch/R-manual/R-devel/library/stats/html/p.adjust.html

Previous: Holdout set Next: Holt-Winters forecasting