Does it require independent tests?

No, provided the individual tests are valid.

Must an overall ANOVA be significant first?

The Bonferroni principle itself does not require that preliminary gate; follow the analysis plan and the assumptions of the individual comparisons.

Can adjusted p-values exceed one?

Products can, but reported adjusted probabilities are capped at one.

AI Glossary / Statistics and probability / Advanced

Bonferroni test

Definition

A Bonferroni test is a multiple-testing procedure that controls the probability of making at least one false rejection across a specified family of hypothesis tests. With m tests and a familywise significance level alpha, the ordinary equal-allocation procedure compares each unadjusted p-value with alpha divided by m. It applies a common correction to otherwise valid individual tests.

What it means

An AI evaluation may compare several systems, outcomes, or user groups. Testing each comparison at the usual five-percent level does not generally keep the probability of any false positive across the whole family at five percent. The Bonferroni procedure makes the individual thresholds stricter to protect the combined family.
The procedure does not create evidence or change an estimated effect. It changes how strong the statistical evidence must be for a rejection within the selected family. A result can fail the adjusted threshold while still suggesting an effect worth estimating more precisely in a future study.

How it works

Define the hypotheses, comparison family, and desired familywise error allowance. Obtain valid p-values from statistical tests appropriate to the study design. For m comparisons, reject an individual null hypothesis when its p-value is no greater than alpha/m.
An equivalent approach multiplies each p-value by m and caps the result at one. Compare these adjusted p-values with alpha. Do one adjustment or the other; applying both would impose an unintended extra correction.
The justification uses the union bound: adding the false-rejection probabilities for the true null hypotheses bounds the probability of at least one such error. Independence among tests is unnecessary. However, biased sampling, inappropriate standard errors, or unaccounted repeated examination can invalidate the individual p-values and therefore the intended guarantee.

Three practical examples

A company compares an AI assistant against its existing workflow on five planned quality outcomes. It defines whether those outcomes form one family and applies the corresponding threshold consistently.
Researchers compare three candidate forecasting systems with a common baseline. They adjust the three planned comparisons rather than counting only whichever results appear promising.
An evaluation reports several prespecified subgroup differences. Adjustment addresses multiple statistical comparisons, while effect sizes and uncertainty still describe how large and practically relevant those differences might be.

Think of it this way

Imagine dividing a fixed error allowance among several checkpoints. Each checkpoint receives a smaller share because an error at any checkpoint matters to the overall assessment. The analogy describes how error probabilities are budgeted, not a literal guarantee that exactly that number of errors will occur.

Worked example

Consider an invented evaluation with five prespecified comparisons and a familywise significance level of 0.05. The unadjusted p-values are 0.004, 0.009, 0.012, 0.041, and 0.28.
The Bonferroni threshold is 0.05/5 = 0.01. The first two p-values meet this threshold; the remaining three do not. Thus, the procedure rejects the first two null hypotheses.
Multiplying each p-value by five gives 0.020, 0.045, 0.060, 0.205, and 1.400. After capping at one, the adjusted values are 0.020, 0.045, 0.060, 0.205, and 1.000. Comparing these with 0.05 produces exactly the same two rejections.
Without adjustment, four comparisons would meet a 0.05 threshold. That does not mean two of those four findings are known to be false. It means the two additional findings do not satisfy this procedure's stronger familywise evidence requirement.
Now suppose the five reported comparisons were actually selected from twenty examined comparisons. Dividing by five does not automatically account for that larger search. If the appropriate family contains all twenty tests, the ordinary threshold becomes 0.05/20 = 0.0025, and none of the five displayed values meets it. A carefully justified alternative selection-aware procedure could have different rules, but simply omitting results is not such a procedure.
For five simultaneous confidence intervals with overall coverage of at least 95 percent, the corresponding Bonferroni approach uses individual intervals with coverage of at least 99 percent. For equal-tailed intervals, each tail receives probability 0.005, assuming the underlying interval construction is valid.

Illustrative case study

Fictional company Willow Metrics evaluates a summarization assistant on accuracy, readability, completeness, speed, and formatting consistency. The team specifies these five comparisons before collecting the evaluation data.
Its analyst reports the raw and adjusted p-values, estimated differences, and uncertainty intervals together. A small adjusted p-value for speed does not override an unfavorable accuracy estimate. An outcome that misses the adjusted threshold remains inconclusive rather than being labeled identical between systems. The team uses practical performance requirements alongside the statistical results when deciding what to investigate next. This is an illustrative scenario, not a real deployment report.

Common misconceptions and mistakes

An adjusted p-value is not the probability that the null hypothesis is true. Nor does a five-percent familywise threshold assign five-percent error probability to each reported conclusion.
Failure to reject does not prove equality, equivalence, or absence of a useful effect.
Bonferroni adjustment can reduce power, especially with many comparisons. This tradeoff should inform study planning rather than justify hiding tests after seeing their results.
Holm's procedure also uses Bonferroni reasoning but applies sequential thresholds. It is a distinct method, not another name for the ordinary single-step procedure described here.

Business use

Use the procedure when controlling even one false rejection across a defined set of comparisons is relevant to the decision. Record the family, test count, thresholds, and all examined outcomes. Report effect sizes and operational consequences alongside significance; statistical rejection alone does not establish business value.

Check your understanding

What is the threshold for five tests at familywise alpha 0.05?

Answer: It is 0.01.

Does p = 0.012 pass?

Answer: No; its adjusted value is 0.060.

Does that prove no effect?

Answer: No. It indicates that this test does not reject under the specified procedure.

Related terms

Bonferroni inequality
Familywise error rate
Multiple comparisons
adjusted p-value
Holm procedure
statistical power
simultaneous confidence intervals

Learn next

Compare single-step Bonferroni with Holm's sequential method. Study how a prespecified comparison family and adequate sample size support interpretable multiple-testing results.

Sample size

Sources

NIST/SEMATECH, Bonferroni's Method: https://www.itl.nist.gov/div898/handbook/prc/section4/prc473.htm . Statsmodels, multipletests documentation: https://www.statsmodels.org/stable/generated/statsmodels.stats.multitest.multipletests.html . David C. Howell, glossary: https://www.uvm.edu/~statdhtx/methods8/Glossary/Glossary.html . Worked data and company scenario are original illustrations.

Previous: Bonferroni inequality Next: Boolean