Is a nominal per-comparison level always the actual error rate?

No. It is a bound or target whose validity depends on the test and assumptions.

Does Bonferroni require independent tests?

No, its union-bound guarantee does not.

Is per-comparison error the false discovery rate?

No. False discovery rate concerns the expected proportion of false rejections among rejections, with a defined convention when none occur.

AI Glossary / Statistics and probability / Intermediate

Error rate per comparison ( PC )

Definition

Error rate per comparison is the probability of falsely rejecting the null hypothesis for a particular comparison when that null is true. It describes Type I error control at the individual-comparison level within an analysis that may contain several comparisons. A nominal five-percent level for each test does not generally limit the probability of at least one false rejection across the whole collection to five percent.

What it means

Consider comparing a reference system with several alternatives. Each comparison can have its own false-positive probability under its null hypothesis. That individual probability differs from the chance that the complete analysis makes any false-positive claim.
The supplied term uses PC for per-comparison error. Terminology varies across texts, so reports should define the probability being controlled rather than relying on an abbreviation alone. It is especially important to distinguish a specified comparison from whichever comparison happens to look most impressive after inspecting the results.

How it works

State each null hypothesis and the intended testing procedure. For comparison j, a valid level-alpha_j procedure has Type I error probability no greater than alpha_j under the distributions covered by that null and its assumptions. The actual probability may be lower, for example with conservative or discrete tests.
Define the family of claims that will be interpreted together. The familywise error rate is the probability of at least one false rejection among the true nulls in that family. It depends on the individual rejection events and their dependence. A per-comparison guarantee alone is not the same as a familywise guarantee.
If m true-null tests have independent rejection events, each with exact error probability alpha, the chance of no false rejection is (1 - alpha) to the power m. Consequently the chance of at least one is 1 - (1 - alpha) to the power m. Independence is essential for that exact product formula.
Without requiring independence, the union bound gives a familywise probability no larger than the sum of the true-null error probabilities. Bonferroni uses this fact: testing each of m prespecified comparisons at alpha/m controls the familywise rate at most alpha when the individual tests are valid. Other procedures can exploit additional structure or use stepwise rules.
Choose the error criterion to match the purpose. Confirmatory claims may require familywise control, while large exploratory screens may use a justified false-discovery-rate procedure. Neither criterion is interchangeable with the probability that a particular reported finding is false.
Repeated looks, outcome selection, subgroup searches, and model choices can expand the effective set of opportunities for false claims. Describe the complete analysis procedure rather than reporting only the final tests that were retained.

Three practical examples

A laboratory reports the nominal Type I error level for each of several planned treatment comparisons and separately states its familywise adjustment.
An AI benchmark compares a baseline with multiple model variants, recognizing that selecting the smallest p-value changes the interpretation of the selected claim.
An exploratory analysis distinguishes unadjusted individual p-values from an error-controlled list of discoveries.

Think of it this way

Think of checking several doors, each with a small chance of triggering a false alarm. The false-alarm rate for one specified door differs from the chance that at least one alarm sounds somewhere in the building. The relationship also depends on whether the alarms share common causes.

Worked example

Suppose an original teaching scenario contains ten independent tests, all with true null hypotheses and exact per-comparison error probability 0.05.
Each specified test has a five-percent chance of false rejection. Across all ten, the chance of at least one false rejection is 1 - 0.95 to the tenth power, approximately 0.4013, or 40.13%.
The expected number of false rejections is ten times 0.05 = 0.5. This is an average count over repeated analyses, not a 50% probability of at least one error. Linearity of expectation gives this count even without independence when the individual error probabilities remain 0.05.
For a Bonferroni familywise target of 0.05, use a per-test threshold of 0.05/10 = 0.005. The union bound guarantees a familywise rate at most 0.05 under valid individual tests. Under the example's additional exactness and independence assumptions, the actual rate is 1 - 0.995 to the tenth power, approximately 0.04889.
These are properties of specified repeated-testing procedures, not posterior probabilities attached to an observed p-value.

Illustrative case study

Fictional case: Spruce Search evaluates several retrieval changes and highlights the one with an unadjusted p-value below 0.05. Its initial report calls the chance of a false product-level claim five percent.
The analyst explains the distinction between one planned comparison and selecting a winner from many. The team documents the comparison family, uses an appropriate multiplicity procedure, and validates the chosen change on fresh evaluation data. It reports effect size and practical value alongside the statistical decision, rather than treating an individual threshold as a complete launch criterion.

Common misconceptions and mistakes

Mistake: interpreting a five-percent per-comparison level as five-percent familywise control.
Mistake: multiplying probabilities without an independence justification.
Mistake: confusing expected false-positive count with the probability of at least one false positive.
Mistake: describing a p-value as the probability that its null hypothesis is true.
Mistake: choosing the comparison family after seeing which results are significant.
Mistake: assuming a multiplicity correction repairs invalid individual tests, biased samples, or selective data handling.

Business use

Businesses need this distinction when running many experiments, comparing vendors, searching customer segments, or evaluating model variants. The reporting standard should state which comparisons were planned and what error criterion governs the decision. Adjustments can reduce power, so planning fewer decision-relevant comparisons and collecting adequate evidence can improve efficiency. Statistical control should support, rather than replace, assessment of effect size, cost, and operational consequences.

Check your understanding

In the example, is 0.5 the probability of any false rejection?

Answer: No. It is the expected number; the probability is approximately 0.4013 under independence.

What does an individual five-percent guarantee control?

Answer: The false-rejection probability for that specified true-null comparison under its assumptions.

Related terms

Learn next

compare individual and familywise guarantees, calculate the union bound, and define a testing family before examining a set of experimental outcomes.

Sources

Sources: Howell, Statistical Methods for Psychology glossary, Error rate per comparison: https://www.uvm.edu/~statdhtx/methods8/Glossary/Glossary.html . NIST/SEMATECH handbook, Bonferroni's method: https://www.itl.nist.gov/div898/handbook/prc/section4/prc473.htm . The ten-test probability calculations are original.

Previous: Error rate Next: Error (Statistical error)