Is per-comparison control always inappropriate?

No; it answers an individual-test question, but broader claims need broader error accounting.

Does correction repair a biased sample?

No. Multiplicity control does not fix an invalid underlying test.

Is false discovery rate the same quantity?

No; it concerns false rejections relative to all rejections, with a zero-rejection convention.

AI Glossary / Statistics and probability / Intermediate

Per-comparison error rate

Definition

The per-comparison error rate is the Type I error probability associated with an individual hypothesis test: rejecting that test's null hypothesis when it is true. In a collection of tests, comparisonwise control protects each specified comparison separately. It does not, by itself, control the probability of making any false rejection across the whole collection.

What it means

A nominal level of 5% describes the procedure's behavior over repeated samples under a true null and the stated assumptions. It does not mean that a particular significant result has a 5% probability of being wrong.
Terminology varies across multiple-testing literature. Some formulations summarize errors by the expected number of false rejections divided by the total number of tests. State the definition being used, especially when some null hypotheses are false or individual test levels differ.

How it works

Define each hypothesis, the test statistic, the sampling model, and the rejection rule before evaluating the data. A valid level-alpha test has a Type I error probability no greater than alpha for its null model; discreteness or conservatism may make the actual probability smaller.
For several comparisons, identify the family of scientific or business claims. If each is tested separately at alpha, each retains its individual guarantee when its assumptions hold. Selecting only attractive results afterwards changes the interpretation of the reporting process.
Let V be the number of false rejections and let m be the total number of tests. The expected number of false rejections is the sum of the individual true-null rejection probabilities. This addition does not require the tests to be independent.
Under the complete null, if all m tests have exact Type I error probability alpha, E(V) equals m times alpha, and E(V)/m equals alpha. If only m0 nulls are true with that same probability, E(V)/m equals m0 times alpha divided by m.
Familywise error is different: it is the probability that V is at least one. With independent tests and all nulls true, that probability is 1 minus (1 minus alpha) to the power m. Without independence, that product formula is not generally valid.
Choose the error criterion that matches the claim. Bonferroni uses individual thresholds of a desired familywise level divided by m, giving a familywise upper bound without requiring independence. Other procedures target familywise error or the false discovery rate with different assumptions and power tradeoffs.
Report the individual threshold, whether adjustment was used, and the full comparison family. The test's calibration also depends on ordinary requirements such as appropriate sampling and treatment of dependence.

Three practical examples

AI benchmarking: a team distinguishes a single planned model comparison from an exploratory panel of many metrics.
Marketing: analysts explain why testing many audience segments at 5% can produce false discoveries.
Quality monitoring: engineers separate an individual alarm's false-positive probability from the risk of any alarm across many checks.

Think of it this way

Think of each comparison as one detector with its own false-alarm setting. Installing many detectors does not change an individual detector's setting, but it changes the chance that at least one detector sounds falsely.

Worked example

Consider 20 independent tests, all with true null hypotheses and an exact individual false-rejection probability of 0.05. The expected number of false rejections is 20 × 0.05 = 1. The per-comparison rate is 5%.
The probability of no false rejections is 0.95 to the power 20, approximately 0.35849. Therefore, the probability of at least one is approximately 0.64151, or 64.15%. An expected count of one does not mean every set of 20 tests produces exactly one error.
To target a familywise level of 0.05 using Bonferroni, use an individual threshold of 0.05/20 = 0.0025. If these tests remain independent and have exact size 0.0025, the familywise probability is 1 minus 0.9975 to the power 20, approximately 0.04883. The Bonferroni guarantee of at most 0.05 does not require that independence assumption.

Illustrative case study

Fictional case: Harbor Analytics screens many customer segments for a possible benefit from an AI assistant. The team labels the first analysis exploratory, retains all tested segments in its report, and designs a separate confirmatory evaluation for its selected claim. This avoids presenting an unadjusted individual threshold as a guarantee about the entire search.

Common misconceptions and mistakes

Do not call a 5% threshold a 5% probability that a reported finding is false. Do not confuse expected error counts, familywise error, and the expected proportion of false discoveries.
Dependence matters for the exact familywise formula, even though linearity of expectation still permits summing individual error probabilities.
A preliminary significant omnibus test does not universally make all subsequent unadjusted comparisons safe. Missing comparisons and repeated inspection also need consideration.

Business use

This distinction helps organizations choose an evidence standard before launching a large evaluation. Reporting individual results alongside the intended error criterion makes it easier to judge exploratory signals, confirmatory decisions, and the costs of false alarms and missed improvements.

Check your understanding

In the example, why is 64.15% different from 5%?

Answer: One concerns any error across 20 independent tests; the other concerns one test.

Which calculation does not need independence?

Answer: Adding individual false-rejection probabilities to obtain the expected number of errors.

Related terms

Learn next

Learn next how to specify a comparison family, distinguish strong from complete-null familywise control, and choose between confirmatory and exploratory multiple-testing procedures.

Sources

Sources: SAS/STAT, GLM Multiple Comparisons, https://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/statug_glm_a0000000862.htm ; NIST/SEMATECH, Bonferroni's method, https://itl.nist.gov/div898/handbook/prc/section4/prc473.htm . These primary references explain comparisonwise versus overall error and simultaneous protection. The probability example and fictional case are original.

Previous: Penalty term Next: Per-word entropy