AI Glossary / Statistics and probability / Intermediate

Multiple comparison techniques

Definition

Multiple comparison techniques are statistical procedures that account for making a family of comparisons rather than interpreting each comparison as if it were the only one performed. They can adjust rejection thresholds, p-values or confidence intervals to control a defined error criterion. Common methods include Bonferroni and Holm procedures for family-wise error control, Tukey procedures for pairwise mean comparisons, and false-discovery-rate procedures for certain screening tasks. The appropriate method depends on the questions, study design, dependence assumptions and consequences of false conclusions.

What it means

Making many comparisons creates more opportunities to obtain an apparently unusual result by chance. A technique for multiple comparisons specifies how that multiplicity is handled. Family-wise error rate concerns the probability of at least one false rejection in a defined family. False discovery rate concerns the expected proportion of false rejections among all rejections, with the proportion defined as zero when none occur. These are different goals. Selecting a method therefore requires more than choosing the smallest adjusted p-values: it requires deciding what type of error protection the analysis needs.

How it works

Define the family of hypotheses and error criterion before inspecting which results look favorable. Use valid individual tests or a joint model appropriate to the design. Bonferroni compares each p-value with alpha divided by the number of tests. Holm orders p-values and applies sequential thresholds, stopping at the first nonrejection. Tukey methods use the structure of pairwise comparisons among means under their model assumptions. Other procedures address different families or false discovery control. Report the selected method, family size, assumptions and adjusted results, and accompany significance decisions with effect estimates and suitable uncertainty intervals.

Three practical examples

• Experimental analysis: compare several conditions while controlling error across the planned conclusions.
• Pairwise group comparison: use a method designed for all pairs of means when that is the actual question.
• Large-scale screening: apply a justified false-discovery-rate procedure when selecting promising candidates for further investigation. A method appropriate for exploratory screening may not provide the protection required for a small set of consequential confirmatory claims.

Think of it this way

Think of setting an alarm policy for a whole control room rather than one sensor. If each sensor is allowed its own false-alarm chance without considering the collection, the room may generate many misleading alerts. Different policies can control the chance of any false alarm or the expected fraction of false alarms among alerts. Those objectives lead to different rules.

Worked example

Consider four valid p-values in a prespecified family: 0.004, 0.014, 0.021 and 0.060. At family-wise alpha = 0.05, Bonferroni compares each with 0.0125 and rejects only the first hypothesis. Holm compares the ordered values successively with 0.05/4 = 0.0125, 0.05/3 approximately 0.01667, 0.05/2 = 0.025 and 0.05. The first three pass and the fourth fails, so Holm rejects the first three. Its adjusted p-values are 0.016, 0.042, 0.042 and 0.060, applying the cumulative maximum needed to preserve ordering. Bonferroni's adjusted values are 0.016, 0.056, 0.084 and 0.240. This original example illustrates that methods controlling the same error criterion can differ in power. It does not justify selecting a method only after observing which gives the preferred conclusion.

Illustrative case study

In a fictional case, Rowan Research compares several workflow variants. The team initially reports every unadjusted pairwise result below 0.05. Reviewers ask it to define the family of claims and choose an appropriate procedure. Analysts distinguish their planned comparisons from exploratory follow-up questions and present adjusted evidence alongside estimated differences. Management learns that some small p-values do not remain convincing under the selected family-wise procedure, while some statistically supported differences are too small to matter operationally.

Common misconceptions and mistakes

Adjustment does not repair biased data, an invalid test or an incorrectly specified model. Family-wise control and false discovery control are not interchangeable. A nonsignificant adjusted result does not prove there is no effect. A significant omnibus ANOVA is not universal permission to report unrestricted unadjusted comparisons. Dependence assumptions differ across methods, so a procedure should not be chosen solely by name. Using adjusted p-values with ordinary unadjusted intervals can create apparently inconsistent messages unless the differing inferential targets are clearly explained.

Business use

Businesses should identify which conclusions are intended to support action and organize the testing family accordingly. Document the procedure before results drive the choice, and include all relevant comparisons rather than only favorable ones. Match interval estimates to the intended simultaneous inference where appropriate. Consider power and sample-size implications during study design. For exploratory work, preserve a distinction between candidate discovery and independent confirmation. The aim is a clear account of the uncertainty across the set of claims, while retaining attention to effect size and decision consequences.

People also ask

Is Bonferroni always required?

No. Other methods may better match the comparison structure or error objective.

Does Holm require independent p-values?

Its standard family-wise guarantee applies under arbitrary dependence when the individual p-values are valid.

Does controlling false discovery rate guarantee no false discoveries?

No. It controls an expected proportion under the procedure's assumptions, not the truth of every selected finding.

Check your understanding

Why should a team state whether it controls family-wise error or false discovery rate?

Answer: The two criteria protect against different patterns of error, so identical numerical targets do not mean identical guarantees.

Related terms

• False discovery rate
• Multiple testing
• family-wise error rate
• Bonferroni
• Holm
• Tukey
• simultaneous confidence interval
• statistical power

Learn next

Compare the error criteria and work through Bonferroni and Holm calculations. Then study methods tailored to the planned comparison family and how their assumptions affect interpretation.

More in Statistics and probability

• Multinomial test - A multinomial test evaluates whether observed category counts are compatible with a specified vector of category probabilities under a multinomial sampling...
• Multioutput regression - Multioutput regression predicts two or more numerical target variables for each input example.
• Multiple baseline design - A multiple baseline design is a single-case experimental design in which an intervention is introduced at different times across participants, behaviors or...
• Multiple comparisons - Multiple comparisons arise when an analysis examines several hypotheses, contrasts or statistical relationships and draws conclusions from the collection.
• Multiple correlation coefficient - The multiple correlation coefficient, conventionally written R, measures how strongly an outcome is linearly associated with its best linear prediction from...
• Multiple regression - Multiple regression models the relationship between one outcome and two or more predictors.

Sources

NIST/SEMATECH e-Handbook, Multiple comparisons: https://itl.nist.gov/div898/handbook/prc/section4/prc47.htm R Core Team, Adjust P-values for Multiple Comparisons: https://stat.ethz.ch/R-manual/R-devel/library/stats/html/p.adjust.html

Previous: Multiple baseline design Next: Multiple comparisons