Are multiple comparisons always a problem?

They are a routine part of analysis, but they can distort conclusions when their consequences are ignored.

Must every test in an organization belong to one family?

No. The family should reflect the scientific or decision context and be justified rather than chosen opportunistically.

Can exploratory findings still be useful?

Yes. They can generate hypotheses and guide future studies, provided their exploratory origin is clear.

AI Glossary / Statistics and probability / Intermediate

Multiple comparisons

Definition

Multiple comparisons arise when an analysis examines several hypotheses, contrasts or statistical relationships and draws conclusions from the collection. The concern is that each comparison creates an opportunity for a false positive, so the overall evidence cannot always be interpreted using single-comparison error rates. Comparisons may involve group differences, outcomes, subgroups, model specifications or repeated looks at data. Multiple comparisons are not inherently improper. They require an analysis and reporting strategy that makes the family of questions and the intended error control explicit.

What it means

A significance threshold describes the long-run behavior of a test under specified assumptions. Applying that threshold many times changes the chance of obtaining at least one apparently significant result when the tested null hypotheses are true. Selecting and reporting only the most favorable result hides the broader search that produced it. Multiplicity can occur even when a report displays only one final finding, because the analyst may have tried many alternatives first. The relevant issue is the inferential process, not simply the number of p-values printed in the final table.

How it works

Identify the set of questions that will support a common conclusion or decision. Distinguish planned confirmatory tests from exploratory searches and document outcomes, subgroups and analysis variants. Choose an error criterion, such as family-wise error or false discovery rate, and a procedure appropriate to the design. Where possible, reserve independent data for confirming discoveries made during exploration. Report the comparisons performed, effect estimates and uncertainty, including results that do not support the preferred explanation. Repeated monitoring and adaptive decisions may require sequential methods rather than a correction designed for a fixed batch of tests.

Three practical examples

Product experiments: examine several outcome measures for the same proposed change.
Group analysis: compare every pair among several experimental conditions.
Model research: search many feature subsets or subgroups and highlight the strongest apparent association. These settings differ in purpose, but each can produce selective conclusions if the extent of the search is ignored.

Think of it this way

Think of buying many lottery tickets and then discussing only the winning ticket. The observed win looks different when the number of attempts is known. Statistical testing is not literally a lottery, but the analogy highlights why an unusual result needs to be interpreted in the context of the opportunities that could have produced it.

Worked example

Suppose twenty independent tests each have an exact 5% false-positive probability and all twenty null hypotheses are true. The probability of no false positives is 0.95 to the twentieth power, approximately 0.3585. Therefore the probability of at least one false positive is about 0.6415, or 64.15%, not 5%. The expected number of false positives is twenty times 0.05 = 1; this expectation is different from the probability of at least one. If a Bonferroni procedure allocates a family-wise target of 0.05 across the twenty tests, each threshold is 0.05/20 = 0.0025. Its family-wise bound does not require independence when the individual tests are valid. This original example's 64.15% calculation does require independence. Separately, comparing all pairs among six groups creates six times five divided by two = 15 comparisons.

Illustrative case study

In a fictional case, Elm Insights evaluates a website change using several outcomes and many customer subgroups. One subgroup shows a small p-value, but the team initially omits the unsuccessful comparisons from its summary. Reviewers reconstruct the analysis history and label the subgroup result exploratory. The team plans an independent confirmation with a stated primary outcome and subgroup definition. Its final report preserves the initial observation as a hypothesis while avoiding the claim that an unrestricted search provided the same evidence as a prespecified test.

Common misconceptions and mistakes

A 5% threshold does not mean that 5% of all reported significant findings are necessarily false. The proportion depends on the underlying effects, power and selection process. The formula 1 - (1 - alpha)^m requires independent tests with the stated false-positive probabilities; it is not universal. Multiplicity is not limited to pairwise comparisons after ANOVA. A nonsignificant omnibus result is not the only setting where selective testing can mislead. Adjustments do not cure confounding or poor measurement, and a statistically adjusted finding can still be too small to matter practically.

Business use

Businesses should define primary decisions and outcomes before data are examined, while allowing transparent exploration for new ideas. Maintain an analysis record that captures alternate specifications and subgroup searches. Consider the number of comparisons when planning sample size and the strength of evidence required for action. Report effect sizes and costs of errors alongside significance. When ongoing dashboards trigger decisions repeatedly, establish a monitoring policy rather than treating every refresh as a new isolated experiment. Clear separation of discovery and confirmation can preserve useful learning without overstating evidence.

Check your understanding

Twenty independent true-null tests each use a 5% threshold. Is the chance of at least one false positive 5%?

Answer: No. Under those assumptions it is 1 - 0.95^20, approximately 64.15%.

Related terms

False discovery rate
p-hacking
Multiple testing
selective reporting
family-wise error
preregistration
exploratory analysis
confirmatory analysis

Learn next

Learn the difference between per-test and family-level error rates. Then map the complete set of choices in a real analysis and decide which findings need independent confirmation or a suitable multiplicity procedure.

Sources

NIST/SEMATECH e-Handbook, How can we make multiple comparisons?: https://itl.nist.gov/div898/handbook/prc/section4/prc47.htm NIST/SEMATECH e-Handbook, Bonferroni's method: https://itl.nist.gov/div898/handbook/prc/section4/prc473.htm R Core Team, p-value adjustment methods: https://stat.ethz.ch/R-manual/R-devel/library/stats/html/p.adjust.html

Previous: Multiple comparison techniques Next: Multiple correlation coefficient