Does FWER control guarantee no false positives?

No. It bounds their probability across the family.

What is strong control?

Protection for every configuration of true and false null hypotheses.

Does Holm require independent p-values?

No, but the individual null p-values must be valid.

AI Glossary / Statistics and probability / Intermediate

Familywise error rate

Definition

Familywise error rate, or FWER, is the probability of rejecting at least one true null hypothesis within a specified family of hypothesis tests. A procedure controlling FWER at level α limits that probability to α under its stated assumptions. The family is the collection of claims considered together; it should be defined in relation to the study's purpose and analysis plan.

What it means

Testing many hypotheses creates several opportunities for a false positive. Even when each test individually has a 5% false-positive probability under its null, the probability of at least one false positive across the family can be much larger.
FWER concerns whether any false rejection occurs, not how many occur or what fraction of all discoveries are false. It is useful when protecting a set of confirmatory claims from even one false-positive conclusion is an important goal.

How it works

Specify the family of hypotheses before using the results to choose which claims to emphasize. The relevant family may include outcomes, treatment comparisons, subgroups, or other planned tests. Merely splitting one analysis into several tables does not settle the multiplicity question.
Let V be the number of true null hypotheses rejected. FWER = P(V ≥ 1). Strong control means this probability is bounded for every possible configuration of true and false nulls. Weak control only guarantees the bound when all hypotheses in the family are true.
The Bonferroni procedure tests each of m hypotheses at level α/m. If each null p-value is valid, the union bound ensures FWER ≤ α regardless of dependence among tests. This is a bound, not generally an equality.
Holm's method is a step-down improvement. Sort the p-values increasingly and compare the smallest with α/m, the next with α/(m − 1), and so on. Continue rejecting until the first comparison fails, then stop and retain the remaining hypotheses. It also gives strong control under arbitrary dependence when the individual p-values are valid.
Other procedures exploit specific model structures, dependence assumptions, or logical relationships between hypotheses. Choose the method according to the design and intended claims rather than searching retrospectively for whichever adjustment produces the most discoveries.
Account for repeated looks at accumulating data when relevant. A correction for several outcomes does not automatically control errors from repeatedly testing those outcomes over time. The full analysis procedure, including selection and stopping, determines the error guarantee.
Report adjusted p-values or corresponding simultaneous inference together with effect estimates and practical importance. Familywise control can reduce power, so study planning should consider sample size and which claims truly require joint protection. Failure to reject a null remains different from establishing equivalence or absence of a meaningful effect.

Three practical examples

A confirmatory experiment controls the chance of any false claim across several prespecified treatment comparisons.
A quality study compares multiple production settings while protecting the family of reported differences.
An AI evaluation defines a small set of primary performance claims and applies a familywise procedure before making deployment-related assertions.

Think of it this way

Think of guarding a collection of doors against any false alarm. Keeping the false-alarm chance small at each individual door does not automatically keep the chance small for the whole collection.

Worked example

Suppose an invented study conducts eight independent tests, all with true null hypotheses and exact individual false-positive probability 0.05. The probability of no false rejection is 0.95⁸. Therefore FWER = 1 − 0.95⁸ ≈ 0.33658, or 33.66%.
To use Bonferroni control at family level 0.05, test each hypothesis at 0.05/8 = 0.00625. The union bound gives FWER ≤ 8 × 0.00625 = 0.05 without requiring independence.
If the eight tests are independent and each has exact size 0.00625, the actual FWER is 1 − (1 − 0.00625)⁸ ≈ 0.04892, or 4.89%.
Thus the Bonferroni bound is slightly conservative in this particular independent example. With dependence or conservative individual tests, the actual probability can differ. These calculations describe a hypothetical repeated-testing setup, not the probability that any particular reported hypothesis is true.

Illustrative case study

Fictional case: Juniper Analytics compares a new service process with an existing one across several primary outcomes. The team initially treats each unadjusted significant result as a separate success. Reviewers identify the intended family of claims and apply a prespecified familywise approach. They also report effect sizes for outcomes that do not meet the adjusted threshold. This invented case illustrates joint error protection without suppressing descriptive information.

Common misconceptions and mistakes

Do not confuse FWER with the per-comparison error rate or with the false discovery rate. FWER controls the probability of any false rejection; FDR controls an expected proportion.
Do not assume α/m is the only valid method or that Bonferroni requires independent tests. It does require valid individual tests and an appropriate accounting of the family.
A nonsignificant omnibus test followed by selected unadjusted comparisons does not automatically provide strong familywise control for all possible configurations.

Business use

FWER control helps organizations make a clearly defined group of confirmatory claims with a transparent error criterion. It is particularly useful when a single false claim has substantial consequences. The decision still requires estimates, uncertainty, and operational relevance; statistical protection alone does not establish that an observed improvement is worth implementing.

Check your understanding

Why is the unadjusted example's FWER above 5%?

Answer: There are eight opportunities for a false rejection.

What is the Bonferroni threshold?

Answer: 0.00625 for each test.

Why is the actual adjusted FWER below 5% in the example?

Answer: The union bound can be conservative because false-rejection events can overlap.

Related terms

Multiple comparisons
Type I error
False discovery rate
Bonferroni correction
Holm procedure
simultaneous inference
strong control
statistical power

Learn next

Study the union bound and the distinction between weak and strong control. Then practice Holm's step-down procedure and compare familywise and false-discovery criteria for different research goals.

Sources

David Howell, Statistical Methods for Psychology glossary, “Familywise error rate”: https://www.uvm.edu/~statdhtx/methods8/Glossary/Glossary.html Holm, “A Simple Sequentially Rejective Multiple Test Procedure” (1979): https://www.ime.usp.br/~abe/lista/pdf4R8xPVzCnX.pdf R stats, p.adjust: https://stat.ethz.ch/R-manual/R-devel/library/stats/html/p.adjust.html

Previous: False precision Next: Fan chart (time series)