Does Bonferroni need independent tests?

No. Its upper bound uses the sum of individual error bounds; independence is needed only for the product calculation used in the example.

Is a conservative test invalid?

No. Its actual Type I error can be below the stated level, although power may be lower.

Is nominal significance related to a nominal measurement scale?

They are different concepts: one concerns test thresholds, the other unordered categories.

AI Glossary / Statistics and probability / Advanced

Nominal significance level

Definition

A nominal significance level is the stated significance threshold for a statistical test, commonly written as alpha. In multiple-comparison settings, it often refers to the individual test's threshold before accounting for the whole family of tests. The stated level must be distinguished from the procedure's actual Type I error probability and from the probability of at least one false rejection across several tests.

What it means

An evaluation plan might say that each of four AI features will be tested at the five-percent level. That statement describes an individual-test rule. It does not automatically mean that the chance of any false positive across all four tests is five percent.

“Nominal” also draws attention to assumptions. A stated threshold has its intended error interpretation only when the test, sampling design and analysis procedure justify it. Approximation, discreteness or model misspecification can make actual behavior differ.

How it works

Define the hypothesis, test statistic and decision rule. For a valid test at level alpha, rejection probability under a true null should not exceed alpha within the stated model. An exact continuous test may attain that boundary; a discrete test may be conservative.

For multiple tests, define the family and the error criterion separately. Individual error probabilities do not generally equal the probability of at least one false rejection. Dependence between tests affects that family probability, so multiplying probabilities requires an explicit independence assumption.

Three practical examples

1. Feature evaluation: a product team tests four independent experimental outcomes. Each test uses a nominal threshold of 0.05, but the family of decisions needs its own error calculation.

2. Small-sample testing: a discrete outcome permits only a few p-values. Comparing them with 0.05 may produce an actual false-rejection probability below 0.05 rather than exactly equal to it.

3. Repeated analysis: analysts inspect results several times as observations arrive. A nominal threshold attached to each inspection does not, by itself, describe the error probability of the combined stopping-and-testing procedure.

Think of it this way

Think of a spending allowance and its accounting rule. The number printed in the plan is not enough; you must know whether it applies to one purchase or the entire project. Similarly, an individual-test significance threshold and a familywide error target refer to different scopes. Neither is a probability that a particular published conclusion is wrong.

Worked example

Suppose four independent tests concern four true null hypotheses. Assume each test is exactly calibrated, with probability 0.05 of rejection under its null. These are hypothetical.

For one test, the probability of no false rejection is 0.95. Independence makes the probability of no false rejections across all four equal to 0.95⁴ = 0.81450625. Therefore, the probability of at least one false rejection is 1 − 0.95⁴ = 0.18549375, approximately 18.55 percent.

The expected number of false rejections is a different quantity: 4 × 0.05 = 0.20. This expectation does not require independence when each marginal rejection probability is 0.05. It is not a twenty-percent probability of any false rejection.

If the desired familywise error limit is 0.05, a Bonferroni rule tests each hypothesis at 0.05/4 = 0.0125. Provided each individual test is valid at that level, the probability of any false rejection is at most 4 × 0.0125 = 0.05, regardless of dependence. Under the example's independence and exact-calibration assumptions, its actual value is 1 − 0.9875⁴ = 0.049070288..., approximately 4.91 percent.

For a dependence counterexample, imagine four copies of the same exactly calibrated test using identical data and the same decision rule. All four reject together or none does. At an individual level of 0.05, the probability of any rejection remains 0.05. Thus multiple testing can increase familywise error, but a strict increase is not universal for every dependence structure.

Illustrative case study

Fictional Alder Evaluation labels four test results “five-percent significance tests.” Its presentation initially says the study has a five-percent chance of any false positive. A reviewer asks whether the statement refers to individual tests or to the family.

Using the independent-test calculation above, the team separates the individual nominal level from the familywise probability. It defines the comparison family before the next evaluation and records the chosen adjustment beside the results. It also retains effect estimates and uncertainty rather than presenting only pass/fail labels.

This fictional review demonstrates why individual and familywide error guarantees must be distinguished.

Common misconceptions and mistakes

“Nominal alpha is the observed p-value.” Alpha is a threshold; the p-value is calculated from the observed data using a testing procedure.

“A 0.05 threshold means a rejected hypothesis is five percent likely to be true.” That reverses the conditioning. Type I error concerns rejection under a true null.

“Four tests always produce the independent-test formula.” They do not when rejection events are dependent.

“Actual error must equal nominal alpha.” For four fair coin tosses, an upper-tail test of fairness has minimum attainable nonrandomized p-value 1/16 = 0.0625. At threshold 0.05 it never rejects, so its actual Type I error is zero. This example concerns the stated upper-tail count test, not every possible four-toss procedure.

Business use

Write the error guarantee at the same scope as the intended claim. Specify whether a threshold applies to an individual comparison, a family or a sequential decision process. Record the number of comparisons, adjustment method and any assumptions used to calculate familywise error.

A nominal level is not an operational benefit threshold. A small p-value can accompany an improvement too small to matter, while an inconclusive result can still leave a practically important effect plausible. Report the outcome scale and uncertainty alongside statistical decisions, and avoid selecting the analysis only after seeing which version crosses the threshold.

Check your understanding

What is 0.20 in the four-test example?

Answer: The expected count of false rejections, not the chance of at least one.

Why is the Bonferroni threshold 0.0125?

Answer: The total 0.05 allowance is divided among four tests.

Why can identical repeated tests retain familywise probability 0.05?

Answer: Their rejection events coincide completely.

Related terms

Significance level
Type I error
P-value
Familywise error rate
Multiple comparisons
Bonferroni correction
statistical power
test calibration

Learn next

Compare the individual threshold, expected false-rejection count and familywise probability for two independent tests. Then examine how a planned sequential testing procedure differs from repeatedly inspecting an ordinary fixed-sample test.

Fixed sample

Sources

Frank Harrell, Glossary of Statistical Terms, “nominal significance level”: https://hbiostat.org/glossary/

NIST/SEMATECH e-Handbook of Statistical Methods, Bonferroni's method and the general probability inequality: https://www.itl.nist.gov/div898/handbook/prc/section4/prc473.htm

Previous: Nominal scale Next: NOMINATE (scaling method)