Is Newman-Keuls identical to Tukey?

No. Their largest-range thresholds coincide here, but smaller-range thresholds differ.

What does weak control mean?

The error guarantee applies when all population means are equal; strong control also covers configurations where only some null hypotheses hold.

Does the observed order prove the true order?

No. Sample ordering is subject to sampling variation.

AI Glossary / Statistics and probability / Advanced

Newman-Keuls test

Definition

The Newman-Keuls test, also called the Student-Newman-Keuls test, is a stepwise procedure for comparing group means. It orders the observed means and uses studentized-range thresholds that depend on how many ordered means a comparison spans. It controls familywise error under the complete null hypothesis, but does not generally provide strong familywise error control when only some population means are equal.

What it means

After comparing several versions of an AI interface, a researcher may want to identify which versions differ in average completion time. Newman-Keuls uses the positions of the ordered sample means to determine comparison thresholds.

This makes its results different from applying one fixed threshold to every pair. It also makes the error guarantee important: finding more significant pairs is not, by itself, evidence that a procedure provides more trustworthy conclusions.

How it works

In the classical balanced design, assume independent observations, normally distributed errors and a common population variance. Obtain the pooled within-group mean square, MSE, and its degrees of freedom from the analysis of variance.

Order the means. For positions i and j, with j larger, define r = j − i + 1. Calculate q = (larger mean − smaller mean)/sqrt(MSE/n). Start with the largest range and proceed to smaller eligible ranges. A nonsignificant range blocks comparisons contained within it. Use the appropriate studentized-range critical value for r and the error degrees of freedom.

Three practical examples

1. Interface trials: five independent participant groups test five designs. The analyst preserves participant-level independence and checks whether one common error variance is defensible before applying the classical procedure.

2. Model benchmarks: five models run on the same tasks. Those outcomes are paired by task, so treating them as five independent groups would misrepresent the design. The label “multiple comparison test” does not make every method suitable.

3. Training programs: an old report uses Newman-Keuls to label groups. A reviewer records the original method and its limited error guarantee before comparing it with an analysis designed for strong familywise control.

Think of it this way

Imagine comparing the endpoints of windows placed over an ordered row of means. A wide window uses a larger critical range than a narrow window. The procedure checks wider windows first and restricts what happens inside windows that fail. The windows are defined by the observed ordering, not by a pre-existing ranking of the treatments.

Worked example

Consider an original hypothetical study with five independent groups, eight observations per group, and ordered means A = 10, B = 13, C = 16, D = 17 and E = 23. Suppose every sample variance is eight. The pooled MSE is therefore eight, with 5 × (8 − 1) = 35 error degrees of freedom. The q denominator is sqrt(8/8) = 1.

At the five-percent level, use Howell's rounded table values for 35 degrees of freedom: 2.875 for r = 2, 3.465 for r = 3, 3.815 for r = 4 and 4.07 for r = 5. These are reference thresholds; the study data are invented. Because the denominator is one, the thresholds also equal critical differences in outcome units.

A versus E spans five means: its difference 13 exceeds 4.07. The four-mean comparisons A versus D and B versus E have differences seven and ten, both exceeding 3.815. The three-mean comparisons A versus C, B versus D and C versus E have differences six, four and seven, all exceeding 3.465.

The adjacent comparisons have differences three, three, one and six. Against 2.875, A-B, B-C and D-E are significant; C-D is not. All containing ranges needed to reach these comparisons passed. Thus nine of the ten pairs are declared significant by this calculation.

Using the same rounded five-mean threshold, Tukey's balanced all-pairs procedure would require every difference to exceed 4.07. It would declare six pairs significant. The extra Newman-Keuls findings are A-B, B-C and B-D. This contrast illustrates different procedures, not a reason to choose whichever produces more findings.

Illustrative case study

Fictional Alder Interfaces reviews the hypothetical results above. Its initial presentation says that only C and D are “the same.” The statistician changes this to “the procedure did not detect a difference between C and D,” because nonrejection does not establish equality.

The team also replaces a claim that the analysis guarantees a five-percent familywise error rate under every configuration of means. It explains the distinction between the complete null and situations containing both true and false pairwise null hypotheses.

Before a new experiment, Alder specifies its comparison family and error-control requirement. It reports estimated time differences alongside statistical decisions. This is a fictional review scenario, not a record of an actual interface experiment.

Common misconceptions and mistakes

“Adjacent means have r = 1.” They span two ordered means, so r = 2.

“The standard error is sqrt(2MSE/n).” That is the standard error of a difference used with a t statistic. The q formulation here uses sqrt(MSE/n); its reference distribution accounts for that scaling.

“A nonsignificant comparison proves equivalence.” Equivalence requires its own practical margin and analysis.

“A significant overall ANOVA repairs every multiple-testing issue.” An omnibus result does not turn Newman-Keuls into a procedure with general strong familywise control.

Business use

Recognize this test when interpreting legacy analyses or reproducing a specified method. For a new project, match the comparison procedure to the study design and the required error guarantee before examining results.

In the example, a three-unit difference might be statistically detected yet too small to justify redesign costs. Conversely, a larger uncertain difference may merit further measurement. Keep effect size, uncertainty and operational value visible rather than reducing the report to significance letters. Unequal sample sizes or variances require an explicitly justified method, not silent reuse of the balanced formula.

Check your understanding

Why does B-D use r = 3?

Answer: The interval contains B, C and D.

Why is C-D not significant?

Answer: Its difference one falls below 2.875.

Why do three extra findings need cautious interpretation?

Answer: They arise from a procedure with a different familywise error guarantee.

Related terms

Multiple comparisons
Familywise error rate
Analysis of variance
Pooled variance
Studentized range
Tukey HSD
equivalence testing

Learn next

Reproduce all ten differences and their spans. Then study weak versus strong familywise control and distinguish testing all pairs from testing only preplanned contrasts.

Sources

David C. Howell, Multiple Comparisons Among Treatment Means, Newman-Keuls discussion and rounded critical values: https://www.uvm.edu/~statdhtx/methods8/Supplements/OldChapter12.pdf

CRAN mutoss documentation, Student-Newman-Keuls procedure and weak error control: https://search.r-project.org/CRAN/refmans/mutoss/html/snk.html

Previous: Newline character Next: Newton-Raphson