Does the smallest average rank guarantee significant superiority?

No. A has the smallest rank in the example, but its comparison with B is not significant.

Can rank comparisons establish causality across products?

The statistical test alone does not supply an experimental design or eliminate differences in evaluation conditions.

AI Glossary / Statistics and probability / Advanced

Nemenyi test

Definition

The Nemenyi test is a rank-based multiple-comparison procedure. In the repeated-block setting commonly used to compare learning algorithms, it compares all pairs of methods using their average ranks across the same independent datasets or blocks. A sufficiently large average-rank difference is declared significant using a threshold that accounts for the family of pairwise comparisons.

What it means

The test asks which pairs show evidence of different rank performance within a specified comparison family. It does not measure the size of an accuracy improvement in percentage points. The familiar Friedman - Nemenyi workflow first uses an omnibus test to assess whether there is evidence of any difference, then examines pairs. This entry focuses on that repeated-block version.

How it works

Rank the methods separately within each block, using rank one for the best result and average ranks for ties. Average each method's ranks across blocks. For k methods and N blocks, the conventional critical difference is CD = q times the square root of k(k+1)/(6N). The q value must use the appropriate Nemenyi convention; some tables already divide the Studentized range quantile by the square root of two.

Three practical examples

A research team compares four classifiers on the same collection of benchmark datasets. It ranks results within each dataset so that a dataset with large numerical scores does not automatically dominate the rank comparison.
An evaluation team compares several forecasting methods across independent business units. It checks whether the units provide defensible independent blocks and whether each method was evaluated on comparable information.
A reviewer examines a critical-difference diagram. Two methods connected as not significantly different are not necessarily equivalent, and the connection does not establish that their deployment costs or errors are interchangeable.

Think of it this way

Imagine comparing runners across several races by their finishing positions. Average position summarizes repeated ordering, while a common comparison threshold limits how readily you declare differences among every possible pair. The rankings discard finishing-time margins: winning by a second and winning by a minute can receive the same first-place rank.

Worked example

Consider an original comparison of four methods, A, B, C, and D, across twenty independent datasets. Assume one prespecified performance measure, comparable evaluation conditions, and no tied scores. Ten datasets give ranks (1,2,3,4) to (A,B,C,D). Five give (2,1,3,4), and five give (1,3,2,4).

A's rank total is 10 times 1 plus 5 times 2 plus 5 times 1, which equals 25. Its average rank is 25/20 = 1.25. The corresponding totals for B, C, and D are 40, 55, and 80, giving average ranks 2, 2.75, and 4.

The uncorrected Friedman statistic is 12N/[k(k+1)] times the quantity obtained by subtracting k times the square of (k+1), divided by four from the sum of squared average ranks. Here this is 12 times [1.25 squared + 2 squared + 2.75 squared + 4 squared - 25] = 49.5. A chi-square reference with three degrees of freedom gives an approximate p-value of 1.02 times 10 to the power -10. This is an asymptotic calculation, not an exact finite-sample permutation probability; the chosen omnibus implementation should be reported.

For the conventional two-sided Nemenyi calculation at the 5% family level with four methods, use q = 2.569 from the cited table. Its square-root-of-two adjustment is already included. The critical difference is 2.569 times the square root of 20/120, approximately 1.049 average-rank units.

There are six pairwise comparisons. A versus B differs by 0.75, below the threshold. A versus C differs by 1.50, above it. A versus D differs by 2.75, also above it. B versus C differs by 0.75, below it. B versus D differs by 2.00, above it. C versus D differs by 1.25, above it. Thus four pairs cross the conventional threshold and two do not.

A versus B and B versus C are not significant, but A versus C is significant. Nonrejection is not an equality relation and does not have to be transitive. Drawing one broad group containing A, B, and C as though every pair were indistinguishable would misrepresent these results.

Suppose the same observed rank pattern were repeated across eighty genuinely independent datasets. Keeping the average ranks fixed, the critical difference would halve to approximately 0.524. The 0.75 differences would then exceed it. Simply copying the original twenty rows four times does not justify this calculation: duplication adds no independent evidence. The comparison distinguishes a real increase in information from an artificial increase in spreadsheet row count.

Illustrative case study

This fictional case concerns Birch Benchmarks, an imaginary AI evaluation team. Its first report chooses the method with the smallest average rank and describes it as better than every alternative. A reviewer applies the preceding comparisons and finds that A versus B does not cross the threshold.

The team revises its wording and reports the six comparisons alongside actual performance differences and operating costs. It also checks that repeated evaluation runs from one dataset have not been counted as separate independent datasets. This case illustrates reporting discipline.

Common misconceptions and mistakes

A significant omnibus result does not identify a particular winning pair. Conversely, a nonsignificant pairwise result does not prove equality or practical equivalence.

The number of blocks is not the number of individual predictions, training rows, or correlated cross-validation folds. Missing blocks and extensive ties require a procedure appropriate to those features. Do not combine critical values from one test convention with a formula from another.

Business use

Choose the comparison family before examining the results. Comparing every pair answers a different question from comparing several candidates only against a designated control. Report practical differences as well as ranks: a consistent but tiny improvement may have little operational value, while cost, latency, and failure severity may affect the deployment decision.

Check your understanding

Why is A versus C significant when the two adjacent comparisons are not?

Answer: Its rank difference is 1.50 rather than 0.75.

Why cannot duplicated dataset rows reduce the threshold legitimately?

Answer: They do not create independent evaluation blocks.

Related terms

Friedman test
average rank
critical difference
family-wise error rate
post-hoc comparison
Studentized range
multiple testing
benchmark evaluation

Learn next

Reconstruct the twenty rank rows and all six pairwise differences. Then compare the rank conclusions with hypothetical performance margins to see what the rank analysis omits.

Sources

Janez Demšar, Statistical Comparisons of Classifiers over Multiple Data Sets, section 3.2.2 and Table 5: https://www.jmlr.org/papers/volume7/demsar06a/demsar06a.pdf . Supports the Friedman–Nemenyi workflow and critical-value convention. The twenty-dataset construction, calculations, and Birch Benchmarks case are original illustrations.

Previous: Nelson - Aalen estimator Next: Neocognitron