Can W compare more than two judges?

Yes. That is a central use of the coefficient.

Does W = 1 mean the judgments are correct?

No. It means the rankings are concordant under the applicable formulation.

Can tied rankings be used?

Yes, with midranks and the appropriate tie correction; degenerate cases need special care.

AI Glossary / Statistics and probability / Intermediate

Kendall's coefficient of concordance ( W )

Definition

Kendall's coefficient of concordance, written W, measures agreement among several rankings of the same set of objects. In its standard form, it ranges from 0 to 1: larger values indicate stronger shared ordering, and 1 represents complete concordance. It is a rank-based measure, so it concerns relative order rather than agreement in the original numerical scores. A separate statistical test can assess evidence against an appropriate null model, but W itself is an effect-size summary rather than a p-value.

What it means

Imagine several judges ranking the same proposals. If they consistently place the same proposals near the top and bottom, the sums of ranks differ substantially across proposals. If their rankings cancel one another, those sums are more similar. W compares this observed spread with the spread possible under complete agreement. The judges need not literally be people: they can be methods, measurements or variables, provided the ranking design makes sense. A high value does not establish that the agreed ordering is correct, fair or useful; it establishes concordance within the specified set of rankings.

How it works

For m judges ranking n objects without ties, let Rᵢ be object i's rank sum and let S = Σ[Rᵢ − m(n + 1)/2]². Then W = 12S/[m²(n³ − n)]. Use a consistent direction for all ranks. With ties, assign midranks and use the corresponding tie-corrected denominator: m²(n³ − n) − mΣTⱼ, where Tⱼ sums t³ − t over tie groups of size t for judge j. Check for degenerate denominators and missing rankings. Inference may use suitable permutations or an approximate chi-square statistic m(n − 1)W, with n − 1 degrees of freedom under its assumptions.

Three practical examples

Proposal review: summarize whether independent reviewers order the same proposals similarly before discussing their reasons.
Method comparison: evaluate concordance among several ranking procedures applied to the same objects.
Research measurement: assess shared rank patterns across repeated assessments, while separately considering whether the original scores agree in magnitude. The design and dependence among rankings affect how inferential results should be interpreted.

Think of it this way

Think of each judge giving every object an ordinal position. When the judges pull in the same direction, favored objects collect consistently low rank totals and less favored ones collect high totals. When they pull in opposing directions, those totals flatten. W measures that collective alignment. It does not count a percentage of identical answers and does not tell you whether the group's shared preference reflects an external standard.

Worked example

Three judges rank four objects A, B, C and D. Their rank vectors are (1, 2, 3, 4), (1, 3, 2, 4) and (2, 1, 3, 4), using 1 as the highest preference. Rank sums are 4, 6, 8 and 12. Their expected central rank sum is 3(4 + 1)/2 = 7.5. Thus S = (4 − 7.5)² + (6 − 7.5)² + (8 − 7.5)² + (12 − 7.5)² = 35. W = 12 × 35/[3²(4³ − 4)] = 7/9, approximately 0.7778. There is substantial concordance in this constructed example, but 77.78% is not the proportion of pairwise judgments that match. The approximate test statistic would be 3 × 3 × 7/9 = 7. With such a small example, inference should not rely unquestioningly on a large-sample approximation; a suitable exact permutation calculation is possible.

Illustrative case study

Fictional case study: Ashbury Design asks three reviewers to rank prototype concepts independently. The overall W suggests considerable agreement, but discussion reveals that reviewers used different interpretations of usability. The team clarifies the evaluation criteria and records both rankings and written reasons for a later round. It does not treat the first high coefficient as proof that its rubric is valid. It also keeps the independent rankings separate from any consensus ranking produced after discussion, because agreement induced by group deliberation answers a different question.

Common misconceptions and mistakes

Common mistakes include confusing W with Kendall's tau for a pair of variables, interpreting W as a percentage of exact agreement, or applying the no-ties formula to tied ranks. W is nonnegative and does not represent negative association in the same way as a signed pairwise correlation. Low overall concordance can mask subgroups of judges with strong but opposing preferences. Rankings discard score distances, so W cannot establish absolute agreement. A significant result can accompany modest concordance when the design is large, and a high coefficient does not remove uncertainty in a small study.

Business use

Businesses can use W to summarize agreement in structured ranking exercises, model comparisons and assessment panels. Report the numbers of judges and objects, handling of ties, ranking direction and whether observations were independent. Inspect individual rankings and disagreements rather than using the aggregate coefficient as the only feedback. If the decision depends on agreement in numerical scores or categories, choose an appropriate additional measure. Decide acceptable agreement in relation to the task's consequences; there is no single coefficient threshold that certifies every review process.

Check your understanding

Why is W approximately 0.7778 not an accuracy score?

Answer: No external correct ranking was supplied. The coefficient summarizes how the judges' rank patterns align with one another, not whether their choices match a true or preferred outcome.

Related terms

Learn next

repeat a small calculation with tied ranks and verify the denominator correction. Compare W with pairwise rank correlations, then study how an appropriate permutation design differs from assuming every observed ranking is independently exchangeable.

Sources

Sources: D. C. Howell, statistical glossary, https://www.uvm.edu/~statdhtx/methods8/Glossary/Glossary.html. P. Legendre, Species Associations: The Kendall Coefficient of Concordance Revisited, https://pdodds.w3.uvm.edu/files/papers/others/2005/legendre2005a.pdf (coefficient, ties and inference). Calculation and fictional case are original illustrations.

Previous: Keepout Next: Kendall's tau