Is Friedman for independent samples?

No. It is designed for related conditions within blocks.

Does it require equal numbers of observations per condition?

The standard complete-block formulation requires every block to have each condition.

Does rejection identify the best condition?

No. It supplies an overall comparison; ranking summaries, uncertainty and suitable follow-up are still needed.

AI Glossary / Statistics and probability / Intermediate

Friedman test

Definition

The Friedman test is a rank-based test for comparing three or more related conditions in a complete-block or repeated-measures design. Each block contributes one observation under every condition, and comparisons are made through ranks within blocks. It investigates whether conditions show systematic differences in their within-block rankings under the null model. It is often used when a repeated-measures mean model is unsuitable, but it is not an assumption-free substitute for any comparison. Independent groups require a different design and usually a different test.

What it means

Blocking recognizes that observations from the same unit or matched set belong together. For example, participants may differ greatly in their baseline speed, yet each can be compared across several conditions. Ranking within participants removes the magnitude of those baseline differences from the test statistic. The resulting analysis concerns systematic condition differences in ranks. Interpreting it specifically as a median comparison requires additional distributional assumptions; differences in distributional shape can complicate a simple location-only interpretation. Ranking also discards information about the numerical size of differences.

How it works

Arrange data with blocks as rows and conditions as columns. Rank the observations within each block, using average ranks for ties, then sum ranks for each condition. Without ties, for n blocks and k conditions the statistic is Q = 12/[n k(k + 1)] times the sum of squared condition rank totals, minus 3n(k + 1). Apply an appropriate tie correction when needed. A common large-sample reference is chi-square with k minus one degrees of freedom, but small samples can require exact or permutation calibration under a justified within-block exchangeability null. Blocks should be independent, and the complete-block design must match the data.

Three practical examples

Interface research: compare task-completion measurements when each participant uses every interface under a suitable order-controlled design.
Benchmarking: compare methods on the same independent tasks, while checking that tasks and scores support the intended analysis.
Sensory research: compare ratings for several conditions from the same evaluators, accounting for pairing and the meaning of the ordinal scale.

Think of it this way

Think of asking each judge to rank the same contestants, then looking for a consistent pattern across judges. A judge's tendency to give generally higher scores matters less because the comparison happens within that judge. The method still depends on how judges and conditions were selected and organized.

Worked example

Suppose four hypothetical blocks produce condition values [1,2,3], [5,6,7], [9,10,11] and [13,14,15]. Within every block, ranks are [1,2,3], so condition rank totals are four, eight and 12. With n = four and k = three, Q = 12/(4 times 3 times 4) times (16 + 64 + 144) - 3 times 4 times 4 = eight. The chi-square reference with two degrees of freedom gives an approximate p-value exp(-8/2) = 0.01832. Under a null allowing all six condition-rank permutations independently within each block, there are 6 to the fourth power = 1,296 equally likely arrangements. Only six achieve this maximal statistic, giving an exact permutation probability 6/1,296 = 1/216, approximately 0.00463. The difference illustrates why the small-sample calibration must be stated; neither calculation identifies which pairwise differences should be declared significant.

Illustrative case study

Fictional case: Hazel Interface Lab has each participant complete a task under three layouts, with order handled through its experimental design. Analysts use a Friedman comparison to assess a systematic ranking pattern and report the chosen calibration. They then examine prespecified contrasts with suitable multiplicity handling instead of treating the overall result as proof that every layout differs from every other. This invented case illustrates a matched comparison and careful follow-up, without claiming a real interface performance gain.

Common misconceptions and mistakes

Do not apply the test to unrelated groups and ignore their lack of matching. Do not rank all observations globally; the defining step is ranking within blocks. An overall significant result does not identify every differing pair. Small-sample chi-square p-values can be inaccurate, and missing conditions create an incomplete-block problem requiring a justified approach. Repeated trials from one person are not automatically independent blocks. Rank differences do not quantify an original-unit effect, and ties must be handled consistently. Confounding by presentation order is a design problem that the test does not automatically remove.

Business use

Businesses can use the Friedman test when comparing related conditions and when ranks address the substantive question. Preserve the block identifiers, measurement scale and condition order. Report descriptive condition summaries and practical differences alongside the test, since statistical evidence does not show the size or operational value of an improvement. Plan follow-up comparisons and missing-data handling before inspecting results where feasible. In AI benchmarking, confirm that the selected tasks support the desired generalization; a convenient collection of dependent benchmarks does not automatically justify population-wide claims.

Check your understanding

Three methods are evaluated on the same tasks. Where are ranks formed for a Friedman test?

Answer: Within each task across the methods, provided tasks serve as suitable independent blocks. Ranking all scores together would describe a different calculation.

Related terms

Randomized blocks design
Rank test
Permutation test
Multiple comparisons
Repeated measures
randomized block design
tied ranks
Kruskal-Wallis test
paired comparison
Kendall's W

Learn next

Learn how block structure changes the comparison, then study tie correction and exact calibration. Compare Friedman with repeated-measures models and incomplete-block methods before choosing an analysis for missing or dependent observations.

Sources

SciPy, friedmanchisquare reference and calibration limitations: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.friedmanchisquare.html Howell, Statistical Methods for Psychology glossary, Friedman's rank test: https://www.uvm.edu/~statdhtx/methods8/Glossary/Glossary.html

Previous: Frequentist statistical inference Next: Friendly AI