Can the test compare two conditions?

With two conditions, its uncorrected statistic matches the usual asymptotic McNemar statistic; exact or corrected procedures can differ.

Does Q tell us which model is best?

No. It tests an overall equality hypothesis and needs descriptive results and appropriate follow-up comparisons.

Why keep individual task results?

Column totals alone omit the within-task agreement and disagreement that determine the test denominator.

AI Glossary / Statistics and probability / Intermediate

Cochran's Q test

Definition

Cochran's Q test compares success probabilities across three or more related conditions when each response is binary. The same subjects or matched blocks contribute a success/failure result under each condition. It extends the paired comparison idea behind McNemar's test. In this entry, Q refers to the related-binary-response test, not the differently constructed heterogeneity statistic also called Cochran's Q in meta-analysis.

What it means

Repeated evaluations are linked: an easy task may be passed by every model, while a difficult task defeats all of them. Treating the resulting success counts as independent samples ignores that structure. Cochran's Q uses the pattern of responses within blocks to assess whether condition-specific success rates are equal. It provides an overall test, so rejection indicates evidence of a difference somewhere rather than identifying which pairs differ or how important the differences are.

How it works

Arrange data in rows for independent subjects or blocks and columns for conditions, coding success as one and failure as zero. Let C_j be column totals, R_i row totals, T the grand total and k the number of conditions. Compute Q = (k - 1)[k sum(C_j squared) - T squared]/[kT - sum(R_i squared)]. Under appropriate null assumptions and sufficient information, compare Q with a chi-squared distribution on k - 1 degrees of freedom. Rows with all successes or all failures provide no within-row contrast. If every row is constant, the denominator is zero and the usual statistic is undefined. Small datasets may need a suitable exact or randomization approach.

Three practical examples

Model benchmarking: compare binary pass/fail results from several models on the same independent evaluation items.
Usability testing: compare whether the same participants complete a task under several interface conditions, with appropriate control of order effects.
Quality assessment: compare multiple inspection methods applied to the same independently sampled items.

Think of it this way

Think of asking several runners to attempt the same obstacles. Some obstacles are easy for everyone and others difficult for everyone. A fair comparison considers who clears each particular obstacle, rather than pretending each runner faced an unrelated collection. The shared obstacles are the blocks in the analysis.

Worked example

Consider eight hypothetical items tested under A, B and C. Their response triples are 111, 110, 110, 100, 100, 100, 010 and 000. Column totals are 6, 4 and 1; row totals are 3, 2, 2, 1, 1, 1, 1 and 0. Thus T = 11, the sum of squared column totals is 53 and the sum of squared row totals is 21. Q = 2(3 x 53 - 121)/(33 - 21) = 76/12 = 6.333. The chi-squared approximation with two degrees of freedom gives p approximately 0.0421. For comparison, under within-row exchangeability and fixed row totals, enumerating all 729 equally likely label configurations yields 36 at least as extreme, or p = 0.0494. This small example shows why the reference procedure must be stated explicitly.

Illustrative case study

Fictional case: Willow Evaluation compares three assistants using the same set of independently sampled tasks. Reviewers assign binary success labels with a documented rubric. The analyst preserves item-level triples rather than retaining only each assistant's total score. After an overall comparison, the team reports observed success rates and planned paired follow-up comparisons with suitable multiplicity handling. It also checks whether tasks share templates or sources that create additional dependence. This invented case illustrates evaluation design; it does not establish that any assistant is superior.

Common misconceptions and mistakes

Cochran's Q is not intended for unrelated groups or continuous scores. Matching across columns does not excuse dependence between rows, such as many variants of the same source question. A significant result does not identify all differing pairs, and a nonsignificant result does not establish equivalence. Exact permutation calculations require their own exchangeability assumptions; they are not automatically justified by equal marginal rates alone. Missing observations cannot simply be coded as failures. Order, learning and carryover can also explain differences in repeated-condition experiments unless design and interpretation address them.

Business use

Businesses comparing AI tools can use the test as one component of a paired evaluation when success is meaningfully binary. Preserve the evaluation unit, rubric and complete response pattern. Report effect sizes and practical tradeoffs alongside the overall test result, including performance on important task categories. If evaluation items form families, or outcomes need adjustment for additional variables, a model that explicitly handles that structure may be more appropriate. Do not select a winner solely because a global p-value crosses a conventional threshold.

Check your understanding

Why should the eight-item example not report 0.0421 as an exact probability?

Answer: That value uses a large-sample chi-squared approximation. The explicitly specified conditional enumeration gives 0.0494 instead. Both calculations depend on assumptions, and the small dataset makes the distinction relevant to interpretation.

Related terms

Matched samples
Randomized blocks design
Chi-squared distribution
Permutation test
Multiple comparisons
McNemar's test
repeated measures
binary outcome
randomized block design
exchangeability
evaluation rubric

Learn next

understand discordant pairs in McNemar's test, then generalize to several related binary conditions. Compare asymptotic and exact reference procedures and study the assumptions behind label permutations. For AI benchmarks, examine the independence of evaluation items before treating a large number of near-duplicate prompts as separate evidence.

Discordant pairs

Sources

NIST Dataplot, Cochran Test: https://itl.nist.gov/div898/software/dataplot/refman1/auxillar/cochran.htm statsmodels, cochrans_q documentation: https://www.statsmodels.org/stable/generated/statsmodels.stats.contingency_tables.cochrans_q.html statsmodels, Contingency tables documentation: https://www.statsmodels.org/stable/contingency_tables.html

Previous: Cochran's C test Next: Codd's 12 rules