Are CMH statistics limited to two-by-two tables?

No. The broader family includes correlation, row-mean-score and general-association statistics.

Does stratification guarantee that confounding is gone?

No. Unmeasured factors and differences within strata can remain, and inappropriate adjustment can create problems.

Is the test statistic the same as the common odds ratio?

No. One assesses evidence against a null hypothesis; the other summarizes an association's magnitude on an odds scale.

AI Glossary / Statistics and probability / Advanced

Cochran - Mantel - Haenszel statistics

Definition

Cochran - Mantel - Haenszel statistics are a family of methods for assessing association between categorical variables while accounting for specified strata. A familiar application combines several two-by-two tables to test association between a binary exposure and outcome after stratification. Broader versions address scored ordinal relationships or general association in larger tables. These methods adjust the comparison for the included strata; they do not automatically establish causation or remove every source of bias.

What it means

A pooled table can mix groups with different baseline outcome rates or exposure patterns. Stratification compares exposure and outcome within defined groups before combining evidence. For example, an AI-assisted workflow might be evaluated separately within task-difficulty bands. The resulting question differs from simply asking whether the raw overall success percentages differ. Strata need a substantive justification, and the categories and score choices should be defined before selecting whichever analysis gives the most attractive result.

How it works

For a series of two-by-two tables, denote the exposed-success count in stratum i by a_i. Conditional on that table's margins under independence, its expected value is the exposed total times the success total divided by n_i. Its variance is the product of both row totals and both column totals divided by n_i squared times (n_i - 1). Without continuity correction, the common CMH statistic is the squared sum of observed-minus-expected a_i values divided by the sum of those variances. A chi-squared reference with one degree of freedom is commonly used. Exact alternatives and continuity corrections change the calculation. Larger-table CMH statistics use different scoring or covariance structures.

Three practical examples

Workflow evaluation: compare binary success between assisted and unassisted cases while stratifying by a prespecified task category.
Survey analysis: examine an ordinal association within demographic strata using clearly justified category scores.
Measurement research: assess a categorical relationship across study sites, while checking whether the direction and magnitude differ substantially between sites.

Think of it this way

Think of comparing two delivery services within each neighborhood before combining the comparisons. A service handling more difficult neighborhoods could look worse in a crude citywide average. Stratification preserves those local comparisons, although it cannot fix important differences that were never measured or included.

Worked example

In a hypothetical example, stratum 1 has exposed counts 12 successes and 8 failures, and unexposed counts 6 successes and 14 failures. Stratum 2 has exposed counts 9 and 11, and unexposed counts 3 and 17. Both totals are 40. Expected exposed successes under within-stratum independence are 20 x 18/40 = 9 and 20 x 12/40 = 6. Observed minus expected totals 6. The two conditional variances sum to approximately 4.6923, so the uncorrected statistic is 6 squared/4.6923 = 7.6721. Its chi-squared-one-degree-of-freedom p-value is approximately 0.00561. Separately, the Mantel - Haenszel odds-ratio estimate is [(12 x 14)/40 + (9 x 17)/40]/[(8 x 6)/40 + (11 x 3)/40] = 3.963. That estimate is an odds ratio, not a risk ratio.

Illustrative case study

Fictional case: Alder Operations reviews whether an assistant is associated with successful case completion. Because teams handle different case types, analysts prepare separate tables for prespecified complexity bands. They examine the band-specific results and compare a stratified summary with the crude association. One band shows a different pattern, prompting closer review rather than an unconditional claim that a single summary describes every case. The team also acknowledges that staff selected when to use the assistant, leaving possible confounding within bands. This invented case illustrates analysis and interpretation, not evidence of a causal product benefit.

Common misconceptions and mistakes

A significant CMH test is not proof that every stratum has the same association. The pooled common-odds-ratio interpretation needs attention to heterogeneity; a separate homogeneity test is not interchangeable with the association test. Opposing effects can cancel in a summary, and a nonsignificant homogeneity test does not prove identical effects. Conditioning on an inappropriate variable can introduce bias. Sparse or uninformative strata may limit inference. Ordinary stratified-table formulas also do not automatically handle arbitrary repeated observations from the same person. Match the sampling assumptions to the data structure.

Business use

Businesses can use CMH methods for transparent categorical comparisons when a modest set of meaningful strata captures important differences in the cases being compared. Retain the individual tables, effect direction and uncertainty alongside the summary. State whether the analysis uses nominal categories or numerical scores for ordered categories. If many continuous confounders, interactions or complex dependencies matter, a suitable regression or design-based analysis may be more practical. A concise adjusted result should not obscure the population and comparison it actually represents.

Check your understanding

Why should 3.963 in the example not be described as nearly four times the success probability?

Answer: It estimates an odds ratio. Odds equal probability divided by one minus probability, so an odds ratio and a risk ratio are generally different, especially when outcomes are common.

Related terms

Stratification
Contingency table
Odds ratio
Confounding
Simpson's paradox
Conditional independence
Mantel–Haenszel estimator
effect modification
Breslow–Day test
conditional logistic regression

Learn next

calculate within-stratum odds ratios and compare them with crude associations. Study conditional table distributions, effect modification and the distinction between odds and risks. Then examine how nominal and ordinal CMH statistics encode different questions, and when a regression model provides a clearer representation of the design.

Regression model

Sources

SAS, Cochran-Mantel-Haenszel Statistics: https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/procstat_freq_details86.htm statsmodels, StratifiedTable documentation: https://www.statsmodels.org/stable/generated/statsmodels.stats.contingency_tables.StratifiedTable.html statsmodels, Contingency tables: https://www.statsmodels.org/stable/contingency_tables.html

Previous: Coarse-to-fine warping Next: Cochran's C test