Does a significant result mean every study has an effect?

No. It rejects the joint claim that all component nulls hold.

Does the method average effect sizes?

No. It combines p-values and does not estimate a common effect magnitude.

Can it be used unchanged with dependent tests?

Not generally; the standard chi-square reference relies on the stated independence conditions.

AI Glossary / Statistics and probability / Intermediate

Fisher's method

Definition

Fisher's method combines several p-values into one test statistic by summing their negative logarithms. For m independent p-values that are uniformly distributed under their null hypotheses, the statistic -2 times the sum of log(p_i) follows a chi-square distribution with 2m degrees of freedom under the combined null. The method tests whether all component null hypotheses hold together. It combines statistical evidence, not effect sizes, and it does not establish that every component study shows an effect.

What it means

Several studies or tests may address related questions while using different outcomes or measurement scales. Combining their p-values can summarize evidence against the joint null when the method's assumptions fit. Small p-values contribute strongly because their negative logarithms are large. The combined result does not indicate the magnitude, direction or practical importance of an effect. When the input tests are two-sided, effects in opposite directions can both contribute evidence, so a small combined p-value does not necessarily imply a consistent directional finding.

How it works

Define the component hypotheses and ensure each p-value is valid for its stated null. Check independence and the continuous-uniform reference assumption before applying the standard chi-square calibration. Calculate S = -2 sum log(p_i), using natural logarithms, then find the upper-tail probability under chi-square with 2m degrees of freedom. Dependence requires an appropriate adjustment or alternative calibration. Discrete tests can produce non-uniform null p-values, affecting the standard result. Preserve the complete set of eligible inputs and distinguish prespecified evidence synthesis from selection of only favorable or published results.

Three practical examples

Research synthesis: combine independent tests of related null hypotheses when effect sizes are not directly comparable.
Experimental review: summarize evidence across independent replications while separately reporting directions and magnitudes.
Statistical teaching: demonstrate how logarithmic contributions transform a collection of p-values into one joint-null test.

Think of it this way

Think of translating each study's incompatibility with its null into a common evidence score and adding the scores. The total can challenge the claim that every null holds, but it cannot say that every study contributes equally or that all effects point in the same direction.

Worked example

Suppose two hypothetical independent tests produce p-values 0.02 and 0.04, with valid continuous-uniform p-values under their respective nulls. Fisher's statistic is -2[log(0.02) + log(0.04)] = -2 log(0.0008), approximately 14.2618. With four degrees of freedom, the chi-square upper-tail formula is exp(-S/2)(1 + S/2), yielding approximately 0.00650472. This is evidence against the joint null under the assumptions. It is not the arithmetic mean 0.03, the product 0.0008 or the probability that both nulls are false. If the two p-values instead came from heavily overlapping data, the independence calibration would need reconsideration. Duplicating one result in the input list does not create an independent replication, even though the arithmetic would produce another contribution.

Illustrative case study

Fictional case: Redwood Evaluation combines two independently run tests of an operational hypothesis. Before synthesis, analysts confirm that the datasets do not overlap and that the component analyses were selected according to a documented plan. They report each estimate, direction and uncertainty alongside the combined result. A disagreement in effect direction is discussed explicitly rather than hidden by the summary p-value. This invented case illustrates evidence aggregation without claiming that a joint-null rejection proves a common effect or identifies its magnitude.

Common misconceptions and mistakes

Do not assume any collection of numbers labeled p-values can be inserted into the standard formula. Independence and calibration matter. Shared controls, overlapping samples or correlated outcomes can invalidate the usual chi-square reference. Rounded p-values reported as zero cannot be logged directly; obtain suitable numerical values or bounds rather than replacing them arbitrarily. Combining only significant results creates selection bias. The method does not correct a family of separate discoveries in the same way as a multiple-testing adjustment, nor does it estimate a pooled treatment effect.

Business use

Businesses synthesizing several evaluations should decide whether the desired conclusion concerns a joint null, a common effect size or consistency across settings. Fisher's method addresses the first question under its assumptions. Preserve evidence about study quality, outcome definitions and data overlap, and report practical effects separately. A highly significant combined result may be driven mainly by one study. Further deployment decisions may therefore depend on replication and heterogeneity analysis rather than the combined p-value alone. Evidence synthesis should retain inconvenient results as well as favorable ones.

Check your understanding

Why are there four degrees of freedom in the worked example?

Answer: Under the standard null assumptions, each of the two transformed p-values contributes two chi-square degrees of freedom, giving 2 x 2 = four.

Related terms

Meta-analysis
combined p-value
joint null hypothesis
chi-square distribution
Stouffer's method
publication bias
multiple testing

Learn next

compare p-value combination with effect-size meta-analysis, then examine how dependence, discreteness and selective reporting alter the validity of an evidence-synthesis procedure.

Sources

Sources: SciPy, combine_pvalues, Fisher statistic and assumptions: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.combine_pvalues.html ; SAS, p-Value Adjustments, Fisher combination method and joint-null interpretation: https://support.sas.com/documentation/cdl/en/statug/66859/HTML/default/statug_multtest_details11.htm .

Previous: Fisher's Least Significant Difference Test (LSD) Next: Fitness approximation