Does every systematic review need a meta-analysis?

No; numerical synthesis may be inappropriate or impossible.

Is the largest study automatically the most trustworthy?

No; statistical precision and risk of bias are distinct.

Can two studies be combined?

Sometimes, but estimating heterogeneity and generalizing from very few studies can be difficult.

AI Glossary / Statistics and probability / Intermediate

Meta-analysis

Definition

Meta-analysis is the statistical synthesis of results from multiple studies addressing a sufficiently related question. It combines effect estimates, usually with weights reflecting their uncertainty, and may examine variation between studies. Meta-analysis is often part of a systematic review, but it is not synonymous with the review itself: searching, selecting, appraising, and interpreting evidence remain essential even when no numerical pooling is appropriate.

What it means

Individual studies provide estimates that differ because of sampling variation, design, populations, measurements, and potentially real differences in effects. Meta-analysis organizes that evidence quantitatively rather than simply counting how many studies report statistical significance.
A pooled estimate can improve precision, but its meaning depends on what was combined and how. Pooling several biased or incomparable studies does not automatically create reliable evidence. The mathematical result is only as interpretable as the question, input data, and assumptions supporting it.

How it works

Define the review question, eligibility criteria, outcomes, and analysis plan. Identify relevant evidence systematically and assess study limitations. Decide whether studies are sufficiently comparable to combine, including their intervention or exposure, comparator, population, outcome definition, and timing.
Express study results on a compatible effect scale. This may be a mean difference, a standardized effect, or a logarithm of a ratio, depending on the question. Keep each estimate paired with its uncertainty and account for dependence when studies share participants or contribute multiple related outcomes.
In a common-effect inverse-variance analysis, weight each estimate by one divided by its sampling variance. The weighted average estimates a common effect under the corresponding model. A random-effects analysis allows underlying study effects to vary and commonly includes a between-study variance component in the weighting.
Examine heterogeneity rather than hiding it behind a single number. Differences may reflect genuine variation, bias, incompatible measures, or extraction errors. A confidence interval for an average effect is different from a prediction interval describing a possible effect in a new comparable setting.
Investigate sensitivity to analysis decisions and risks from missing or selectively reported evidence. Report the individual study results alongside the synthesis so readers can assess consistency and the influence of particular studies.

Three practical examples

AI evaluation research: analysts synthesize comparable studies of a workflow tool while separating different outcome definitions and accounting for shared benchmarks.
Learning research: a review compares effects of related training interventions across independent studies, with attention to population and delivery differences.
Evidence literacy: a reader uses a forest plot to examine the study estimates, uncertainty, and pooled result rather than relying only on a headline conclusion.

Think of it this way

Think of assembling several measurements of a question rather than holding a vote among study headlines. More precise estimates may contribute more to the calculation, but precision is not the same as trustworthiness. A carefully weighted average can still be misleading if the underlying measurements concern different things or share the same bias.

Worked example

Three invented independent studies estimate the same mean difference in task-score units. Their estimates are 2, 4, and 3, with standard errors 1, 2, and 1. Assume a common-effect model is appropriate and these standard errors are suitable for the calculation.
The sampling variances are 1, 4, and 1. Inverse-variance weights are therefore 1, 0.25, and 1, totaling 2.25. The pooled estimate is (1 × 2 + 0.25 × 4 + 1 × 3)/2.25 = 6/2.25 = 2.6667 units, approximately.
Its standard error under this simple model is the square root of 1/2.25, or 0.6667 units. A normal-approximation 95% interval is 2.6667 plus or minus 1.96 × 0.6667, approximately 1.36 to 3.9733 units. This interval reflects the stated model and does not incorporate an allowance for study bias.
For illustration only, suppose a between-study variance of one were supplied. Weights would become 1/2, 1/5, and 1/2, and the weighted estimate would be 2.75 units. The value one is assumed here, not estimated from these three studies. The calculation shows how allowing heterogeneity changes weights; it is not a completed random-effects analysis.

Illustrative case study

Fictional case: Ash Evidence reviews studies of an AI writing assistant. Some report speed, others report accuracy, and several evaluate the same participants in overlapping publications. The team does not pool all reported numbers into one performance score.
It groups compatible outcomes, checks overlapping samples, assesses study limitations, and presents a quantitative synthesis only where the comparison is defensible. This fictional scenario illustrates evidence organization and avoids claiming that a pooled number establishes universal effectiveness.

Common misconceptions and mistakes

Do not count significant studies as if significance were an effect size. A small study and a large study can estimate similar effects while differing in statistical significance. Do not weight solely by sample size when the intended method requires actual estimate variances.
A random-effects model does not repair biased studies or make fundamentally different questions comparable. A nonsignificant heterogeneity test does not prove that effects are identical, particularly with few studies.
Do not interpret the confidence interval around a pooled average as the range of effects expected in every setting. Do not treat multiple reports from one underlying study as independent confirmations, or assume that published evidence is a complete record of all relevant results.

Business use

Meta-analysis can inform research priorities, procurement evidence reviews, and comparisons of interventions, provided the synthesized estimand matches the decision. Retain a clear audit trail from source reports through extracted values to calculations.
For AI-supported reviews, verify extracted effects, units, denominators, and uncertainty against the original reports. Automation can help organize evidence, but an incorrectly extracted sign or duplicated study can materially change a synthesis. Report uncertainty, heterogeneity, and limitations in the evidence base alongside any pooled estimate.

Check your understanding

under inverse-variance weighting, which gets more weight: standard error one or two?

Answer: one, with four times the weight in the common-effect calculation.

does pooling eliminate shared bias?

Answer: no.

is a prediction interval the same as an interval for the average effect?

Answer: no; they describe different uncertainty targets.

Related terms

Effect size
Inverse-variance weighting
Random Effects Model
Prediction interval
Systematic review
Heterogeneity
Forest plot
Publication bias

Learn next

Next, study forest plots and between-study heterogeneity. Then examine sensitivity analysis, dependent effect estimates, and the distinction between an average effect and applicability to a new setting.

Sensitivity analysis

Sources

Cochrane Handbook, Chapter 10, Analysing data and undertaking meta-analyses: https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-10 NIST/SEMATECH, glossary: https://www.itl.nist.gov/div898/handbook/glossary.htm

Previous: Message queuing service Next: Meta-learning