Must many tests actually be run?

No. Data-responsive choices can create the problem even when one final test is computed.

Is exploratory analysis bad?

No. It is valuable, but its findings should not be presented as if all hypotheses were fixed beforehand.

Does preregistration guarantee correctness?

No. Design, measurement, assumptions, compliance and transparent reporting still matter.

AI Glossary / Statistics and probability / Intermediate

Forking paths problem

Definition

The forking paths problem arises when analytical choices depend on the observed data, but the final statistical result is interpreted as though the entire analysis had been fixed in advance. Choices may concern outcomes, exclusions, transformations, subgroups, covariates or stopping rules. Each choice can open another plausible route to a result. The problem can occur without deliberate manipulation and even when only one final analysis is actually carried out. Conventional significance calculations can become misleading if they ignore how a different dataset would have led to a different analytical path.

What it means

A p-value is calibrated for a specified testing procedure under its assumptions. If the procedure includes choosing a favorable question or model after seeing the data, that selection is part of what must be considered. Reporting only the final test can hide the larger process that produced it. The issue is therefore broader than openly running many tests and selecting the smallest p-value. Researchers may make seemingly reasonable, data-responsive decisions without realizing that these decisions change the relevant reference distribution or strength of the evidence.

How it works

Map the decisions that connect the research question to the reported analysis. Ask which were set before inspecting outcomes and which could have changed in response to the observed pattern. Prespecification can clarify confirmatory hypotheses and decision rules, while transparent reporting identifies subsequent exploratory work. Independent validation or appropriately designed sample splitting can test discoveries on data not used to select them. Multiplicity or selective-inference methods may help when the selection procedure is defined. Sensitivity analyses can show how conclusions vary across defensible choices, but displaying many specifications does not automatically produce a calibrated confirmatory test.

Three practical examples

Experiment analysis: choose a subgroup after noticing a strong effect, then mistakenly report its ordinary p-value as if that subgroup had been the original target.
Data cleaning: decide which observations to exclude after seeing which exclusions strengthen a preferred finding.
Model evaluation: try several outcome definitions and report the most favorable result without describing the selection process.

Think of it this way

Think of walking through a garden where every junction offers another path. The route you eventually describe is only one possibility. If the signs at each junction were chosen after seeing where the paths led, the final destination cannot be evaluated as though the route had been fixed from the start.

Worked example

Suppose a hypothetical project allows two outcome definitions, two exclusion rules and two subgroup choices. That creates 2 times 2 times 2 = eight possible analysis combinations. For a simplified illustration, imagine eight independent valid tests of true null hypotheses, each using a 5% threshold, and a rule that declares a discovery if any test is significant. The probability of at least one false positive is 1 - 0.95 to the eighth power, approximately 0.33658, or 33.66%. Real specification paths commonly share data and are dependent, so this number is not a general false-positive rate for eight analytical choices. For a prespecified family of eight valid tests, Bonferroni's per-test threshold would be 0.05/8 = 0.00625, controlling the familywise rate without requiring independence. Even that calculation is not a complete remedy when the family or selection process has been left undefined.

Illustrative case study

Fictional case: Linden Product Research initially finds no clear overall effect in a trial, then notices a promising result among a subset of new users. Its analysts label the subgroup finding exploratory, explain how the subgroup was selected and design a later study with that comparison specified in advance. They preserve the original overall result rather than replacing it with the more attractive story. This invented case illustrates how useful discovery can continue while keeping the strength of confirmatory claims aligned with the research process.

Common misconceptions and mistakes

Do not assume the problem requires fraud or conscious p-hacking. A single reported test can reflect data-dependent selection. Do not treat preregistration as a ban on learning from unexpected results; changes can be valuable when documented and interpreted appropriately. Do not assume a larger sample automatically repairs selection bias. Trying many reasonable analyses is not inherently wrong, but their role must be clear. A robustness plot cannot prove that every relevant choice was covered, and a low p-value does not reveal how many opportunities the workflow created to obtain it.

Business use

Businesses can reduce misleading discoveries by documenting experimental questions, metrics, exclusions and analysis rules before reviewing outcomes. Separate exploratory dashboards from decisions requiring confirmatory evidence. Keep records of changes and test promising findings on genuinely new information where feasible. Encourage analysts to report uncertainty and unfavorable results, so performance incentives do not reward only attractive patterns. When automated or AI-assisted analysis proposes many paths, preserve the selection history. The goal is a reviewable chain from question to evidence, allowing stakeholders to distinguish discovery from a finding that has survived an independent test.

Check your understanding

A subgroup is chosen because it looks promising, and its ordinary p-value is then reported without qualification. What is missing?

Answer: The selection process. The reported test alone does not account for why that subgroup was chosen or which alternatives could have been selected.

Related terms

Multiple comparisons
p-hacking
Replication (statistics)
Researcher degrees of freedom
preregistration
exploratory analysis
confirmatory analysis
selective inference
sample splitting
replication

Learn next

Learn how testing procedures are calibrated, then examine preregistration and independent validation. Study multiplicity and multiverse analysis as complementary tools, while recognizing that transparent exploration and confirmatory inference serve different purposes.

Sources

Gelman and Loken, The garden of forking paths: https://stat.columbia.edu/~gelman/research/unpublished/forking.pdf Center for Open Science, Preregistration: https://www.cos.io/initiatives/prereg Center for Open Science, Answering Your Preregistration Questions: https://www.cos.io/blog/answering-your-preregistration-questions

Previous: Fork - join model Next: Formal concept analysis