Can data inspire hypotheses?

Yes; that is an important part of research.

Can the same data always confirm them with ordinary tests?

No; selection must be addressed.

Does a multiplicity correction fix every problem?

No; design, measurement, modeling, and the actual search process still matter.

AI Glossary / Statistics and probability / Intermediate

Testing hypotheses suggested by the data

Definition

Testing hypotheses suggested by the data means evaluating claims that were formulated or selected after examining the same observations. This can be useful for discovery, but conventional tests may overstate evidence if they ignore the process that selected the claim. Reliable inference must account for selection or use appropriately independent evidence.

What it means

Unexpected patterns are valuable starting points for new questions. The difficulty arises when a pattern found by searching a dataset is then tested as though it had been the single preplanned question.
The data have already influenced what is being tested. That creates opportunities to select unusually favorable results, even when no genuine effect exists. Transparent exploration is different from presenting a post hoc hypothesis as if it had been specified before the results were known, a practice often called HARKing.

How it works

Record which questions and analysis choices were specified before examining relevant outcomes. Distinguish these from ideas developed during exploration. A date-stamped plan helps clarify the distinction, but only if the subsequent analysis accurately reports deviations.
Identify the selection process. Researchers may inspect many outcomes, subgroups, time windows, transformations, model specifications, or stopping points. The effective search can be much broader than the number of tests eventually shown in a report.
Understand why ordinary p-values can mislead after selection. A reference distribution for one fixed hypothesis does not automatically describe the most favorable result chosen from many possibilities. The selected result tends to look more extreme than a randomly chosen analysis.
Use independent confirmation when feasible. One dataset can generate a hypothesis and another can test a specified version of it. The confirmation data must not have guided the hypothesis, model tuning, or choice of reported result.
Sample splitting can reserve data for confirmation, but repeated inspection of the held-out portion weakens that separation. Cross-validation is useful for prediction assessment, yet does not automatically make every selected scientific hypothesis's ordinary p-value valid.
For defined families of tests, multiplicity methods can control quantities such as familywise error or false discovery rate under their conditions. The appropriate method depends on the objective and dependence structure. A correction for a short reported list may miss a much larger unreported search.
Selective-inference procedures can sometimes condition on or otherwise account for a known selection rule. They require an explicit mathematical treatment of that rule; merely labeling an analysis exploratory does not numerically recalibrate a p-value.
Report the search context, effect sizes, and uncertainty. An exploratory estimate can be exaggerated by selection, so replication should assess both whether a pattern persists and how large it is. Avoid treating a first selected estimate as a settled forecast of future benefit.
Keep discovery and confirmation connected. Exploration can improve theory and suggest better measurements, while subsequent evidence tests whether the pattern generalizes. The issue is not that new ideas are forbidden, but that their evidential status should be represented honestly.

Three practical examples

AI benchmarking: investigate a surprising subgroup advantage while reserving new cases for confirmation.
Product analytics: label a segment discovered after extensive dashboard exploration as exploratory.
Model research: account for searching many features or outcomes before highlighting a selected association.

Think of it this way

Think of drawing a target around the tightest cluster after seeing where arrows landed. The cluster may suggest something worth studying, but it cannot be judged as though the target had been fixed before the arrows were shot. A new round with a fixed target provides a different kind of evidence.

Worked example

Consider an original simplified search across 20 independent tests. Assume all 20 null hypotheses are true and each test has exactly a 5% false-rejection probability. The chance of no false rejection is 0.95 raised to the twentieth power, approximately 0.3585.
The chance of at least one false rejection is therefore 1 - 0.95 to the twentieth power, approximately 0.6415, or 64.15%. Selecting and reporting only that favorable result would hide the search that made it likely.
A Bonferroni rule for this prespecified family would test each hypothesis at 0.05/20 = 0.0025, controlling the probability of one or more false rejections at no more than 5%, even without independence, provided each individual test is valid.
The exact 64.15% calculation relies on independence and exact 5% individual rates. Real exploratory searches can involve dependent tests and adaptive choices, so the formula should not be applied mechanically. The correction also does not establish practical importance or repair biased data.

Illustrative case study

Fictional case: Maple AI compares a new assistant across many customer segments and finds one with a striking improvement. The team initially drafts a claim that the tool was designed specifically for that segment. A review documents that the segment was selected after analysis. The finding becomes an exploratory hypothesis, and a new evaluation defines the segment and outcome in advance. The follow-up provides a clearer basis for deciding whether targeted deployment is justified.

Common misconceptions and mistakes

Do not call every post hoc idea misconduct; transparent discovery is legitimate. Do not present a selected result as if no search occurred. A p-value below 0.05 does not automatically survive multiplicity or selection adjustment. Preregistration does not excuse unreported deviations. A held-out set repeatedly used to choose the final claim is no longer untouched confirmation data.

Business use

Use a clear discovery-and-confirmation workflow for AI experiments and business analytics. Keep an analysis record, report the extent of exploration, and budget independent evaluation for consequential claims. This helps prevent selected patterns from becoming overstated product promises or unstable operational rules.

Check your understanding

What is the chance of at least one false rejection in the independent toy example?

Answer: About 64.15%.

What is the Bonferroni per-test threshold?

Answer: 0.0025.

Why might independent confirmation help?

Answer: It evaluates a specified claim using evidence that did not select it.

Related terms

Harking
Replication (statistics)
Data dredging
Winner's curse
Exploratory analysis
multiple testing
selective inference
preregistration
replication

Learn next

Study familywise error and false discovery rate, then compare sample splitting with selective inference. Practice documenting which analysis choices came before and after inspecting outcomes.

False discovery rate

Sources

Kerr, HARKing: Hypothesizing After the Results are Known: https://pubmed.ncbi.nlm.nih.gov/15647155/ American Statistical Association, press release announcing the Statement on Statistical Significance and P-Values: https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf Wasserstein and Lazar, The ASA Statement on p-Values: https://doi.org/10.1080/00031305.2016.1154108 The 20-test probability example is original.

Previous: Test-time compute Next: Tetrachoric correlation