Does p-hacking require dishonest intent?

No. Undisclosed result-driven choices can distort evidence even when the analyst believes they are improving the analysis.

Is running many models always wrong?

No. Multiple analyses can be legitimate for exploration, prediction or sensitivity assessment. Their selection and interpretation need to be transparent and appropriate.

Does a p-value above 0.05 prove there is no effect?

No. It does not establish equivalence or absence of a practically meaningful effect. The design, estimate and uncertainty still matter.

AI Glossary / Data science and analytics / Intermediate

p-hacking

Definition

P-hacking is the practice of selecting or changing data analyses in response to their results so that a desired finding appears statistically significant, while failing to account for or disclose the selection process. It can make evidence look stronger than the complete analysis history supports.

What it means

An analyst may try many outcome measures, subgroups, exclusion rules or model specifications, then report only the version with a small p-value. The problem is not simply that several analyses were run. It is that the final result is interpreted as if it came from the claimed testing procedure without the hidden search.

This can happen through deliberate manipulation or through ordinary-looking decisions made after inspecting results. An analyst may sincerely believe that the most favorable model is also the most reasonable one.

Exploration is valuable when presented as exploration. The distinction is whether the reporting and statistical interpretation accurately reflect how the claim was selected. AI makes it easier to generate many plausible analyses, increasing the importance of keeping their purpose and history visible.

How it works

A p-value is calculated under a statistical model and a testing procedure. If the procedure changes according to intermediate results, the behavior of the final selected p-value can differ from that of a single planned test.

For example, trying several subgroups gives more opportunities to encounter an apparently strong result. Repeatedly checking a conventional fixed-sample test and stopping when it crosses a threshold can also change the false-positive behavior unless the procedure accounts for that monitoring.

Safeguards include specifying primary questions and decision rules in advance, documenting deviations, reporting relevant analyses and using methods appropriate to multiple comparisons or sequential monitoring. Independent confirmation can help distinguish a promising discovery from a stable result.

No single safeguard proves that a conclusion is correct. Good reporting also considers effect size, uncertainty, study design, measurement quality and whether the proposed explanation fits the wider evidence.

Three practical examples

A product team tests many engagement metrics and promotes only the one that crosses its preferred significance threshold, without revealing the others.
An analyst repeatedly changes an outlier-exclusion rule after inspecting results and reports the most favorable version as the original plan.
An AI-assisted study searches many customer subgroups for a positive result, then describes the selected subgroup as if its test had been the sole prespecified question.

Think of it this way

Think of throwing many darts and drawing a target around the most convenient landing point afterwards. The highlighted dart looks impressive, but the target was selected using the outcome.

The analogy does not mean that every unexpected pattern is worthless. Such a pattern can motivate a new question. The issue is claiming the same evidential status as a target and evaluation rule fixed before the relevant throws.

Worked example

Suppose an invented exercise runs 20 independent tests. Every null hypothesis is true, and each test has a 5% probability of a false positive under its valid assumptions.

The probability that none produces a false positive is 0.95 raised to the twentieth power, approximately 0.3585. Therefore, the probability of at least one false positive is 1 − 0.3585 = 0.6415, or about 64%.

If only the smallest p-value is reported as a lone planned test, the reader misses those additional opportunities. The calculation does not mean that a particular observed finding has a 64% probability of being false. It describes repeated-test behavior under the stated all-null, independent-test setup.

Real analyses often involve correlated tests, so their exact combined probability differs. The teaching point is that an undisclosed search changes the interpretation; the number 64% is not a universal p-hacking rate.

Illustrative case study

Illustrative fictional case: Maple Learning evaluates a new onboarding email. Its overall completion result is inconclusive. An analyst asks an AI assistant to search combinations of device, weekday and experience level until it finds a favorable comparison.

The team initially drafts a launch announcement around that subgroup. A reviewer asks how it was chosen. The analysts disclose the search, label the subgroup result exploratory and plan a separate evaluation with a specified question and analysis.

They also review whether the estimated improvement would matter operationally. This fictional case does not assert that the email works or fails. It shows how a selected result can become a useful hypothesis without being promoted as already confirmed evidence.

Common misconceptions and mistakes

Do not call every exploratory analysis p-hacking. Exploration becomes misleading when selection is hidden or the inference ignores it. Do not assume a small p-value measures the probability that the hypothesis is true or the practical size of an effect.

Avoid treating preregistration as immunity from poor design or measurement. Do not select only favorable robustness checks. A clearly reported departure from a plan is more informative than pretending the final analysis was the plan all along.

Business use

Before an experiment, record the primary outcome, observation window, exclusions and decision procedure. Keep an analysis log, including AI-generated alternatives and the reasons for changes.

When many comparisons are necessary, choose an appropriate inferential strategy and explain what it controls. Separate discovery from confirmation in reports and presentation titles. Business decisions should weigh estimated effects, uncertainty, costs and operational context rather than being reduced to whether one chosen p-value falls on a preferred side of a threshold.

Check your understanding

An AI assistant finds one favorable result among 100 unplanned subgroup tests. Can it be reported as an ordinary single confirmatory test without mentioning the search?

Answer: No. The selection process is part of the evidence. Report it transparently and use an appropriate analysis or independent confirmation before making a confirmatory claim.

Related terms

P-value
False positive
Replication (statistics)
multiple testing
selective reporting
preregistration
exploratory analysis
replication
sequential testing

Learn next

Read multiple testing and confidence interval. Compare a planned primary analysis with an exploratory subgroup search, and write how each should be described. Practice recording analysis choices before seeing the relevant results and documenting later changes.

Confidence interval

Sources

American Statistical Association, statement on p-values and statistical interpretation: https://doi.org/10.1080/00031305.2016.1154108

Simmons, Nelson and Simonsohn, False-Positive Psychology: https://journals.sagepub.com/doi/10.1177/0956797611417632

ASA President's Task Force, statistical significance and replicability: https://magazine.amstat.org/blog/2021/08/01/task-force-statement-p-value/

The independent-test calculation and email case are original fictional illustrations.

Previous: P-frame Next: P-rep