Is post hoc analysis the same as p-hacking?

No. Transparent, appropriately interpreted follow-up analysis is legitimate. Hidden result-driven selection and misleading inference are the concern.

Must a post hoc comparison always follow ANOVA?

No. That is a common statistical usage, but post hoc investigation occurs in many types of analysis.

Can follow-up comparisons be planned in advance?

Yes. Their timing after an initial analysis does not prevent the procedure, comparison family and decision rules from being specified beforehand.

AI Glossary / Data science and analytics / Intermediate

Post hoc analysis

Definition

Post hoc analysis is analysis undertaken after results have been examined, often to explore an unexpected pattern or investigate a broader finding in more detail. In statistics, the term also commonly refers to follow-up comparisons, such as comparing group means after an analysis of variance.

What it means

The phrase means “after the event,” but its practical meaning depends on context. A researcher may discover a surprising subgroup pattern and investigate it, or may use a recognized multiple-comparison procedure to examine which groups differ.

These uses should not be collapsed into one judgment. A follow-up procedure can be specified before the study, even though it is performed after an overall analysis. An unplanned exploratory investigation is different because the data helped choose the question.

Post hoc work can be valuable. It can reveal mistakes, generate hypotheses and clarify a result. The important requirement is to explain how the question and analysis were selected, use an appropriate method and avoid presenting a result discovered through searching as though it were an untouched confirmatory test.

How it works

Start by identifying the original question and what prompted the follow-up. Record whether the comparison was planned, selected after inspection or one of many possibilities examined. Then define the quantity being estimated and the relevant uncertainty.

When comparing several groups, an overall test may indicate that the means are not all equal without identifying every differing pair. Procedures such as Tukey's method address families of pairwise comparisons under their assumptions. Other questions may require different contrast or multiplicity methods.

Exploratory subgroup analysis adds further interpretation challenges. A result within one subgroup and a different significance label in another do not by themselves prove that the groups respond differently; the difference between effects needs direct assessment.

Report the full relevant context, including unsuccessful comparisons and departures from the original plan. A useful discovery can be evaluated in new data or a clearly separated validation sample before supporting a stronger claim.

Three practical examples

A researcher follows a multi-group comparison with appropriately adjusted pairwise comparisons to identify which mean differences merit attention.
A product team investigates an unexpected drop in completion on one device type and labels the finding exploratory.
An AI evaluation team examines error categories after an overall benchmark result to formulate questions for a new, prespecified evaluation.

Think of it this way

Think of reviewing a match recording after the final score to understand what happened. The review can identify useful patterns, but explanations chosen after watching the outcome need further scrutiny.

The analogy does not make every follow-up speculative. Some statistical procedures explicitly account for a planned family of comparisons. The key distinction is between a valid analysis of specified questions and a selected explanation whose discovery process has been hidden.

Worked example

Suppose a fictional study compares four training formats, A, B, C and D. There are six pairwise comparisons: A - B, A - C, A - D, B - C, B - D and C - D.

For a simple teaching illustration, a Bonferroni procedure controlling a family-wise level of 0.05 across those six tests compares each valid p-value with 0.05/6, approximately 0.0083. A pairwise p-value of 0.02 does not pass that threshold, even though it is below 0.05 when viewed in isolation.

This does not prove that the pair's true difference is zero. It shows that the claimed decision rule must reflect the family of comparisons. Bonferroni is not the only method, and the appropriate choice depends on the design and question.

If the analyst actually searched many additional outcomes and subgroups, accounting only for these six comparisons might still omit part of the selection process. The analysis history matters alongside the arithmetic.

Illustrative case study

Illustrative fictional case: Meadow Apps tests three onboarding flows. After reviewing the results, the team notices that one flow appears more successful among returning users and asks an AI assistant to write a confident explanation.

The reviewer separates the observed subgroup estimate from the proposed explanation. The team checks data quality, examines the interaction question directly and reports that the subgroup was selected after inspection. It then plans a new evaluation focused on that hypothesis.

The team keeps the exploratory result in its report rather than hiding it or treating it as definitive. This fictional case illustrates transparent learning from data; it does not claim that the selected flow has a confirmed advantage.

Common misconceptions and mistakes

Do not assume post hoc analysis is automatically misconduct. Do not treat every follow-up comparison as unplanned merely because it occurs after an overall test.

Avoid claiming that “significant in one group, not significant in another” establishes a difference between groups. Do not use an overall significant test as permission to ignore the assumptions or multiplicity of later tests. Finally, a plausible narrative generated after seeing a pattern is not itself evidence for the proposed causal explanation.

Business use

Use post hoc analysis to investigate surprises and improve future questions, while maintaining an analysis log. Reports should state which questions were planned and which emerged during examination of the results.

For business decisions, consider effect sizes, uncertainty, data quality and the cost of acting on an unstable pattern. An AI tool can help organize error examples and suggest follow-up questions, but its fluency must not erase the distinction between observation, interpretation and confirmation. Where a selected result would drive a substantial change, seek evidence that evaluates the claim beyond the same search that produced it.

Check your understanding

An unexpected subgroup pattern was discovered after trying many splits. Should a report describe it as the study's original primary hypothesis?

Answer: No. State how it was discovered, report the relevant search context and distinguish exploratory evidence from an independently evaluated confirmatory claim.

Related terms

Multiple comparisons
p-hacking
Exploratory analysis
ANOVA
Tukey's method
Bonferroni correction
interaction
preregistration

Learn next

Read multiple testing and interaction effects. Take one study report and identify the original question, follow-up comparisons and any result-selected subgroups. Write an accurate description of the evidence and a separate plan for evaluating its most useful new hypothesis.

Sources

NIST, multiple-comparison procedures and follow-up questions: https://www.itl.nist.gov/div898/handbook/prc/section4/prc47.htm

Center for Open Science, distinguishing planned and exploratory analyses: https://www.cos.io/initiatives/prereg

Center for Open Science, exploratory analyses within Registered Reports: https://www.cos.io/initiatives/registered-reports

ASA, selective reporting and interpretation of p-values: https://doi.org/10.1080/00031305.2016.1154108

The training-format calculation and onboarding case are fictional examples.

Previous: Post-editing Next: Post-hoc explanation