AI Glossary / Governance, ethics and safety / Intermediate
Data dredging
Definition
Data dredging is searching through many analyses or patterns and presenting selected favorable findings without adequately accounting for the search. It can make chance patterns appear more convincing than the full analytical process supports.
What it means
Exploring data is useful. The problem arises when exploration is presented as if it were a single, independently planned confirmation. Trying many outcomes, subgroups, transformations or stopping points creates more opportunities to find an apparently impressive result.
The term overlaps with p-hacking and selective reporting, although usage varies. It does not require that numbers be fabricated. Each calculation may be technically correct in isolation while the overall conclusion is misleading because unsuccessful searches are omitted.
AI can accelerate the search by generating analyses or proposing subgroup explanations. That makes a transparent record of the process more important. A fluent explanation for the winning pattern is not independent evidence that the pattern will persist.
How it works
An analyst examines many candidate relationships, sometimes adjusting choices after seeing the results. A favorable result is selected, while the rest of the search becomes invisible in the final report.
A statistical test's interpretation depends on the procedure used. A conventional p-value from one test does not automatically account for all the alternatives tried before that test was selected. Likewise, repeatedly evaluating models on the same test set can make the selected performance optimistic.
Good practice separates exploratory findings from confirmatory claims. Document the analyses, justify multiple-testing methods where relevant and evaluate promising patterns on suitable independent data. Planning key outcomes in advance can help, but it does not prevent all analytical mistakes. The goal is an honest account of the evidence rather than a ban on curiosity.
Three practical examples
- A marketing analyst reports one successful subgroup from dozens of comparisons while omitting the number of groups examined.
- An AI assistant tries many feature combinations and describes the best relationship as a confirmed business driver without independent validation.
- A product team repeatedly checks an experiment and stops at the first favorable conventional significance result without a suitable sequential-testing plan.
Think of it this way
Think of throwing many darts and drawing a target around the tightest cluster afterward. The cluster exists, but the target's placement was chosen because of the observed throws.
The analogy highlights selection after seeing results. Real exploration can discover useful patterns; the issue is whether later claims acknowledge how the pattern was found and receive appropriate follow-up testing.
Worked example
Why twenty chances differ from one
In an invented statistical setting, 20 independent tests each examine a true null hypothesis and each has a 5% false-positive probability under its assumptions. The probability that none produces a false positive is 0.95 raised to the power 20, approximately 0.358.
The probability of at least one false positive is therefore about 1 minus 0.358, or 64.2%. Reporting only a favorable test as though it were the sole planned analysis hides this repeated opportunity.
The calculation depends on independence and the stated test behavior; real analyses may be correlated or more complex. It does not mean that any selected finding has a 64.2% probability of being false. That is a different question. The example shows why the search procedure matters to inference.
Illustrative case study
Fictional company Fern Retail asks an AI analysis assistant to find evidence that a new recommendation layout improved repeat visits. After many segment definitions, the assistant identifies a favorable result among customers using a particular device on certain days.
The analyst records the full search and labels the subgroup finding exploratory. The company defines a follow-up question and collects new evidence under a suitable evaluation plan rather than building a confident causal story from the selected result.
The original exploration remains useful as a source of hypotheses. The organization avoids treating the assistant's ability to generate a plausible explanation as a substitute for testing that explanation.
Common misconceptions and mistakes
- Exploring many patterns is inherently wrong. Exploration is valuable when reported and interpreted appropriately.
- A small p-value proves the selected hypothesis. It does not directly give the probability that a hypothesis is true.
- More data automatically removes selective-analysis problems. A large dataset can still support a misleading search-and-select process.
- AI-generated analysis is objective by default. The requested objective and selection procedure can favor particular findings.
Business use
This concept matters in product experiments, customer segmentation, operational dashboards and model evaluation. Business pressure to find a positive result can shape the analysis even without deliberate misconduct.
Keep a record of major analytical choices and distinguish questions planned before inspection from those suggested by the data. When a selected finding will drive substantial action, seek appropriate independent evaluation and consider effect size, uncertainty and practical relevance. Avoid making a single significance threshold the sole basis for a decision. The final report should explain the search sufficiently for readers to assess the claim.
People also ask
Is data dredging the same as data mining?
No. Data mining broadly concerns finding useful patterns; data dredging criticizes an inadequately accounted-for search and selective claim.
Does it require dishonest intent?
No. Flexible analysis and incomplete reporting can mislead even when an analyst believes the conclusion.
Does a multiple-testing correction solve everything?
No. It addresses particular statistical issues under assumptions, not poor measurement, confounding or every form of selection.
Check your understanding
An assistant tries 100 segment definitions and reports only the most favorable one. Should the team treat it as a single prespecified test?
Answer: No. The search must be disclosed and accounted for. The selected pattern should be treated as exploratory unless suitable additional evidence supports a confirmatory claim.
Related terms
- p-hacking
- Hypothesis testing
- Multiple testing
- selective reporting
- exploratory analysis
- test-set leakage
Learn next
Read multiple testing and test-set leakage. Review one recent analysis and list every outcome, subgroup or model tried. Mark which choices were made before examining results and which arose afterward.
More in Governance, ethics and safety
- Cybersecurity - Cybersecurity is the management of risks to digital systems, services and information from unauthorized access, disruption, misuse or other harmful activity.
- Dark pattern - A dark pattern is a design practice that misleads, pressures or obstructs people in ways that undermine informed choice.
- Data control - Data control is the practical ability to set and enforce conditions on how data is collected, accessed, changed, used, shared and retained.
- Data fusion - Data fusion combines information from multiple sources to form a joint estimate, representation, or assessment.
- Data Generation Bias - Data generation bias is a distortion introduced when synthetic or redundant examples are added to a dataset in ways that misrepresent the intended...
- Data point - A data point is one unit of information considered in an analysis or machine-learning task.
Work in finance? Get the free CFO AI Prompt Pack: 7 prompts for finance leaders.
Sources
Wasserstein and Lazar, The ASA Statement on p-Values: Context, Process, and Purpose: https://doi.org/10.1080/00031305.2016.1154108. Official ASA announcement: https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf. Relevant indexed statement text and official announcement were reviewed; publisher full-text access was unavailable. The probability calculation and company scenario are original illustrations.
Previous: Data domain Next: Data drift