AI Glossary / Data science and analytics / Intermediate

Cherry-Picking

Definition

Cherry-picking is the selective use or presentation of evidence that favors a desired conclusion while leaving out relevant evidence that would weaken, complicate, or contradict it. In data analysis, it can involve choosing particular observations, time periods, metrics, subgroups, or analyses after seeing their results. The problem is the misleading picture created by the selection, even when every included number is individually correct.

What it means

A product demonstration may show impressive answers without revealing how often the system fails on comparable tasks. A report may highlight the one improving metric while omitting several deteriorating measures. Readers then mistake a selected view for a balanced account. Cherry-picking can be deliberate, but it can also arise through confirmation bias, enthusiasm, or undocumented analytical choices. The evidence of selection should be examined without automatically assuming a person's intent.

How it works

Selection can enter at several stages: which data are collected, which cases are retained, which comparisons are tried, and which results reach the report. The analyst may explore many possibilities and publish only those supporting the preferred story. To assess the problem, compare the stated question and planned analysis with the available evidence and reported results. Look for unexplained exclusions, changing outcome definitions, selective time windows, and missing unfavorable findings. Keep legitimate exploratory analysis visible as exploration rather than presenting it as a single preplanned test.

Three practical examples

• AI demonstrations: a team selects three excellent responses from a much larger set and describes them as representative performance without reporting how they were chosen.
• Sales reporting: a presentation begins its trend line at an unusually weak month, making a subsequent recovery appear stronger than a fuller period suggests.
• Model evaluation: a report highlights accuracy for one subgroup while omitting other relevant groups where the same model performs poorly.

Think of it this way

Think of describing a restaurant by showing only its best reviews while hiding the rest of the review book. The selected reviews may be authentic, yet the impression can still be distorted. The analogy captures selective presentation. Scientific and business analysis also involve technical choices about denominators, uncertainty, and comparison methods that a review collection does not fully illustrate.

Worked example

Suppose a fictional AI assistant is tested on five equally sized sets of 20 questions. It answers 18, 16, 12, 10, and 9 correctly. Reporting only the first set produces an apparent success rate of 18/20, or 90%. Across all five sets, it answers 65 of 100 correctly, or 65%.
The first figure is not arithmetically false. The misleading step is presenting the selected set as overall performance without explaining the selection. If the first set genuinely represents a separately defined use case, its 90% result may be useful, but the report should name that use case and avoid extending the claim to all tested work. The broader 65% figure also needs context: these invented sets are equally weighted here and may not represent the distribution of real-world requests.

Illustrative case study

In a fictional procurement exercise, a vendor presents several successful document summaries. The buyer asks for the full evaluation procedure, including document selection, failed outputs, and scoring criteria. Reviewers discover that the examples were chosen for presentation quality rather than drawn from a representative test. They keep the examples as illustrations and run a separately defined evaluation on relevant documents. This does not establish that the vendor's system is poor; it establishes that a polished demonstration cannot support the same claim as a transparent performance study.

Common misconceptions and mistakes

Not every selection is cherry-picking. A clearly defined scope, a justified exclusion, or a labeled illustrative example can be legitimate. The issue is whether the selection misrepresents the relevant evidence or hides how the conclusion was obtained. Including more charts does not automatically solve the problem if all come from the same favorable subset. Cherry-picking also differs from fabrication: genuine observations can be presented selectively. Conversely, a balanced report need not give unreliable evidence the same weight as stronger evidence.

Business use

Businesses can reduce this risk by agreeing on evaluation questions and important measures in advance, preserving the analysis record, and explaining changes. Report denominators, relevant failures, and limitations alongside successes. When AI helps summarize evidence, instruct reviewers to check whether contrary findings and scope restrictions survived the summary. A short executive report can still be balanced if it states what was assessed and links to the complete relevant results. Decision-makers need an honest basis for comparison, not an exhaustive dump of every intermediate calculation.

People also ask

Is cherry-picking always intentional?

No. Motivated attention and workflow habits can produce selective reporting without a conscious plan to deceive.

Can analysts explore data?

Yes. Exploration is valuable when it is identified as such and later claims reflect the selection process and appropriate validation.

Does reporting an unfavorable result prove a report is balanced?

No. The overall inclusion rules and missing evidence still need examination.

Check your understanding

A report gives a model's best subgroup score and clearly labels it as overall accuracy. The calculation is correct for that subgroup. Is the claim sound?

Answer: No. The label extends a selected result beyond its denominator and scope. The report must distinguish subgroup performance from the relevant overall evaluation.

Related terms

• Selection bias
• Confirmation bias
• p-hacking
• Benchmarks
• Transparency
• selective reporting
• publication bias
• benchmark
• evaluation protocol

Learn next

take a simple performance claim and identify its numerator, denominator, inclusion rules, and time window. Ask which reasonable alternative views could change the conclusion. Compare a labeled illustration with a representative evaluation to understand why both can be useful but support different kinds of claims.

More in Data science and analytics

• Box plot - A box plot summarizes the distribution of numerical data using a median, quartiles and whiskers.
• Categorical variable - A categorical variable records which category an observation belongs to, rather than a measured quantity.
• Causal analysis - Causal analysis investigates whether and how changing one factor would change an outcome.
• Cluster Sampling - Cluster sampling is a probability-sampling approach that selects groups of population units, such as branches, schools, or geographic areas.
• Co-occurrence network - A co-occurrence network represents items as nodes and connects items that appear together within a defined context.
• Collocation (remote sensing) - Collocation in remote sensing is the matching of observations from different sources so that they can be compared as measurements of sufficiently...

Sources

Sources: Cochrane's methodological handbook describes selective omission, incomplete reporting, and selection among multiple results: https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-07 . Its trial-bias chapter discusses selection among outcome measurements and analyses: https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-08 . Relevant methodological excerpts were consulted. The numerical model test and procurement case are original business illustrations; clinical findings are not generalized as business prevalence estimates.

Previous: Chernoff's distribution Next: Chi-square automatic interaction detection