Is circular analysis the same as circular reasoning?

They are related in spirit, but circular analysis is a specific problem in empirical evidence and selection.

Is all data reuse forbidden?

No. Valid methods can account for dependence; the design and claim must match.

Can it happen accidentally?

Yes. A helpful-looking preprocessing or filtering step can introduce it.

AI Glossary / AI and machine learning / Intermediate

Circular analysis

Definition

Circular analysis occurs when information used to select a pattern also enters the analysis presented as independent evidence for that pattern, without a valid adjustment for the dependence. It can make noise look convincing and inflate apparent performance.

What it means

Suppose a team examines many customer groups and picks the one with the largest sales increase. Reporting that same increase as confirmation that this is an exceptional group overlooks the selection process. Some group had to look best, even if differences were mostly random.

The problem is sometimes called double dipping. In machine learning, related mistakes occur when evaluation data influences feature selection or model choices and is then treated as untouched evidence.

Reusing data is not automatically invalid. The issue is whether the statistical argument properly accounts for how the result was chosen. Independent evaluation is often the clearest practical remedy, but a specialist may use other valid designs.

How it works

First, an analyst makes a selection using observed data: a region, feature, subgroup, model or threshold looks promising. Next, the same information contributes to a reported result intended to confirm the selected pattern.

Because the selection favored unusually strong observations, the reported effect can be exaggerated. The apparent evidence partly reflects the decision rule that chose it.

A sound workflow separates exploration from confirmation. In predictive modeling, data-dependent preprocessing and feature selection belong inside the training procedure, including inside each relevant cross-validation split. A final untouched test set answers a different question from the data used to develop the method.

Three practical examples

A marketing team selects its best-performing audience from dozens of segments and mistakenly treats the selection-period result as confirmation.
An analyst chooses predictors using the whole dataset before evaluating a model with cross-validation.
A dashboard highlights the largest weekly improvement, then describes that chosen department as consistently superior without checking later weeks.

Think of it this way

Think of drawing many targets after throwing darts, placing the bullseye around the tightest cluster, and then claiming unusually accurate aim.

The analogy has limits: exploring data is valuable and does not imply dishonesty. The mistake is presenting an exploratory pattern as though the target had been fixed beforehand or independently confirmed.

Worked example

Separating discovery from confirmation

A fictional retailer tests ten promotional messages on a small exploratory dataset. One message has 18 purchases from 100 recipients, while the others have lower observed rates. The team selects the 18% message.

That 18% is a description of the selection sample, not an unbiased forecast for future customers. Random variation contributed to which message won. Reporting only the winner also hides how many alternatives were examined.

The retailer fixes the chosen message and compares it with the existing message in a new, appropriately designed test. Suppose the new observations are 150 purchases from 1,000 recipients for the chosen message and 140 from 1,000 for the existing message. The observed rates are 15% and 14%, a one-percentage-point difference.

These invented counts still require an uncertainty analysis before a strong conclusion. The key improvement is that the second test evaluates a decision already made, instead of using the winner's original result as its own confirmation.

Illustrative case study

A fictional support-analytics team proudly reports excellent classification performance. During review, an engineer discovers that the list of useful words was selected using labels from every record before the data was divided into evaluation folds.

The team rebuilds the workflow so each training fold selects its own vocabulary without seeing evaluation labels. It then reruns the evaluation and documents the corrected procedure. The earlier number is retained only as a superseded result with an explanation.

The case illustrates why checking the full analytical process matters. A familiar evaluation method can still produce misleading evidence when an earlier selection step has already seen the answers.

Common misconceptions and mistakes

Assuming cross-validation automatically prevents leakage: preprocessing and selection must also respect its boundaries.
Treating exploration as wrongdoing: exploration is useful when labeled honestly and followed by suitable confirmation.
Hiding the number of alternatives examined: the search process affects interpretation.
Believing a larger dataset fixes every circular design: dependence can remain even with many records.

Business use

This concept matters whenever a project reports unusually strong discoveries or model scores. Ask how the candidate was selected and what evidence remained independent of that selection.

Keep an analysis log recording choices, datasets and evaluation dates. Separate exploratory dashboards from confirmation reports. When repeated testing is unavoidable, involve someone who can design an appropriate statistical procedure rather than repeatedly reusing a supposedly final test set.

Check your understanding

A team chooses features using all outcome labels and then cross-validates only the final classifier. Is the resulting evaluation fully independent?

Answer: No. Evaluation labels influenced feature selection. The selection must be included within the training process for each split, or assessed with genuinely untouched data.

Related terms

Learn next

Read data leakage. Sketch your project's complete path from raw data to reported score and mark every step that uses outcome information.

Data leakage

Sources

Kriegeskorte and colleagues, Circular analysis in systems neuroscience: https://pmc.ncbi.nlm.nih.gov/articles/PMC2841687/. The PMC access challenge previously limited review to excerpts. On 14 October 2026, the institutional author manuscript was accessible at https://www.mrc-cbu.cam.ac.uk/personal/nikolaus.kriegeskorte/Kriegeskorte_Simmons_Bellgowan_Baker_Circular_analysis_in_systems_neuroscience_incl_supplement_author_version.pdf. The introduction, Example 1 and noncircular-analysis policy were consulted for selection dependence, preselection before splitting and independent-test remedies. Business examples and arithmetic are original teaching scenarios.

Previous: Circuit verification Next: Circular convolution