AI Glossary / Data science and analytics / Beginner

Data mining

Definition

Data mining is the use of computational and statistical methods to discover patterns or build useful models from data. Tasks can include finding associations, grouping similar records, predicting outcomes, and detecting unusual observations. In a narrower methodological sense, mining is the pattern-discovery stage within a broader knowledge-discovery process that also includes preparation, evaluation, and interpretation.

What it means

A database can answer a known query, such as how many orders arrived yesterday. Data mining explores or models relationships that may be less obvious, such as recurring combinations of products or characteristics associated with delayed orders. The output is a candidate pattern or model, not automatically reliable knowledge. Its value depends on the data, the analytical method, and whether the result survives appropriate evaluation.

How it works

Define the question and choose suitable data with an understood origin and purpose. Prepare the records, select a method aligned with the task, and discover patterns or fit a model. Evaluate whether the result is stable, relevant, and supported beyond the circumstances that produced it. Interpret findings with domain knowledge and decide how they can be used. Preserve the analysis choices and limitations. Exploration can generate useful hypotheses, but searching many possibilities also creates opportunities for accidental patterns and overfitting.

Three practical examples

• Retail analysis: association mining identifies products frequently purchased together, prompting a team to investigate a possible merchandising idea.
• Service operations: a predictive model estimates which cases may require additional handling, subject to evaluation against the intended workflow.
• AI quality review: analysts examine recurring characteristics of failed outputs to identify task categories that need better testing or data preparation.

Think of it this way

Think of examining a large collection of workshop records for recurring relationships that a quick glance would miss. Finding that two problems often appear together can guide investigation, but the records still need interpretation. The mining metaphor should not imply that unquestionable truths lie ready-made in the data; analytical choices help determine which patterns are found.

Worked example

Suppose a fictional shop has 100 baskets. Product A appears in 40, product B in 30, and both appear in 18. The joint support is 18/100 = 18%. Among baskets containing A, the proportion also containing B is 18/40 = 45%, often called the confidence of the rule A implies B. B's overall frequency is 30%, so the lift is 45%/30% = 1.5.
These quantities describe an association in this dataset. They do not mean buying A causes a customer to buy B, or that a promotion will increase revenue. A common event, shopper type, or existing promotion may explain the relationship. Before acting, the business should check the pattern's stability and test the proposed decision appropriately. The implication arrow in an association rule is not a causal arrow.

Illustrative case study

In a fictional subscription service, a mining project finds that customers contacting support are more likely to cancel. A manager proposes discouraging support contact to reduce cancellations. Analysts explain that the pattern may reflect underlying problems that cause both contact and cancellation. They use the finding to inspect support topics and service failures, then design an evaluation of a specific improvement. The scenario shows how a useful association can support investigation while failing to justify a harmful causal interpretation.

Common misconceptions and mistakes

Data mining is not the same as cryptocurrency mining, web scraping, or simply collecting more data. It can use machine learning, but it is not limited to deep learning. A statistically detectable pattern may have little practical importance, while an apparently strong result may depend on leakage or selection bias. Automated discovery does not remove the need for human judgment. Also, a pattern useful for describing a past dataset may not remain useful after the population, process, or environment changes.

Business use

Businesses should connect mining tasks to clear decisions and evaluate the resulting evidence before deployment. Record what counts as success, which costs or harms matter, and how outcomes will be monitored. For AI projects, distinguish training performance from performance on relevant unseen cases. Consider whether data were collected for a different purpose or omit important groups. Appropriate access and use remain necessary even if a mining technique is technically capable of processing the data. A transparent, modest finding can be more useful than an impressive but unsupported claim.

People also ask

Does data mining always predict the future?

No. It can describe structures and associations as well as support prediction.

Is every discovered pattern meaningful?

No. Some are chance findings, artifacts, or irrelevant to the intended question.

Can small datasets be mined?

Yes, but the available information and uncertainty limit what can be learned; a sophisticated algorithm does not create missing evidence.

Check your understanding

A rule has high confidence because its consequent is already very common. Is confidence alone enough to establish an interesting association?

Answer: No. Compare with the consequent's baseline frequency and examine support, stability, and the decision context. A high percentage can be unsurprising when the outcome is common everywhere.

Related terms

• Machine learning
• Clustering
• Predictive modeling
• Overfitting
• knowledge discovery in databases
• association rule
• data preparation
• CRISP-DM

Learn next

calculate support, confidence, and lift for a small transaction table, then explain what each denominator means. Compare a descriptive finding with a prediction task and a causal question using the same dataset. This helps prevent an analytical result from being stretched beyond what it supports.

More in Data science and analytics

• Dark data - Dark data is information an organization has collected or retained but does not meaningfully use for analytics or decision-making.
• Data cleaning - Data cleaning is the process of identifying and addressing errors, inconsistencies, and other quality problems in data so that the data are more suitable...
• Data farming - Data farming is the purposeful generation of data from simulation models through designed experiments.
• Data physicalization - Data physicalization represents data through the geometry or material properties of physical artifacts.
• Data stream mining - Data stream mining discovers patterns or updates analytical models from data arriving over time, often under limits on memory and processing time.
• Data structure - A data structure is a way of organizing and representing data so that particular operations can be performed on it.

Sources

Sources: Fayyad, Piatetsky-Shapiro, and Smyth's original knowledge-discovery article distinguishes mining from the broader process: https://fayyad.com/from-data-mining-to-knowledge-discovery-in-databases/ . IBM Research describes the field's method families: https://research.ibm.com/publications/top-10-algorithms-in-data-mining . IBM's overview supplies business-process context: https://www.ibm.com/think/topics/data-mining . Relevant primary author and publisher excerpts were consulted. The basket calculations and subscription example are original illustrations rather than reported research results.

Previous: Data minimization principle Next: Data model