Is every duplicate row an error?

No. Repeated events or intentionally repeated observations can be valid; interpret the record's meaning and key.

Can automated cleaning fix all problems?

No. Tools can identify patterns and apply rules, but factual corrections often need reliable source evidence and judgment.

Is clean data guaranteed to produce a good model?

No. Task suitability, representation, labels, evaluation design, and modeling choices still matter.

AI Glossary / Data governance and quality / Intermediate

Dirty data

Definition

Dirty data is an informal name for data with quality problems that make it unreliable or unsuitable for an intended use. Problems can include incorrect values, missing information, unintended duplicates, inconsistent representations, stale facts, or broken relationships. The label is useful as a starting point, but a good assessment names the specific defect and its consequence. This entry uses the data-quality meaning, not the separate computing use of dirty to describe modified cached data awaiting persistence.

What it means

Data is not dirty merely because it is unusual or inconvenient. A missing value may be legitimate, repeated records may represent repeated events, and an extreme measurement may be genuine. The intended task and relevant business rules determine which conditions are defects. A useful investigation therefore asks what each field means, how it was produced, and what evidence supports a correction. Making a table look uniform is not the same as making its information accurate.

How it works

Profile the dataset to find patterns, gaps, duplicates, and unexpected values. Compare those findings with documented definitions and source evidence. Prioritize issues according to their effect on the intended use, then choose justified repairs, exclusions, or explicit uncertainty labels. Preserve the original where appropriate and record transformations so corrections can be reviewed. Investigate recurring causes in collection or integration rather than repeatedly cleaning the same symptom. Recheck important quality measures after changes and monitor new incoming data.

Three practical examples

Sales reporting: order amounts recorded in different currencies are combined as though they share one unit, producing a misleading total.
Customer records: two records for the same person appear separately because an integration duplicated an existing customer identifier.
AI training: labels use inconsistent definitions across annotator teams, so apparently similar examples teach conflicting target meanings.

Think of it this way

Think of a pantry inventory where some quantities use kilograms, some use boxes, and some have no date. The list may be complete enough to read but unsuitable for placing an accurate order. The analogy is limited because apparent irregularities in data may reflect valid distinctions, so cleaning requires evidence and purpose rather than automatically making every item look alike.

Worked example

An original toy sales file contains three rows: 100 USD, 100 EUR, and a second copy of the first 100 USD transaction. Adding the numeric amounts produces 300, but that number has no coherent currency meaning and includes a possible duplicate. The analyst first confirms that the repeated transaction ID is an accidental duplicate rather than a separate event. The analyst then removes that duplicate according to the reporting rule and applies an approved exchange-rate method, if a common-currency total is needed. The result cannot be calculated responsibly from the three numbers alone because the conversion date and rate are missing. Recognizing insufficient evidence is part of cleaning responsibly.

Illustrative case study

Fictional example: Rowan Logistics finds unusually long delivery times and proposes removing them from a training set. Investigation reveals that several are valid journeys involving remote destinations. The team corrects actual timestamp errors but preserves legitimate difficult cases, documenting their context. This invented case illustrates how indiscriminate cleaning can erase important parts of the population and make an AI system's evaluation look better while reducing its usefulness in practice.

Common misconceptions and mistakes

Do not delete every row with a missing value or remove all statistical outliers automatically. Do not merge similar names without evidence that they identify the same entity. A valid date format does not prove the date is correct, and standardizing an incorrect value can make the error less visible. Also, do not hide unresolved uncertainty behind a clean-data label. The database meaning of dirty pages or dirty reads is different and should not be confused with general data-quality defects.

Business use

Specific quality improvements can make reports and models more dependable, but cleaning effort should follow business impact. Identify which errors could change a decision, affect a customer, or distort evaluation. For AI projects, retain lineage between raw and prepared datasets and assess whether filtering changes the represented population. Assign owners for recurring problems and communicate known limitations to users. A useful quality report describes affected fields, evidence, correction rules, and remaining issues rather than merely claiming that the data has been cleaned.

Check your understanding

A dataset has a blank middle-name field. Should you fill it with a guessed name to improve completeness?

Answer: No. The person may have no middle name, or the field may be unknown. Preserve the appropriate meaning and uncertainty rather than inventing information to satisfy a metric.

Related terms

Data Quality
Validity
Accuracy
Missing data
Outlier
data cleansing
data profiling
deduplication

Learn next

compare completeness, validity, and accuracy using a small dataset. For each suspected problem, write the rule being violated and the evidence needed to correct it. Practice distinguishing safe standardization, such as an agreed code mapping, from a factual change that requires confirmation from an authoritative source.

Sources

Sources: IBM, What Is Dirty Data?, https://www.ibm.com/think/topics/dirty-data , relevant publisher excerpts consulted; UK Government, The Government Data Quality Framework, https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework , fitness-for-purpose and management sections consulted; IBM, Data Quality, https://www.ibm.com/think/topics/data-quality , relevant excerpts consulted. Sources ground the quality concepts, while the sales calculation and fictional case are original. No publisher statistics or claimed model-performance improvements are adopted. The cache-state meaning is explicitly outside this entry's scope.

Previous: Dirichlet process Next: Disambiguation