AI Glossary / Data science and analytics / Beginner
Data cleaning
Definition
Data cleaning is the process of identifying and addressing errors, inconsistencies, and other quality problems in data so that the data are more suitable for a defined use. It can involve correcting formats, resolving duplicate records, investigating invalid values, and handling missing information. Cleaning should preserve meaning and evidence, rather than simply make a dataset look neat.
What it means
A value that looks unusual is not necessarily wrong, and a consistently formatted value is not necessarily correct. A negative transaction might be a legitimate refund, while a plausible positive amount might belong to the wrong account. Cleaning therefore requires rules based on the data's purpose and context. Some issues can be corrected confidently; others should be flagged, investigated, or left unresolved with an explicit limitation.
How it works
Profile the data to identify patterns and potential problems. Define quality rules and determine which changes have a reliable basis. Preserve appropriate source records and record transformations so the process can be reviewed and repeated. Apply corrections or other documented treatments, then validate both the individual changes and their effects on the dataset. Investigate recurring issues at their source. In machine learning, transformations that learn from data must respect training and evaluation boundaries to avoid leaking information into the model-development process.
Three practical examples
- Customer records: a team standardizes verified country codes while keeping ambiguous address matches for review rather than merging them automatically.
- Sales analysis: analysts distinguish genuine duplicate imports from separate transactions with coincidentally identical amounts.
- AI preparation: a project reviews inconsistent labels, documents missing-value treatments, and checks that cleaning has not removed an important group from the evaluation set.
Think of it this way
Think of preparing an archive for careful study. You repair damaged labels and organize records, but do not rewrite uncertain historical facts to make the collection more convenient. The analogy emphasizes preservation and justified correction. Digital cleaning also requires reproducible transformations, validation rules, and attention to how changes affect statistical analysis.
Worked example
Consider a fictional file with four rows: transaction T1 for 100 USD, a second identical T1 row introduced by a confirmed duplicate import, transaction T2 for −20 USD representing a refund, and transaction T3 with a blank amount. The confirmed duplicate can be removed from the analytical copy under the documented rule. The refund remains because its negative sign is meaningful. The blank amount is not automatically zero.
The known net amount from T1 and T2 is 100 − 20 = 80 USD, while the full three-transaction total remains unresolved because T3 is missing its amount. A cleaning routine that deletes negatives and fills blanks with zero would report 100 USD and conceal two different issues. The example shows that preserving meaning can be more important than eliminating every visually inconvenient value.
Illustrative case study
In a fictional AI scheduling project, a team removes all unusually long task durations as outliers before training. A subject expert discovers that many are genuine complex jobs, which the deployed system must handle. The team restores those records, investigates actual measurement errors separately, and evaluates performance across task types. It documents the correction rules and their effect on the data distribution. The result is a more defensible preparation process; the case does not promise that retaining every extreme value is always the correct choice.
Common misconceptions and mistakes
Cleaning is not synonymous with deleting rows or filling every gap. Missingness can itself reflect the collection process, and imputed values are estimates rather than recovered facts. Deduplication requires a clear definition of what makes two records the same entity or event. Standardizing formats does not establish accuracy. Also, cleaning cannot repair every limitation of a poorly designed dataset, such as an unobserved population. A technically valid file may still be unsuitable for the intended decision.
Business use
Businesses should prioritize cleaning issues according to their effect on users and decisions. Record what was changed, why, and how uncertain cases were handled. Measure whether the process improves the relevant quality dimensions rather than only reducing the number of warning messages. For AI projects, inspect how exclusions and corrections affect group coverage and evaluation difficulty. Keep the cleaning procedure consistent between development and intended use where appropriate, while fitting learned transformations only on the permitted training data. Addressing upstream causes can prevent the same errors from returning.
People also ask
Should every outlier be removed?
No. An outlier may be a genuine rare case, an important event, or an error; its meaning must be investigated.
Is imputation the same as finding the missing truth?
No. It supplies a value under a chosen method and assumptions, which should remain documented.
Can cleaning be fully automated?
Some well-defined checks and corrections can be automated, but ambiguous meaning and consequential changes often need knowledgeable review.
Check your understanding
Two sales rows have the same amount and date but different transaction identifiers. Is that enough evidence to delete one as a duplicate?
Answer: No. They may be separate genuine transactions. The team needs an appropriate duplicate definition and supporting evidence before removing a record.
Related terms
- Data Quality
- Validation
- Data wrangling
- Data leakage
- data profiling
- deduplication
- imputation
- provenance
Learn next
write one cleaning rule with its purpose, evidence, exceptions, and validation check. Apply it to a small test collection containing both genuine errors and unusual valid records. Compare results before and after cleaning, including which records and groups were affected.
More in Data science and analytics
- Cosine similarity - Cosine similarity measures the alignment of two nonzero numerical vectors by taking the cosine of the angle between them.
- Cross-industry standard process for data mining - The Cross-Industry Standard Process for Data Mining, usually abbreviated CRISP-DM, is a process model for organizing data-mining projects.
- Dark data - Dark data is information an organization has collected or retained but does not meaningfully use for analytics or decision-making.
- Data farming - Data farming is the purposeful generation of data from simulation models through designed experiments.
- Data mining - Data mining is the use of computational and statistical methods to discover patterns or build useful models from data.
- Data physicalization - Data physicalization represents data through the geometry or material properties of physical artifacts.
Sources
Sources: IBM describes common cleaning issues and the profiling stage: https://www.ibm.com/think/topics/data-cleaning . UK Government data-quality guidance emphasizes contextual rules and investigation of root causes: https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework-guidance . scikit-learn documents separation of training and test information during learned preprocessing: https://scikit-learn.org/1.0/common_pitfalls.html . Relevant primary documentation excerpts were consulted; the older scikit-learn reference supports the stable leakage principle, not current interface instructions. Transaction values and business cases are original examples.
Previous: Data-centric security Next: Data collaboratives