AI Glossary / Statistics and probability / Intermediate

Data editing

Definition

Data editing is the process of checking data for errors, omissions, and inconsistencies and handling identified problems through documented procedures. In statistical work, it can include checking individual fields, relationships within records, and patterns across records or summaries. The purpose is to improve the reliability of the data while preserving legitimate variation and the meaning of the measurements.

What it means

An edit is often a rule or check, not simply a manual change to a value. A record can fail a check because it is wrong, because it is unusual but correct, or because the rule does not fit the circumstances.

Editing therefore separates detection from resolution. A flag identifies something to examine; evidence determines whether a correction, follow-up, imputation, or no change is appropriate. Statistical editing is closely related to data cleaning and validation, but terminology differs across organizations. Imputation specifically supplies replacement values for missing or unusable data and should remain identifiable within the wider process.

How it works

Define checks from the measurement concepts and expected relationships. These can include valid code lists, required fields, date ordering, arithmetic identities, duplicates, and comparisons with earlier observations. Distinguish hard inconsistencies from soft warnings about unusual values.

Run checks, identify the affected records, and prioritize review according to likely impact on the intended outputs. A reviewer may consult source records or recontact the respondent. Automated corrections are appropriate only when their conditions and consequences are sufficiently clear.

Preserve original values and record the rule, evidence, action, and responsible process. Recheck the corrected data and inspect aggregate effects. A correction that satisfies one rule can create another inconsistency. Monitor how frequently rules fire and how often flagged values are confirmed as erroneous so the process can improve.

Three practical examples

• A business survey checks whether reported component revenues sum to the total and follows up discrepancies before publishing estimates.
• An AI training-data workflow flags invalid category codes, duplicated records, and inconsistent annotations, retaining review decisions and original labels.
• A sensor-data process identifies impossible timestamps and suspicious jumps while distinguishing genuine operational events from instrument faults.

Think of it this way

Think of proofreading a financial table with both a calculator and the original receipts. The calculator can reveal that a sum does not reconcile, but it cannot tell which receipt or entered figure is wrong. Automatically changing the total may make the page look tidy while moving it further from the truth. Editing requires evidence as well as consistency.

Worked example

Consider an original three-record dataset containing two components and a reported total. Record A has 100,60,160; record B has 120,80,250; record C has 2,000,3,000,5,000. Assume the definitions require the two components to sum exactly to the total.

A passes because 100 + 60 = 160. B fails because 120 + 80 = 200, which differs from the reported 250 by 50. C passes the arithmetic check, although its size may trigger a separate soft warning.

The arithmetic failure alone does not reveal which field in B is wrong. Source verification establishes that the first component should be 170, while the second component and total are correct. The documented correction gives 170 + 80 = 250. Changing the total to 200 would have been an unjustified alternative.

If a reviewer also verifies C as a genuine large record, it should remain unchanged. The sum of reported totals remains 160 + 250 + 5,000 = 5,410 after the justified correction. This example shows that improving consistency need not change an aggregate, and that unusual size alone is not proof of error.

Illustrative case study

In this fictional case, Vista Learning prepares reviewed support conversations for evaluating an AI assistant. A rule flags conversations whose recorded resolution time precedes their creation time. Another flags unusually long conversations.

Investigation reveals a timezone conversion problem in some timestamps. The team fixes the conversion using source metadata and records the affected entries. Several long conversations are valid complex cases, so they remain in the evaluation dataset rather than being discarded as inconvenient outliers.

The team then checks that its edits have not disproportionately removed particular languages or task types. This invented case illustrates traceable correction and retention of valid difficulty; it makes no claim about a measured accuracy increase.

Common misconceptions and mistakes

Passing all edit rules does not prove that a record is true. Two wrong values can satisfy an arithmetic identity, and valid codes can still be assigned to the wrong person or event.

Do not force every unusual observation toward the average. Overediting can remove real variation, introduce bias, and distort relationships an AI system needs to learn. Excessive follow-up also consumes resources without necessarily improving important estimates.

An AI-generated correction is a proposal unless supported by an approved rule or evidence. Keep uncertain values and decisions distinguishable from verified corrections. Silent changes make later investigation, reproducibility, and comparison with source systems much harder.

Business use

Good editing supports trustworthy reports and model evaluation, but its success should be judged by data quality and downstream consequences rather than the number of values changed. Prioritize errors likely to affect decisions materially.

Define who can approve corrections, which transformations are automatic, and how original records are retained. Review changes in aggregate totals, distributions, and subgroup coverage. For machine-learning projects, fit data-dependent cleaning procedures only on appropriate training information to avoid leaking evaluation knowledge. Maintain consistent rules across versions while documenting justified updates when definitions or data sources change.

People also ask

Is editing the same as imputation?

No. Editing identifies and handles problems; imputation supplies values for missing or unusable items and may be one part of the workflow.

Should every outlier be removed?

No. Outliers can be valid and important. Investigate their origin and relevance before deciding how to treat them.

Can editing be automated?

Many checks and some corrections can be, but automation needs documented rules, monitoring, and a way to review exceptions.

Check your understanding

Check your understanding: Why was B's total not changed?

Answer: Verification showed the first component was wrong.

Why was C retained?

Answer: Its large values were confirmed, and size alone did not establish error.

Does the unchanged total of 5,410 mean editing achieved nothing?

Answer: No; the component data became more accurate even though the aggregate total stayed the same.

Related terms

• Data cleaning
• Outlier
• Measurement error
• Data validation
• edit rule
• logical consistency
• selective editing
• imputation
• audit trail
• data provenance

Learn next

Design separate hard and soft checks for a small dataset, then record how each failure would be investigated. Compare error counts with their impact on key estimates and examine whether editing changes subgroup representation.

More in Statistics and probability

• Cyclical effect - A cyclical effect is a recurring rise and fall in a time series.
• Data assimilation - Data assimilation is the process of combining observations with a model-based estimate to infer the state of a changing system.
• Data collection - Data collection is the planned process of obtaining and recording observations or measurements for a defined purpose.
• Data generating process - A data generating process, often abbreviated DGP, is the mechanism or probabilistic specification that produces observations.
• Data normalization - Data normalization is the transformation of numerical data into a scale or form suitable for analysis or machine learning.
• Data reduction - Data reduction is the creation of a smaller or simpler representation of data for a defined analytical purpose.

Sources

Sources: OECD Glossary of Statistical Terms, Data editing, https://www.oecd.org/content/dam/oecd/en/publications/reports/2008/09/oecd-glossary-of-statistical-terms_g1gh9ad7/9789264055087-en.pdf. Statistics Canada, Quality Guidelines, editing section, https://www150.statcan.gc.ca/n1/pub/12-539-x/4194542-eng.pdf. The reconciliation example and fictional AI evaluation case are original teaching material.

Previous: Data-driven model Next: Data entry