Should every missing value be imputed?

No. Repair, exclusion, explicit unknown values, model-supported missingness, or human review may be more suitable, depending on the field and task.

Can I calculate replacements before splitting the data?

Learned replacements should normally be fitted using training data only, including within each cross-validation fold.

Does a missingness indicator recover the original value?

No. It records absence and may help prediction; it does not reveal the unobserved measurement.

AI Glossary / Mathematics and optimization / Intermediate

Missing data

Definition

Missing data are values that should be available for an observation but are absent, unknown, corrupted, or otherwise unusable. In machine learning, missing data require an explicit treatment because their absence can affect what the training set represents, how a model processes inputs, and whether its predictions remain reliable.

What it means

A blank cell does not explain why a value is absent. A customer may skip a question, a sensor may fail, or two systems may use different identifiers when combining records. A field can also be inapplicable rather than unknown. Those situations should not automatically receive the same numerical replacement.

Missingness may itself contain useful information, but it can also reveal a broken collection process. The practical task is to understand what is missing, when it becomes missing, and what information will be available when predictions are made. A complete-looking table can still contain misleading replacements.

How it works

Start by defining valid missing markers. These might include a database null, a numerical NaN, or a documented sentinel. Zero should count as missing only when the data definition explicitly says so; zero sales can be a genuine observation.

Inspect missingness by column, time period, source, and relevant groups. Decide whether to repair collection, exclude an unusable variable, remove selected records, use a model that supports missing inputs, or impute values. Imputation estimates replacements using a stated rule, such as a training-column median or a model based on other variables.

Split training and evaluation data before fitting an imputation rule. During cross-validation, fit that rule separately inside each training fold. Apply the learned rule to its held-out fold. A missingness indicator can preserve whether a replacement occurred, but its usefulness and stability require evaluation.

Three practical examples

1. Equipment monitoring: a temperature sensor stops reporting during certain operating conditions. A team investigates the dropout pattern before treating observed readings as representative of every condition.

2. Customer analytics: an optional preference field is blank for many customers. The model may use an explicit unknown category while keeping it distinct from a customer selecting “no preference.”

3. Document processing: an extraction system cannot read some invoice totals. Those records may need review or a separate processing route; inventing a plausible total would misrepresent the source document.

Think of it this way

Think of a survey form with unanswered boxes. Filling every empty box with the most common answer makes the form look finished, but it does not establish what those respondents would have said.

The analogy captures the difference between a usable substitute and an observed fact. It does not imply that imputation is always wrong: a carefully designed method can support useful analysis while documenting its assumptions and uncertainty.

Worked example

Suppose a training feature contains 10, 20, missing, and 30. Mean imputation learns (10+20+30)/3=20, producing 10, 20, 20, and 30. A separate indicator can be 0, 0, 1, and 0, recording the replacement.

Now the evaluation feature contains 100 and missing. Apply the training mean: the second value becomes 20. If the observed evaluation value were included when learning the mean, the result would instead be (10+20+30+100)/4=40. That allows evaluation information to influence preprocessing and compromises the separation between training and evaluation.

Replacement also changes the visible spread. The population variance of the three observed training values is 200/3, approximately 66.67. After filling the blank with 20, the four-value population variance is 200/4=50. This calculation does not establish the true unknown value or its variance; it shows how a convenient replacement can shrink measured variation.

Illustrative case study

Fictional case study: a greenhouse operator trains a model using sensor records. Rows with missing humidity readings are initially deleted. A maintenance review then finds that dropouts occur disproportionately during wet operating conditions, so the remaining rows underrepresent those conditions.

The team investigates the hardware, retains dropout indicators, and evaluates candidate handling methods on separately collected records. It also creates a route for unusually incomplete inputs. No performance gain is assumed: the lesson is that correcting the table and understanding the collection process are connected tasks.

Common misconceptions and mistakes

Do not assume deletion is harmless because it is simple. Removing rows can reduce sample size and change the represented population.

Do not treat a filled value as if someone measured it. Keep provenance when it matters to interpretation or downstream decisions.

Do not assume a single average preserves relationships between variables or uncertainty. Likewise, sophisticated imputation cannot reconstruct information that the observed data do not support. Reasons for missingness are not always identifiable from the observed table alone.

Business use

Businesses can record missingness rates as part of data quality monitoring, alongside completeness requirements for each workflow. A forecasting input, a customer preference, and an accounting amount need different handling policies.

Compare methods using evaluation data representative of the intended setting. Monitor whether missingness changes after deployment, particularly when a model learns to rely on an indicator. A change in forms, equipment, or data suppliers can alter the meaning of absence without changing the feature names.

Check your understanding

A training-column median is 12, but the evaluation-column median is 19. Which should a training-fitted median imputer use for missing evaluation values?

Answer: It should use 12. Relearning from evaluation values would let that dataset influence preprocessing and change the evaluation procedure.

Related terms

Data leakage
Preprocessing
Cross-validation
Data Quality
imputation
missingness indicator
distribution shift
multiple imputation

Learn next

compare mean and median imputation on a small dataset with an outlier. Then place the imputer inside a training pipeline and examine how evaluation changes when missingness patterns differ between training and deployment.

Training Pipeline

Sources

Sources: Aalto Dictionary of Machine Learning, missing data: https://aaltodictionaryofml.github.io/ADictML_Main.pdf ; scikit-learn, Imputation of missing values: https://scikit-learn.org/stable/modules/impute.html ; scikit-learn, Common pitfalls and recommended practices: https://scikit-learn.org/stable/common_pitfalls.html . Relevant primary definitions and preprocessing guidance were read. Calculations and case study are original teaching examples; no missingness mechanism is inferred solely from the observed values.

Previous: Mirrorless Next: Missing effect