AI Glossary / Statistics and probability / Intermediate

Data validation according to a list

Definition

Data validation according to a list is the checking of a data value against an explicitly defined set of acceptable values for a field. A value passes the check if it belongs to that set under the stated matching rules. The method is commonly used for category codes, controlled vocabularies, form options, and structured data exchanged between systems.

What it means

The list defines permitted representations, such as approved status codes or department identifiers. It prevents unexpected spellings or unsupported categories from entering a process unnoticed.

Passing the check establishes membership in the allowed set, not the factual correctness of the record. A valid department code can still refer to the wrong department. A valid status can still be inappropriate for the event or date.

The list also needs context. Values may be valid only during certain periods, for particular populations, or within a specific version of a classification. Validation must use the correct reference rather than treating any available list as universally authoritative.

How it works

Define the field's data type, permitted values, and matching convention. Decide how to handle capitalization, surrounding spaces, leading zeros, missing values, and alternative labels. If normalization is allowed, perform the documented transformation before checking membership.

Apply the check at collection or ingestion and retain a clear failure reason. A failed value may need correction from the source, review, mapping through an approved dictionary, or an update to the reference list. Do not silently add every new value simply to make the check pass.

Maintain list ownership and version history. Validate related rules separately, such as required-field presence or whether a permitted status transition is allowed. A dropdown can reduce entry variation, but imported files and automated outputs still need independent checks using the same definitions.

Three practical examples

• A survey accepts only published industry codes for a classification field, with explicit handling of missing or not-applicable responses.
• An AI extraction system returns a case status that is checked against approved labels before the result enters a business workflow.
• A data exchange validates unit codes against a controlled vocabulary so that unsupported abbreviations are reviewed before measurements are combined.

Think of it this way

Think of checking a ticket against a guest list. A listed name satisfies one admission condition, but the list does not establish that the person presenting it is the right individual. Additional checks establish the rest of the situation.

Worked example

Consider an original status field whose allowed values are OPEN, REVIEW, and CLOSED. Five incoming records contain OPEN, closed, REVIEW, PAUSED, and an empty string. First use exact, case-sensitive matching, with no blank value permitted.

OPEN and REVIEW pass. The other three fail: closed differs in case, PAUSED is absent from the list, and the empty string is not an accepted status. The initial pass proportion is 2/5 = 40%.

Suppose the documented ingestion policy allows trimming surrounding spaces and converting letters to uppercase before validation. The second value becomes CLOSED and passes. PAUSED and the empty value still fail, so the pass proportion becomes 3/5 = 60%.

This improvement reflects a permitted representation change, not evidence that all three passing records are factually correct. If the second case is actually still open, CLOSED remains a valid code assigned incorrectly. Conversely, if PAUSED is a legitimate new business state, the appropriate response is to review the vocabulary version rather than automatically recode it as CLOSED.

Illustrative case study

In this fictional case, Linden Records uses an AI tool to extract document categories from incoming files. The receiving system permits INVOICE, RECEIPT, and CONTRACT. The tool sometimes returns “purchase invoice” or a new category for a document outside those three types.

The team creates an approved mapping for verified synonymous labels and routes unsupported categories for review. It keeps the original extracted text, the mapped value, and the vocabulary version. It does not force every unfamiliar document into the closest existing category.

The case is invented to illustrate controlled representation and exception handling. It does not claim that list validation establishes correct document understanding or eliminates the need for evaluation.

Common misconceptions and mistakes

List membership is not a completeness, accuracy, or consistency test for the entire record. A field can contain a permitted value while other required fields are missing or contradictory.

Do not confuse an absent field, an empty string, and a designated unknown value. These may have different meanings and different validation outcomes. An explicit null or unknown code belongs in the policy only when its use is intended.

Uncontrolled fuzzy matching can turn an error into a plausible but wrong code. A stale list can reject genuinely valid new categories, while an overly broad list can admit values inappropriate for the application. Review both the data and the reference definitions when failure patterns change.

Business use

Businesses can use list validation to make exchanges more predictable, improve category consistency, and prevent unsupported AI outputs from triggering downstream actions. Its value is greatest when each field has a clear meaning and reference owner.

Record failure rates by source and category, but do not treat a low failure rate as proof of high factual accuracy. Keep a review route for legitimate exceptions and a controlled process for vocabulary updates. When labels change, provide mappings and effective dates so historical records remain interpretable and reports do not silently mix incompatible classification versions.

People also ask

Is a dropdown sufficient?

It helps during manual entry, but other ingestion paths may bypass it, and users can still choose the wrong permitted option.

Can a list contain numbers?

Yes. The permitted set can contain numeric codes or other supported types, but type and formatting rules must be explicit.

Should every invalid value be deleted?

No. Retain enough source information to investigate and resolve the issue according to a documented policy.

Check your understanding

Check your understanding: Why did closed fail the first check?

Answer: Matching was case-sensitive.

Why did PAUSED still fail after normalization?

Answer: Changing letter case did not add it to the allowed set.

Does CLOSED passing establish that the case is closed?

Answer: No; it establishes only that the code is permitted.

What should happen when a valid new state appears?

Answer: Review and update the controlled vocabulary through its defined process.

Related terms

• Controlled vocabulary
• Schema
• Categorical variable
• Data editing
• Missing values
• Referential integrity
• Data validation
• code list
• enumeration
• reference data
• missing value

Learn next

Design a small vocabulary with explicit missing-value and version rules. Then add separate checks for required fields and permitted transitions, and test examples that are valid codes but incorrect or inconsistent records.

More in Statistics and probability

• Data generating process - A data generating process, often abbreviated DGP, is the mechanism or probabilistic specification that produces observations.
• Data normalization - Data normalization is the transformation of numerical data into a scale or form suitable for analysis or machine learning.
• Data reduction - Data reduction is the creation of a smaller or simpler representation of data for a defined analytical purpose.
• De Finetti's theorem - De Finetti's theorem connects exchangeability with conditional independence.
• De Moivre's law - De Moivre's law is a simple actuarial survival model in which lifetime is uniformly distributed between zero and a specified limiting age.
• Deck imputation - Deck imputation fills a missing data item using information from a donor record.

Sources

Sources: OECD Glossary of Statistical Terms, Data validation according to a list, https://www.oecd.org/content/dam/oecd/en/publications/reports/2008/09/oecd-glossary-of-statistical-terms_g1gh9ad7/9789264055087-en.pdf. JSON Schema, Enumerated values, https://json-schema.org/understanding-json-schema/reference/enum, for an implementation example of fixed-set validation. The status calculation and fictional extraction case are original.

Previous: Data type Next: Data vault modeling