How does this differ from content validity?

Content validity concerns relevant coverage within an assessment; criterion validity concerns its relationship with an external reference or outcome.

Does the criterion have to occur later?

No. Concurrent validation uses a criterion observed at roughly the same time.

Can an AI model be used as the criterion?

Sometimes as a justified evaluation tool, but its own errors and shared biases must be examined; agreement between two models is not automatically truth.

AI Glossary / Governance, ethics and safety / Intermediate

Criterion validity

Definition

Criterion validity concerns how well a measurement relates to an appropriate external reference or outcome. It provides evidence for a particular use of the measurement by comparing its results with a criterion that has been justified for that purpose.

What it means

Suppose an organization uses a short assessment to estimate whether someone can complete a practical task. A useful question is whether assessment results correspond to independently observed task performance. That comparison supplies criterion-related evidence.

The criterion is not automatically perfect. A supervisor's rating can contain inconsistent judgments, and a recorded outcome can reflect resources or opportunity as well as the ability of interest. Calling something ground truth does not remove those limitations.

Terminology and validity frameworks differ across disciplines. A common distinction is between concurrent evidence, collected at roughly the same time as the measurement, and predictive evidence, concerning a later outcome. Neither alone establishes every claim that could be made about the measurement.

How it works

State the intended interpretation and choose a relevant criterion. Explain why that criterion is appropriate, how it is measured and which limitations it has. Collect measurement and criterion data in a way that avoids circular comparison.

For example, an evaluator rating an answer should not simply copy the AI's own confidence score. Similarly, a purported forecast cannot use information available only after the outcome occurred.

Select analysis that fits the claim. Correlation can describe association between numerical measures, but it does not establish close agreement or accurate individual predictions. Depending on the task, error sizes, calibration or classification performance may also matter. Evaluate uncertainty, differences across relevant settings and whether the relationship persists outside the sample used to develop the measure.

Three practical examples

A skills quiz is compared with independently scored practical exercises before its scores are used to describe task readiness.
An AI quality score is checked against structured expert assessments of the same outputs, with reviewers unaware of the AI score.
A forecasting indicator is evaluated against outcomes observed later, using only information available when the indicator was calculated.

Think of it this way

Think of checking a new ruler against a trusted reference. The comparison is useful only if the reference suits the measurement and the method checks the property that matters.

Human-performance measures rarely have a perfectly trusted ruler. The analogy therefore illustrates comparison, while reminding us to examine the external criterion rather than assume it is unquestionably correct.

Worked example

Perfect association, wrong numerical agreement

In an invented evaluation, an automated scoring tool assigns four submissions scores of 40, 50, 60 and 70. Independent reviewers assign those submissions 50, 60, 70 and 80 using the intended scoring scale.

The two sets have perfect positive linear correlation: each ten-point increase in one corresponds to a ten-point increase in the other. Yet the automated tool is ten points lower on every submission.

If the intended use requires reproducing the reviewers' numerical scores, correlation alone hides an important error. If the use is only ranking these four submissions, their ordering agrees, but four cases provide very limited evidence for broader performance. The choice of criterion and analysis must therefore match the actual decision the score is meant to support.

Illustrative case study

Fictional company Elm Training develops an AI score for employees' practice explanations. It initially validates the score against the number of positive reactions on an internal social feed.

The learning team points out that reactions depend on audience size and visibility as well as explanation quality. It instead defines a task-specific rubric and arranges independent review of a suitable sample. Reviewers record disagreements rather than hiding them behind a single supposedly perfect label.

The team examines how the AI score relates to those assessments and reports where the evidence is weak. It also keeps the broader question of practical teaching effectiveness separate from agreement with a rubric score.

Common misconceptions and mistakes

A high correlation proves interchangeable measurements. A systematic offset or scale difference may remain.
Any available outcome is an appropriate criterion. The outcome must fit the intended interpretation.
The criterion is always error-free. Its reliability and limitations need examination.
Predicting an outcome proves causation. A useful relationship can exist without the measurement causing the outcome.

Business use

Criterion-related evidence helps businesses judge claims that an AI score reflects quality, readiness, risk or another operationally important property. It encourages teams to ask what the score has actually been compared with.

Request the criterion's definition, collection method, sample and timing. Check whether the measure was tuned on the same evidence used to advertise its validity. Where the criterion is itself a proxy, describe that limitation rather than converting agreement into a broader claim about the underlying concept. Decisions may require several complementary forms of evidence.

Check your understanding

Two scoring systems correlate perfectly, but one always reports twice the other's score. Are they interchangeable on the same numerical scale?

Answer: No. Correlation captures their association, not numerical agreement. The intended use determines whether that scale difference is acceptable or requires correction.

Related terms

Construct validity
Content validity
Correlation
Measurement error
predictive validity
concurrent validity

Learn next

Read construct validity and calibration. Choose one operational score, identify the external result used to justify it, and explain why that result is relevant. Then identify one way the criterion could be misleading.

Construct validity

Sources

Cronbach and Meehl, Construct Validity in Psychological Tests, original 1955 article hosted by York University: https://psychclassics.yorku.ca/Cronbach/construct.htm. Its discussion distinguishes concurrent and predictive criterion-oriented studies and questions inadequate criteria. The workplace scenarios and score calculations are original teaching examples; the article is not presented as a current universal taxonomy.

Previous: Criss-cross algorithm Next: Criterion variable