Is differential validity the same as differential prediction?

Terminology varies, but a useful distinction is correlation differences versus differences in prediction equations and systematic over- or underprediction.

Does it prove discrimination?

No. It is evidence requiring interpretation alongside measurement, sampling, context, and decision consequences; it is not a legal conclusion.

Can equal validity coefficients establish fairness?

No. They address one relationship and can coexist with other measurement or decision problems.

AI Glossary / Governance, ethics and safety / Intermediate

Differential validity

Definition

Differential validity concerns whether the relationship between a score or predictor and an intended outcome differs across groups. In its narrower psychometric usage, it often compares test - outcome correlations. Some sources use the term more broadly for differences in predictive relationships. For AI governance, it asks whether evidence supporting a score's interpretation carries across the populations in which it is used.

What it means

An AI-generated assessment score may relate strongly to a relevant outcome in one group and weakly in another. A single overall correlation can conceal that difference. However, a difference is not automatically proof of an unfair model or a difference in underlying ability. Outcome quality, measurement error, selection into the sample, and the range of scores can all affect the observed relationship. The term concerns evidence for a particular interpretation and use of scores. It should not be treated as a universal label that a test either possesses or lacks in every setting.

How it works

Analysts define the score, outcome, population, and intended use before comparing groups. They examine suitable measures of association and uncertainty, checking whether the outcome is measured comparably. In a narrow analysis, they compare validity coefficients such as correlations. A related differential-prediction analysis examines whether prediction equations systematically overpredict or underpredict outcomes for groups, including differences in slopes or intercepts. The distinction matters because correlation measures association without describing the entire prediction equation. Credible investigation also checks sample size, restricted score ranges, missing outcomes, and whether the observed groups represent the proposed deployment population.

Three practical examples

A training provider examines whether an AI-supported readiness score relates to later course completion similarly for participants using different language versions.
A skills platform compares the relationship between an assessment and an independently evaluated work sample across delivery formats.
A learning team checks whether the score - outcome relationship found among experienced employees also supports using the score with newcomers. Each comparison needs a defensible outcome and adequate data, not just a convenient group label.

Think of it this way

Think of two measuring instruments compared with a reference measurement. Both may move in the right direction, yet one follows the reference more closely or has a consistent offset. Correlation and calibration answer different questions about that behavior. The analogy is limited because human assessments involve context, opportunity, and imperfect outcomes rather than a simple physical reference. It still helps explain why one number cannot establish that scores function equivalently everywhere.

Worked example

Consider a deliberately tiny illustration with scores 1, 2, and 3 in each of two groups. In Group A, the outcomes are 2, 4, and 6. In Group B, they are 6, 4, and 2. Within Group A, the correlation is plus one; within Group B, it is minus one. When all six observations are pooled, the mean score is two and the mean outcome is four. The positive score - outcome cross-products in Group A sum to four, while those in Group B sum to minus four, so the pooled covariance and correlation are zero. The pooled result hides two very different relationships. Three cases per group are nowhere near enough to justify a deployment conclusion; these exact values only demonstrate the arithmetic. Real validity comparisons need uncertainty estimates, representative evidence, and examination of how outcomes were obtained.

Illustrative case study

Fictional case: Willow Skills evaluates a readiness score for an internal learning program. Its overall report shows a positive relationship with final project quality. A subgroup review finds that the relationship is less stable for staff who received a translated assessment, but that group also has fewer completed projects and a narrower score range. The team investigates the translation, scoring rubric, and missing outcomes before deciding what the difference means. It limits claims to the evidence available and gathers additional validation data. This invented case illustrates a careful inquiry rather than an automatic finding against a group or product.

Common misconceptions and mistakes

Different average scores do not by themselves establish differential validity; the question concerns the score - outcome relationship. A statistically significant correlation in one group and a nonsignificant correlation in another does not, by itself, prove that the correlations differ. Equal correlations also do not guarantee equal calibration, equal error rates, or fair decisions. The outcome can itself be biased or inconsistently measured. Finally, a small subgroup analysis may be too uncertain to support a confident conclusion even when a plotted difference looks substantial.

Business use

When purchasing or deploying an AI assessment, ask what outcome supports its use and for which populations that evidence was collected. Request subgroup sample sizes and uncertainty, together with information about outcome reliability and missing data. Keep association, prediction calibration, and decision fairness as distinct questions. Investigate plausible measurement and sampling explanations rather than attributing differences to inherent group characteristics. If evidence is weak for a proposed use, narrow the claim or collect suitable validation data. Decisions affecting people also need broader review of relevance, accessibility, and consequences beyond this statistical analysis.

Check your understanding

Two groups have the same score - outcome correlation, but a prediction equation consistently overestimates outcomes for one group. Has equal correlation ruled out a prediction problem?

Answer: No. Correlation does not determine the intercept or calibration of a prediction equation. Differential prediction still needs examination.

Related terms

Criterion validity
Correlation
Calibration
Range restrictions
Construct validity
predictive validity
differential prediction
measurement invariance
range restriction
subgroup evaluation

Learn next

Plot a score against its intended outcome separately for relevant groups. Compare association, prediction errors, and uncertainty, then document how sampling and outcome measurement might explain apparent differences. Keep the intended use visible throughout the analysis.

Sources

ACT Research, Criterion-Related and Differential Validity of ACT WorkKeys in Arkansas, original technical brief: https://www.act.org/content/dam/act/unsecured/documents/R2131-Criterion-Related-and-Differential-Validity-of-ACT-WorkKeys-in-Arkansas-07-2023.pdf Roohr and colleagues, Investigating Validity Evidence for the ETS Proficiency Profile, original research: https://doi.org/10.1002/ets2.12127 Kane and Mroch, Modeling Group Differences in OLS and Orthogonal Regression: https://doi.org/10.1080/08957347.2010.485990

Previous: Differential return Next: Differential value function