Does good discrimination imply good calibration?

No. Ranking can remain unchanged even after a transformation makes probability values misleading.

Is AUC a classification threshold?

No. It summarizes performance across thresholds; an operational threshold must be chosen separately.

Is statistical discrimination the same as unlawful discrimination?

No. They are different concepts. Predictive ranking performance does not settle questions of fairness or acceptable treatment.

AI Glossary / Statistics and probability / Intermediate

Discrimination

Definition

Discrimination, in predictive statistics, is a model's ability to distinguish observations with different outcomes by assigning appropriately ordered predictions. For a binary outcome, it concerns whether cases experiencing the event tend to receive higher predicted scores than cases without it. This source-specific statistical meaning differs from unfair treatment of people. Good predictive discrimination also differs from calibration, which concerns whether predicted probabilities agree with observed event frequencies.

What it means

A model can rank cases well while systematically overstating their probabilities. Conversely, a constant prediction matching the overall event rate can be calibrated in a limited aggregate sense while distinguishing no individual cases. These observations show why ranking and probability accuracy require separate assessment. Discrimination is often summarized by the area under the receiver operating characteristic curve, or AUC, for binary outcomes. Other outcome types and censored event times require appropriate measures and definitions rather than automatic reuse of a binary metric.

How it works

Evaluate predictions on suitable observations not used to tune the reported assessment. For binary outcomes, compare the scores of event and nonevent cases. The usual empirical AUC counts the fraction of such pairs in which the event case has the higher score, assigning half credit to tied scores. It corresponds to the area under the curve of sensitivity versus false-positive rate across thresholds. An AUC of one indicates perfect separation in the evaluated data; 0.5 represents no overall ordering advantage by this measure. Report uncertainty and the evaluation population. Discrimination can change with case mix, measurement quality and deployment conditions, so it is not an immutable property of an algorithm.

Three practical examples

Customer retention: assess whether customers who later cancel tend to receive higher cancellation scores than those who remain.
Quality monitoring: compare scores for items with and without verified defects before selecting an inspection threshold.
Model review: evaluate ranking performance alongside probability calibration and the consequences of errors at operational thresholds.

Think of it this way

Think of sorting a queue by expected urgency. Discrimination asks whether genuinely urgent cases tend to appear ahead of less urgent ones. Calibration asks whether a stated seventy-percent urgency probability corresponds to the right frequency. A useful ordering and accurate numerical probabilities are related goals, but they are not the same achievement.

Worked example

Suppose three hypothetical event cases have scores 0.9, 0.6 and 0.4, while three nonevent cases have scores 0.7, 0.3 and 0.2. There are nine event-nonevent pairs. The event score 0.9 exceeds all three nonevent scores; 0.6 exceeds two; and 0.4 exceeds two. With no ties, AUC is (3 + 2 + 2)/9 = 7/9, approximately 0.778. Now square every score. The transformed event scores become 0.81, 0.36 and 0.16, and the nonevent scores become 0.49, 0.09 and 0.04. Their ordering is unchanged, so AUC remains 7/9. Yet the numerical probability claims have changed substantially. This demonstrates why the same discrimination can coexist with different probability accuracy. Six cases are far too few to treat the numerical AUC as a precise population result.

Illustrative case study

Fictional case: Ash Customer Analytics compares two cancellation models. They have similar ranking performance, but one assigns probabilities consistently higher than the observed cancellation frequencies in later data. The team examines calibration and the costs of contacting customers at different thresholds before choosing a deployment approach. It also reviews performance for relevant customer groups. This invented case illustrates multidimensional model evaluation; it does not establish that a higher AUC produces higher retention or that a statistically discriminating model is socially fair.

Common misconceptions and mistakes

Do not interpret AUC as the percentage of individual predictions that are correct. It summarizes pair ordering across thresholds, not accuracy at one chosen cutoff. A high AUC does not prove calibration, causal usefulness or fairness. Strictly increasing transformations preserve rankings and hence ordinary AUC, even when probability interpretations change. An AUC below 0.5 can reflect reversed score direction or other problems, but reversing scores after inspecting evaluation data must not be mistaken for an independently validated improvement. Censored survival data require attention to which pairs are comparable and how censoring is handled.

Business use

Businesses use discrimination measures to assess prioritization systems and compare models, but operational decisions need more information. A model may rank well overall while performing poorly at the narrow threshold used for action or within an important subgroup. Evaluate calibration, error costs, data coverage and decision consequences alongside ranking. Distinguish performance estimates from their uncertainty and avoid selecting a model through repeated informal comparisons on the same test set. Validation should resemble the intended deployment population, including relevant time changes and differences in event prevalence or case mix.

Check your understanding

Why does squaring every example score leave AUC unchanged?

Answer: Squaring is strictly increasing over these positive scores, so every event-nonevent comparison has the same ordering. The metric uses those orderings rather than the numerical distances or absolute probability values.

Related terms

Calibration
ROC (receiver operating characteristic) Curve
Classification threshold
Validation
ROC curve
area under the curve
concordance index
sensitivity
specificity
predictive performance
case mix
fairness

Learn next

calculate AUC by counting event-nonevent pairs and compare it with accuracy at a chosen threshold. Apply a monotonic score transformation and observe unchanged ranking. Then inspect calibration curves and decision-specific metrics to understand what discrimination alone leaves unresolved.

Calibration curves

Sources

Harrell, Glossary of Statistical Terms, discrimination and ROC concepts: https://hbiostat.org/glossary/ Harrell, Biostatistical Modeling Plan: https://www.fharrell.com/post/modplan/ Harrell, Statistically Efficient Ways to Quantify Added Predictive Value of New Measurements: https://www.fharrell.com/post/addvalue/index.html

Previous: Discriminant analysis Next: Discriminative model