Can ROC AUC be calculated from hard labels?

Software may produce a value, but hard labels discard most ranking information and yield a much less informative curve than continuous scores.

Does AUC 0.5 prove a model is useless?

It indicates chance-level overall pairwise ranking in the evaluated setting, but interpretation still requires examining the task, scoring direction, and relevant regions.

Is AUC always ROC AUC?

No. Area can summarize other curves, so the curve type must be named.

AI Glossary / Causal inference and experimentation / Intermediate

AUC (Area under the ROC curve)

Definition

AUC, when specified as area under the ROC curve, summarizes how well prediction scores separate positive from negative examples across decision thresholds. The ROC curve plots true positive rate against false positive rate. For ordinary binary empirical ROC AUC, the score also equals the fraction of positive-negative pairs ranked correctly, counting tied scores as half a success. It measures ranking discrimination, not accuracy at one threshold or the calibration of probabilities.

What it means

A model can assign scores before anyone chooses which values trigger a positive decision. ROC AUC assesses the ordering implied by those scores across the full threshold range. An AUC of 1 indicates perfect separation in the evaluated sample; 0.5 corresponds to chance-level pairwise ordering in the usual reference setting. An AUC below 0.5 can reflect reversed scoring or poor ranking. None of these values by itself specifies whether a useful operating threshold exists for the application's particular error costs.

How it works

Start with held-out binary labels and scores oriented so larger values favor the designated positive class. At each threshold, calculate true positive rate as true positives divided by all actual positives, and false positive rate as false positives divided by all actual negatives. Integrate the resulting ROC curve using the stated convention. Binary ROC AUC requires both classes in the evaluation data. Multiclass and multilabel summaries need additional choices, such as one-versus-rest or one-versus-one comparisons and averaging, which should be reported explicitly.

Three practical examples

Model comparison: compare two ranking models on the same held-out binary task before selecting a deployment threshold.
Screening workflows: assess separation between cases that need review and those that do not, then inspect the low-false-positive region separately.
Monitoring: track whether score discrimination changes over time while also checking class composition, calibration, and threshold-specific outcomes.

Think of it this way

Think of asking whether a randomly chosen relevant item tends to stand above a randomly chosen irrelevant item in a ranked list. The analogy captures the pairwise meaning. Its limit is that a ranking score does not tell you how many items to review, whether scores are reliable probabilities, or how expensive a false decision will be.

Worked example

Suppose three positive examples have scores 0.9, 0.6, and 0.4, while three negative examples have scores 0.8, 0.4, and 0.1. There are three times three = nine positive-negative pairs. The positive score 0.9 beats all three negatives, contributing 3. The positive 0.6 beats 0.4 and 0.1 but loses to 0.8, contributing 2. The positive 0.4 beats 0.1, ties 0.4, and loses to 0.8, contributing 1.5 under the half-credit tie rule. ROC AUC is therefore 6.5/9, approximately 0.7222.
If a decision threshold of 0.5 is chosen, two positives and one negative are flagged. The true positive rate is 2/3, and the false positive rate is 1/3. That is one operating point, not the AUC itself. Applying a strictly increasing transformation to every score preserves their ordering and ties, so ROC AUC stays the same, even though the numerical values and the meaning of threshold 0.5 may change. This illustrates why discrimination and probability calibration are separate properties.

Illustrative case study

A fictional quality-control team compares two models with similar ROC AUC. Its review capacity requires a very low false-positive rate, and one model performs noticeably worse in that narrow region despite comparable total area. The team reports the full summary but bases its workflow assessment on recall at the allowable false-alarm level. It also checks how many actual items those rates imply. The selected model is evaluated as part of a decision process rather than chosen by the largest single number alone.

Common misconceptions and mistakes

AUC is not the percentage of correct predictions. It does not mean that a prediction with score 0.8 is correct 80% of the time. A high ROC AUC can coexist with low precision when positives are rare. Changing prevalence alone, while holding the class-conditional score distributions fixed, does not change the population ROC curve, but real population shifts can change those distributions. AUC values from different datasets or label definitions are therefore not automatically comparable.

Business use

Use ROC AUC as one view of ranking performance, with clear labeling of the positive class and score direction. Report the dataset, sample sizes by class, averaging convention, and uncertainty where appropriate. Pair it with precision-recall analysis and the confusion matrix at the intended threshold. If only a limited false-positive range matters, inspect that region directly or use a clearly specified partial-AUC measure. Keep ranking assessment separate from calibration assessment and from the economic or operational consequences of decisions.

Check your understanding

Why does the tied pair at score 0.4 receive half credit?

Answer: Under the standard empirical ROC AUC pairwise convention, equal scores do not prefer either class, so the tie contributes 0.5. Counting it as a full win or full loss would use a different and potentially misleading calculation.

Related terms

ROC (receiver operating characteristic) Curve
True positive rate
False positive rate
Precision-recall curve
Calibration
ROC curve
ranking discrimination
decision threshold
partial AUC
class imbalance
score direction

Learn next

Construct a confusion matrix at several thresholds and plot true positive rate against false positive rate. Then verify the area through positive-negative pair comparisons, including a tie. Study calibration and precision-recall analysis next so a strong ranking score is not mistaken for a complete assessment of a decision system.

Sources

Scikit-learn, roc_auc_score: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html Scikit-learn, roc_curve: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html Davis and Goadrich, The Relationship Between Precision-Recall and ROC Curves: https://pages.cs.wisc.edu/~jdavis/davisgoadrichcamera2.pdf

Previous: Attributional calculus Next: Auction algorithm