What does the diagonal line represent?

A no-discrimination ranking has a diagonal population ROC and AUC 0.5; finite samples can fluctuate around that reference.

Can AUC be below 0.5?

Yes. Scores may rank positives below negatives more often than the reverse, or the score direction may have been specified incorrectly.

Does the curve require probability scores?

No. Ordered decision scores can be used if their direction is defined consistently.

AI Glossary / Causal inference and experimentation / Intermediate

ROC (receiver operating characteristic) Curve

Definition

A receiver operating characteristic curve, or ROC curve, plots a classifier's true-positive rate against its false-positive rate as the decision threshold changes. True-positive rate is recall: the fraction of actual positives detected. False-positive rate is the fraction of actual negatives incorrectly flagged. The curve describes the trade-off across threshold choices. It is not itself a choice of threshold, and it does not directly show the fraction of alerts that are correct.

What it means

A score-producing classifier can operate conservatively or aggressively. A high threshold may flag few cases; a lower one usually flags more. The ROC curve shows how those choices affect detection of positives and erroneous selection of negatives on a labeled dataset.

The horizontal axis is false-positive rate and the vertical axis is true-positive rate. A useful operating point aims for strong detection while keeping false-positive rate suitably low for the task. The upper-left region represents that preference, though practical requirements may focus on only a small part of the curve.

Area under the ROC curve, often called ROC AUC, summarizes ranking discrimination across thresholds. It is a related numerical summary, not another name for the curve itself.

How it works

Obtain a score and a reference label for each evaluation example. Define which class is positive and whether higher scores indicate stronger evidence for it. For each threshold, calculate true positives, false positives, true negatives, and false negatives.

Compute TPR = TP/(TP + FN) and FPR = FP/(FP + TN). Plot the resulting pairs with FPR horizontally. Moving from selecting no examples to selecting all examples traces the operating possibilities; tied scores must be treated according to the chosen threshold rule.

AUC can be interpreted as the probability that a randomly selected positive scores above a randomly selected negative, with the usual half-credit treatment for tied scores. It assesses ordering, not probability calibration. A strictly increasing transformation of scores preserves their ordering and ROC curve even though the numerical thresholds change.

Three practical examples

1. Quality inspection: engineers compare defect detectors over a range of false-positive rates before selecting an inspection threshold.

2. Message routing: a team checks how much urgent-ticket recall is available when the rate of unnecessarily escalated routine tickets is constrained.

3. Matching systems: analysts compare score rankings for genuine and nongenuine record pairs, while separately evaluating the precision of the final review queue.

Think of it this way

Think of adjusting the sensitivity of a motion sensor. More sensitivity can detect additional real movement while also responding to more irrelevant disturbances. The analogy is limited because the ROC curve uses rates within labeled positive and negative groups, not counts of all events. A low false-positive rate can still produce many false alarms if ordinary cases are extremely numerous.

Worked example

Suppose six fictional examples have scores 0.95, 0.80, 0.70, 0.50, 0.30, and 0.10, with labels positive, negative, positive, negative, positive, and negative. There are three positives and three negatives. Predict positive for scores at least the threshold.

At threshold 0.95, one positive and no negatives are selected: TPR = 1/3 and FPR = 0. At 0.80, one of each is selected: TPR = 1/3 and FPR = 1/3. At 0.70, two positives and one negative are selected: TPR = 2/3 and FPR = 1/3. At 0.30, all three positives and two negatives are selected: TPR = 1 and FPR = 2/3.

For the AUC interpretation, compare every positive with every negative. There are nine pairs. The positive scores 0.95, 0.70, and 0.30 outrank three, two, and one negative scores respectively. Six of nine pairs are correctly ordered, giving AUC 6/9, approximately 0.667. There are no ties in this toy example.

Illustrative case study

Fictional case: Poplar Monitoring evaluates an alert model with a small false-positive rate. In daily operation, however, ordinary cases vastly outnumber target events, so even that small rate would create a large review queue.

The team presents the relevant portion of the ROC curve together with projected alert counts and precision under a representative case mix. It chooses a candidate threshold for further assessment based on both detection needs and review capacity. The case shows why rates and workloads must be connected; it does not report an actual deployment or measured benefit.

Common misconceptions and mistakes

False-positive rate is not one minus precision. FPR divides false positives by actual negatives, while precision divides true positives by predicted positives. Confusing those denominators can seriously distort interpretation.

A high AUC does not guarantee good performance at the very low false-positive rate an application requires. Curves can cross, and the overall area can favor a model that is weaker in the operating region of interest. Nor does AUC show whether probabilities are calibrated.

ROC rates condition on the actual class, so simply changing class proportions while preserving class-conditional score distributions does not change the population ROC. Nevertheless, case-mix changes can change those distributions, and precision and workload can change with prevalence. Evaluate representative conditions rather than treating any metric as immune to dataset differences.

Business use

ROC curves help businesses compare score discrimination and examine threshold trade-offs. They are most useful when reports identify the relevant false-positive range and translate candidate points into understandable numbers of detections and reviews.

Pair ROC evidence with precision-recall analysis when positive cases are rare and alert usefulness matters. Record the positive-class definition, test population, and threshold-selection process. A strong ranking summary supports further evaluation, but operational success depends on the chosen point and the consequences of each error type.

Check your understanding

A threshold detects 18 of 20 actual positives and incorrectly flags 8 of 80 actual negatives. What ROC point does it produce?

Answer: FPR is 8/80 = 0.10 and TPR is 18/20 = 0.90, so the point is (0.10, 0.90).

Related terms

Learn next

Create six scored examples with binary labels and calculate ROC points at each distinct threshold. Compare one operating point's false-positive rate with its precision to practice keeping their denominators separate.

False positive rate

Sources

Google, Classification: ROC and AUC: https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc

Scikit-learn, Metrics and scoring, ROC-related evaluation: https://scikit-learn.org/stable/modules/model_evaluation.html

The primary teaching references ground the curve and discrimination interpretation. Worked scores, fictional case, and exercise are original.

Previous: Robustness Next: Rocchio algorithm