Is TOC the same as ROC?

They use different axes; TOC makes the underlying counts visible when totals are supplied.

Can ROC be converted to TOC?

Yes, if the required class totals and dataset size are also known.

Does the score need to be a probability?

No. A ranking score with a defined direction can be evaluated.

AI Glossary / Data science and analytics / Advanced

Total operating characteristic

Definition

The total operating characteristic, or TOC, is a curve for evaluating how a ranked score distinguishes a binary reference outcome across thresholds. In its count-based form, it plots the number classified positive against the number of true positives. Together with the dataset size and reference-positive total, the curve makes the four confusion-matrix counts recoverable at each threshold.

What it means

A score can rank cases without specifying where to draw a decision boundary. Changing that boundary changes how many cases are flagged, how many flags are correct and how many positives are missed. TOC displays this trade-off while retaining the sizes of the relevant groups.
The four counts are hits, misses, false alarms and correct rejections. These correspond to true positives, false negatives, false positives and true negatives. “Positive” means the outcome designated as present in the evaluation; it need not mean a desirable result.
TOC is related to the receiver operating characteristic, or ROC. ROC displays true-positive and false-positive rates. TOC keeps count information visible, which can be useful when the number of cases requiring action matters.

How it works

Choose a binary reference label and a score direction. If larger scores indicate stronger evidence of the positive class, lower the threshold progressively from above the largest score to the smallest score. At each threshold, count the selected cases and the true positives among them.
Let N be the number of observations, P the reference-positive total, x the selected count and y the hits. Then false alarms equal x − y, misses equal P − y, and correct rejections equal N − P − x + y. These identities explain how the full count table can be recovered.
The curve begins at (0, 0) and ends at (N, P). Handle tied scores consistently: a threshold cannot separate tied cases without an additional rule. Record whether the data are unweighted observations or weighted estimates, because the plotted quantities need the appropriate interpretation.

Three practical examples

An editorial team evaluates a score for finding articles with a defined formatting defect. It compares how many articles would require review with how many reference-labeled defects each threshold identifies.
A service team evaluates a score for routing requests to specialist review. It examines missed cases and unnecessary referrals alongside the total review workload.
A research group compares two ranking methods on the same labeled collection. It inspects the region corresponding to its feasible review capacity instead of relying only on an overall summary.

Think of it this way

Imagine reading down a ranked inspection list. Every additional item increases the workload by one, but only some items add a genuine finding. TOC records both the growing workload and the accumulated findings. The reference-positive total also tells you how many findings remain outside the selected portion of the list.

Worked example

Suppose a fictional collection contains N = 100 articles, of which P = 30 have a reference-labeled formatting defect. At one threshold, the score flags x = 40 articles and correctly identifies y = 24 defective articles. The TOC point is (40, 24).
False alarms are 40 − 24 = 16. Misses are 30 − 24 = 6. Correct rejections are 100 − 30 − 40 + 24 = 54. The four counts sum to 24 + 16 + 6 + 54 = 100.
For comparison, the ROC coordinates would be false-positive rate 16 / 70, approximately 22.9%, and true-positive rate 24 / 30 = 80%. Precision is 24 / 40 = 60%, another ratio.
At a lower threshold, suppose 50 articles are flagged and 27 defects found. Ten additional reviews find three additional defects and add seven false alarms. Whether that trade-off is worthwhile depends on the workflow's costs and capacity. These invented counts assess agreement with a reference; they do not establish that the reference labels are perfect.

Illustrative case study

In a fictional project, Hawthorn Publishing tests two scores for prioritizing a formatting check. The team has capacity to inspect only part of the collection, so it records findings and workload across thresholds.
Analysts build TOC curves on a held-out labeled sample and inspect the range near the team's practical review capacity. They retain the confusion counts and review disputed reference labels. They also check whether the evaluation sample resembles the article types likely to arrive later.
Hawthorn chooses a provisional operating rule for a pilot and monitors the resulting workload and missed defects. It does not treat the curve as selecting a universally best threshold. This fictional case combines diagnostic evidence with operational requirements.

Common misconceptions and mistakes

TOC does not automatically choose an optimal threshold. Different costs for missed positives and false alarms can justify different choices, as can limited review capacity.
Do not confuse its horizontal coordinate with the false-positive count alone. In the count-based formulation used here, the coordinate is hits plus false alarms: all selected positives.
A score need not be a calibrated probability to generate a TOC curve. Ranking performance does not establish probability calibration. Also, raw counts from a deliberately unrepresentative sample should not be presented as expected operational counts without considering the sampling design.

Business use

TOC connects ranking performance to operational workload. It is useful when decision-makers need to understand workload, correct findings and both types of error together.
Report the positive-class definition, evaluation population and threshold rule. Compare curves under compatible conditions, and keep a table of relevant operating points for exact decisions. If the population size or positive prevalence changes, old counts may no longer describe the new workload. Validate the intended operating rule on suitable evidence before relying on it at scale.

Check your understanding

What does (40, 24) mean in the example?

Answer: Forty articles are flagged and 24 are reference positives.

How many positives are missed?

Answer: Six.

Why are ten extra reviews not automatically beneficial?

Answer: Their value depends on additional findings, errors, cost and capacity.

Related terms

Learn next

Calculate confusion counts at several thresholds for a small labeled ranking. Plot selected count against hits, then compare the same operating points in ROC coordinates.

Sources

Robert Gilmore Pontius Jr, Metrics That Make a Difference, 2022, Chapter 2: TOC axes and recovery of contingency-table counts. https://wordpress.clarku.edu/rpontius/wp-content/uploads/sites/884/2024/10/Pontius-Jr-2022-Metrics-That-Make-a-Difference.pdf TOC package documentation, authored by the method's researchers and collaborators: threshold evaluation and distinction from ROC. https://search.r-project.org/CRAN/refmans/TOC/html/TOC-package.html The article counts and Hawthorn Publishing case are original illustrations using unweighted observations.

Previous: Total functions Next: Total Turing Test