Is F1 always the best metric?

No. It combines precision and recall in one way but does not encode every business cost or requirement.

Does a high ROC AUC guarantee a useful threshold?

No. Threshold selection and performance under the relevant class prevalence still need examination.

Are predicted probabilities automatically calibrated?

No. Calibration must be evaluated separately from classification correctness.

AI Glossary / AI and machine learning / Intermediate

Evaluation of binary classifiers

Definition

Evaluation of binary classifiers measures how well a system distinguishes between two outcome classes. It examines correct predictions, different kinds of mistakes, score quality and the consequences of the decision threshold on representative data.

What it means

A binary classifier answers a two-class question, such as whether a record needs review. Calling one class positive and the other negative is a naming convention, not a judgment that one is good or bad.

Evaluation needs more than a single accuracy percentage. If the positive class is rare, a system can be correct most of the time by predicting negative for everything while missing every case of interest.

Different mistakes also have different consequences. Sending an ordinary support request for unnecessary review creates extra work; failing to flag a genuinely urgent request creates a different problem. The useful measures depend on that workflow.

For business readers, evaluation connects a statistical model to an operating decision. It should show what happens at the intended threshold, on the intended population, with the review capacity and costs the organization actually has.

How it works

Predictions are compared with trustworthy outcomes on data kept separate from training and tuning. A confusion matrix counts true positives, false positives, true negatives and false negatives.

Precision asks what share of predicted positives are genuinely positive. Recall asks what share of actual positives were found. Specificity concerns correctly identified negatives. Accuracy counts all correct decisions together.

If the model produces scores, changing a threshold changes the classification results. Curves such as ROC and precision-recall describe aspects of performance across thresholds, while calibration examines whether probability estimates match observed frequencies.

Evaluation also considers sample size, important subgroups, data leakage and change over time. A score from a convenient historical sample may not describe future traffic or a newly deployed operating process.

Three practical examples

Support triage: assess how many urgent requests are detected and how many ordinary requests create extra review work.
Quality inspection: measure missed defects separately from acceptable items unnecessarily rejected by a classifier.
Document screening: compare review workload and missed relevant documents at several candidate thresholds before choosing a policy.

Think of it this way

Think of a smoke alarm. Its usefulness depends both on detecting smoke and on avoiding unnecessary alarms. A device that never sounds can appear quiet and trouble-free while completely failing its purpose.

The analogy has a limit. Real evaluation must use the task's actual outcomes and costs. The same acceptable balance cannot be copied unchanged from one application to another.

Worked example

Seeing what accuracy hides

Suppose a fictional review dataset contains 1,000 records, of which 100 genuinely need attention. A classifier flags 120. Of those, 80 genuinely need attention and 40 do not.

The confusion matrix contains 80 true positives, 40 false positives, 20 false negatives and 860 true negatives. Precision is 80 divided by 120, approximately 66.7%. Recall is 80 divided by 100, or 80%.

Accuracy is 940 divided by 1,000, or 94%. A classifier that flagged nothing would reach 90% accuracy because 900 records are negative, but its recall would be zero.

The invented figures make the operational choice visible: 120 reviews find 80 relevant cases and miss 20. Whether that is acceptable depends on the task, review capacity and alternatives, not on the accuracy figure alone.

Illustrative case study

A fictional operations team selects a classifier using overall accuracy. After deployment, reviewers complain that too many ordinary records are flagged, while some important cases are still missed.

The team reviews the confusion matrix and threshold trade-off using representative traffic. It separates model scores from the routing policy and tests a revised threshold against review capacity and the cost of missed cases.

The lesson is that choosing a model and choosing an operating threshold are related but distinct decisions. A good aggregate ranking score does not automatically determine a suitable workflow.

Common misconceptions and mistakes

Accuracy is sufficient for every binary task. Class imbalance and unequal error costs can make it misleading.
High recall means most alerts are correct. That is a precision question; a system can find many positives while producing many false alerts.
A test set can be reused indefinitely for tuning. Repeated selection against it weakens its role as independent evidence.

Business use

Require a confusion matrix, a clear positive-class definition and results at the intended operating threshold. Ask how the evaluation sample represents future work and whether labels are reliable.

Translate the numbers into workload and consequences. A review process needs counts of cases handled, missed and unnecessarily escalated, along with uncertainty about those estimates. Recheck performance when the population or workflow changes.

Check your understanding

In the example, why can 94% accuracy coexist with missing 20 of the 100 important records?

Answer: Accuracy combines both classes and is heavily influenced by correctly handled negatives. Recall exposes the missed share of actual positives: 20% here.

Related terms

Learn next

Read confusion matrix and classification threshold. Convert a proposed model's percentages into expected daily counts of correct flags, false alerts and missed cases.

Sources

Google, Classification: Accuracy, recall, precision, and related metrics: https://developers.google.com/machine-learning/crash-course/classification/accuracy-precision-recall. scikit-learn, Metrics and scoring: https://scikit-learn.org/stable/modules/model_evaluation.html. Checked 12 October 2026. Worked figures and cases are original illustrations.

Previous: Evaluation metric Next: Evaluation of machine translation