What are false positives and false negatives?

A false positive is flagged but innocent; a false negative is a real case the model missed.

What is the diagonal of the matrix?

The cells where prediction matches truth: the correct calls. Everything off it is a specific error.

Which metric comes from it?

Precision, recall, accuracy and others are all arithmetic on the matrix's cells.

Can I get a confusion matrix for a vendor model?

Yes, if you run it on your own labeled sample; a vendor unwilling to be tested that way is telling you something.

AI Glossary / AI and machine learning / Beginner

Confusion matrix

Definition

A confusion matrix is a table that lays a classifier's predictions against the true classes, showing not just how often it is right but exactly which mistakes it makes.

What it means

Overall accuracy squashes performance into one number, and one number hides everything interesting. The confusion matrix opens it up: each row is the true class, each column the predicted one.

For two classes there are four cells. True positives and true negatives are the wins. False positives are false alarms: flagged but innocent. False negatives are misses: guilty but passed.

Those two error types have different prices. A missed cancer and a false cancer scare are both errors, but nobody prices them alike, and the matrix keeps them separate so the pricing can happen.

With more classes the matrix grows a row and column per class, and its value shifts: you read which classes get confused with which, which is where the name comes from.

For a business, the confusion matrix is the minimum honest report for any classifier. If a vendor shows you accuracy and no matrix, you have not seen the model's report card.

Every common classification metric is arithmetic on this table, so the matrix is the one artifact from which the whole report can be rebuilt.

How it works

Run the classifier on labeled examples it did not train on, and count every outcome into the cell where its true class meets its predicted class.

Read the diagonal: those are the correct calls. Everything off the diagonal is an error of a specific, named kind.

Compute the metrics you need from the cells: precision asks how clean the flags are, recall asks how few real cases were missed, per class.

In multi-class matrices, scan the off-diagonal cells for the largest counts: those pairs are where the model, or the class design, needs work.

Compare matrices, not just scores, when choosing thresholds or models, because the same accuracy can hide very different error shapes.

Three practical examples

Fraud: a two-by-two matrix showing how many frauds were caught versus how many legitimate transactions were blocked.
Ticket routing: a five-by-five matrix revealing that delivery and returns tickets are constantly swapped.
Medical screening: a matrix checked to ensure misses stay near zero even at the price of many false alarms.

Think of it this way

Think of a football referee's season review. Right decisions matter, but the review wants the pattern of wrong ones: does he miss fouls or invent them? The confusion matrix is that review.

The limit: it only shows what happened on the test data. A beautiful matrix on easy cases can accompany a useless model in the real world.

Worked example

reading a routing model's real mistakes

Suppose a firm tests its ticket router on 1,000 held-out tickets. Accuracy is 85 percent, which sounds fine.

The confusion matrix shows the detail: billing and refunds are confused in 60 cases, nearly half of all errors. The figures are illustrative.

The team does not tune the model first; it rewrites the labeling rules for the billing-refund boundary and relabels the disputed slice. Confusion drops by half.

The lesson: the matrix told them where to look, and the fix was in the class definitions, not the algorithm.

Illustrative case study

A fictional manufacturer evaluates two defect-detection models. Model A has 95 percent accuracy, model B 93 percent, and A seems the winner.

The matrices reverse the verdict: A misses twice as many critical defects as B, and a missed critical defect costs fifty times a false alarm. B is deployed.

The invented lesson: never compare classifiers on accuracy when errors have unequal prices. The confusion matrix is where unequal prices meet the evidence.

Common misconceptions and mistakes

High accuracy means a good model. The matrix routinely exposes 95-percent-accurate models that are useless for the class that matters.
The confusion matrix is only for two classes. It grows to any number of classes and gets more useful as it grows.
It is a training tool. It is an evaluation tool, computed on data the model did not train on.

Business use

Ask for the confusion matrix on held-out data before accepting any classifier, and have the error costs in your head when you read it.

Use the largest off-diagonal cells to drive fixes: class definitions, more labels for the confused pair, or a process change for that case.

Keep the confusion matrix from each evaluation with its date; a series of matrices over time is the clearest picture of model health you can own.

Check your understanding

Why does the confusion matrix beat accuracy for judging a classifier?

Answer: Because it separates error types and shows which classes are confused, and those distinctions, not the overall score, carry the business cost.

Related terms

Learn next

AI for Business Owners shows how to read model report cards before you trust them with decisions: https://aiforbusinessandlife.com/academy/ai-for-business-owners. [C2]

Sources

[S2] Google, Machine Learning Glossary; [S3] NIST AI Resource Center glossary; [S12] Aalto University, Dictionary of Machine Learning; [S14] KNIME, Data Science Glossary; [S19] scikit-learn documentation, Metrics and scoring; [S8] Jurafsky and Martin, Speech and Language Processing (3rd ed. draft). Full URLs are in the source ledger. Verified 11 October 2026.

Previous: Confounding Next: Conjoint analysis