Is error probability one minus accuracy?

For ordinary classification with the same cases and definition of correctness, yes.

Can the Bayes error be positive?

Yes. The available inputs may leave genuine uncertainty about the label even under the true distribution.

Does zero observed test error prove zero error probability?

No. A finite sample may miss possible mistakes, especially rare ones.

AI Glossary / Statistics and probability / Intermediate

Probability of error

Definition

Probability of error is the probability that a specified decision or prediction is incorrect under a stated probability model or population distribution. In classification, it is P(g(X) ≠ Y), where g(X) is the predicted class and Y is the true class. The definition requires a clear decision rule and meaning of error. In other settings, such as communications or hypothesis testing, the relevant error event must be specified separately.

What it means

The quantity describes how often a rule would be wrong across the target distribution, rather than whether one particular prediction happens to be correct. An observed test-set error rate estimates this probability when the evaluation data and sampling conditions support that interpretation.

The decision rule matters as much as the predictive model. Different thresholds applied to the same scores can produce different error probabilities. The distribution matters too: changing the proportions of easy and difficult cases, or of different classes, can change overall error even when performance within each group stays unchanged.

How it works

Define the input information, possible labels, prediction rule, and target population. For each possible input, calculate the conditional probability that the chosen label is wrong. Average those conditional error probabilities over the distribution of inputs to obtain the overall probability of error.

Under equal costs for all incorrect classifications, choosing the class with the highest true conditional probability minimizes conditional error. Its error probability at that input is one minus that highest probability. Averaging these minima gives the Bayes error rate for the available information and distribution.

In practice, true conditional probabilities are unknown. Evaluate a fitted rule on appropriately separated labeled data and estimate its error with uncertainty. Repeated observations from the same person, device, or period may require special evaluation designs. A simple fraction of mistakes does not automatically account for dependence, sampling bias, or changes after deployment.

Three practical examples

A document-routing system estimates the probability that a new document is assigned to the wrong department.
A symbol decoder measures how often received signals are decoded into an incorrect transmitted symbol.
An evaluation team compares classification error across operating conditions instead of relying only on the combined rate.

Think of it this way

Think of repeatedly applying the same sorting rule to a stream of items. Probability of error describes the share expected to enter the wrong bin under the stream's assumed composition. Change the rule or the composition, and that share can change even if the bins themselves stay the same.

Worked example

Consider an original binary-classification model with three input groups. Their population probabilities are 0.20, 0.50, and 0.30. The true probabilities of class 1 within the groups are 0.90, 0.70, and 0.40.

With equal error costs, predict class 1 in the first two groups and class 0 in the third. Their conditional error probabilities are therefore 0.10, 0.30, and 0.40.

Overall error probability = 0.20 × 0.10 + 0.50 × 0.30 + 0.30 × 0.40 = 0.29.

The resulting accuracy is 0.71. Always predicting class 1 would instead yield error 0.20 × 0.10 + 0.50 × 0.30 + 0.30 × 0.60 = 0.35.

The informed rule reduces error by 0.06, or six percentage points. Because the example supplies the true probabilities, 0.29 is the Bayes error for these inputs. A real fitted model would not generally know those probabilities exactly.

Illustrative case study

Fictional case study: Cedar Documents evaluates an automatic filing assistant. A low overall error rate initially looks reassuring, but a review finds that uncommon request types account for many costly mistakes.

The team reports the total error rate, class-specific confusion counts, and estimated consequences of each error type. It also tests a different threshold and a human-review option. The decision considers both mistakes and review workload. This fictional case illustrates why minimizing the number of errors is not always identical to minimizing operational harm.

Common misconceptions and mistakes

Probability of error is not the same as the probability that a hypothesis is false given the data. In testing, Type I and Type II errors condition on different states and must be distinguished.

Another mistake is equating training error with future error. A model can fit training labels closely and generalize poorly. An apparently low aggregate error can hide serious failures in a minority class. Finally, an estimated classifier probability is not automatically a calibrated probability of correctness. A confident prediction can still be wrong, and calibration must be evaluated rather than assumed.

Business use

Businesses use error probabilities to estimate correction workloads, compare classification rules, and design monitoring. If the expected volume is known, an error probability can support an expected mistake count, subject to the same population assumptions.

Before using it as a target, decide whether all mistakes have comparable costs. When they do not, report expected loss or separate error types as well. Specify how abstentions, ambiguous labels, and missing ground truth are handled. Evaluation should reflect the situations the deployed AI system will actually encounter.

Check your understanding

A classifier makes 18 mistakes in 600 independently sampled, relevant test cases. What is the observed error rate?

Answer: It is 18/600 = 0.03, or 3%. This is an estimate of the target error probability, with sampling uncertainty, rather than proof that all future batches will have exactly 3% errors.

Related terms

Learn next

Start with confusion matrices and the relationship between accuracy and error. Then study conditional risk, Bayes decision rules, and unequal error costs. Learn evaluation designs and uncertainty intervals before comparing small differences between observed model error rates.

Accuracy

Sources

James, Witten, Hastie, and Tibshirani, An Introduction to Statistical Learning, discusses test error and the Bayes classifier; indexed author-hosted passages were consulted, while full-file retrieval exceeded the browser limit: https://www.statlearning.com/s/ISLRSeventhPrinting.pdf

Scikit-learn documents empirical zero-one classification loss: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.zero_one_loss.html

The finite population model, calculations, and fictional case are original. Classification is the main scope; other error events require their own definitions.

Cornell CS4780 lecture notes explain classification by the largest conditional class probability: https://www.cs.cornell.edu/courses/cs4780/2018fa/lectures/lecturenote05.html

Previous: Probability of direction Next: Probability of improvement