Does “Bayes” mean a prior must appear in the classifier?

Not in the learning convention used here; the benchmark is defined from the joint distribution and loss, regardless of how a model is trained.

Can Bayes risk be zero?

Yes, for example when the class is determined by the available features under zero-one loss.

It is a rule's risk minus the optimal benchmark for the same problem.

AI Glossary / Governance, ethics and safety / Advanced

Bayes risk

Definition

Bayes risk, in the machine-learning sense used here, is the smallest expected loss achievable by a prediction rule given the available information, the underlying probability distribution and the chosen loss function. A rule attaining that minimum is a Bayes-optimal rule.

What it means

Some prediction problems contain uncertainty that the available features cannot resolve. Two parcels with identical recorded characteristics may still have different delivery outcomes. Even an ideal predictor using those characteristics cannot necessarily predict both outcomes perfectly.

Bayes risk describes the best theoretical average performance under the stated setup. It is not the training error of a fitted model, the error of a particular Bayesian software package or a promise that the true distribution is known.

Terminology varies across statistics. In classical Bayesian decision theory, the Bayes risk of a particular decision rule means its risk averaged over a prior distribution; one then minimizes that quantity over rules. In learning discussions, “the Bayes risk” often names the optimal value itself. Knowing the convention prevents confusing a rule's risk with the minimum achievable risk.

How it works

Specify the information X, the outcome Y and a loss that assigns a cost to each prediction and outcome. For each observed input, choose the action with the smallest conditional expected loss. Averaging these minimum conditional losses over inputs gives the optimal overall risk, under the usual conditions for this decision problem.

With zero-one classification loss, every incorrect class prediction costs one and every correct prediction costs zero. The optimal rule selects a class with the highest conditional probability. Its remaining error at an input is one minus that highest probability.

Different losses change the rule. Under squared-error prediction, the conditional mean is optimal when the relevant moments exist. Unequal classification costs can favor an action other than the most probable class. The benchmark therefore belongs to the combination of information, distribution and loss, not to a dataset name alone.

Three practical examples

A classifier researcher separates the error arising from limited information from additional error caused by a fitted model's imperfect learning.
An operations team considers whether an additional sensor could make outcomes more predictable, potentially lowering the theoretical risk for the expanded information set.
A decision designer compares actions under different error costs instead of assuming that the most likely outcome always determines the preferred action.

Think of it this way

Think of guessing which colored token will be drawn from a bag after receiving a clue. Even if the clue's relationship to the bag is known perfectly, it may leave several colors possible.

The analogy concerns residual uncertainty under specified information. It does not imply that real data come from a known bag, or that an organization can directly measure the exact Bayes risk from a small sample.

Worked example

Compute an ideal classification benchmark

In an invented two-class problem, half the cases have feature pattern A and half have pattern B. For A, the conditional probabilities are 0.8 for class 1 and 0.2 for class 0. For B, they are 0.4 for class 1 and 0.6 for class 0.

Under zero-one loss, the optimal rule predicts class 1 for A, incurring conditional error 0.2, and class 0 for B, incurring conditional error 0.4. Overall Bayes risk is 0.5 × 0.2 + 0.5 × 0.4 = 0.3.

A learned classifier with true risk 0.35 therefore has excess risk 0.05 relative to this benchmark. That comparison assumes the supplied probabilities are correct and the same loss and features apply. The 0.3 value is not an immutable property of the task: more informative features or a different loss can change the benchmark.

Illustrative case study

Fictional Cedar Dispatch tries to predict which parcels will require special handling using only package dimensions. A manager expects a larger model eventually to achieve perfect accuracy. The team explains that identical recorded dimensions can correspond to different contents and handling requirements.

It tests an additional declared-content field and checks its availability and reliability at prediction time. The field may reduce uncertainty, but the team does not claim to have measured the exact Bayes risk from its test set. It reports observed model performance and the limitations of the available information.

The case illustrates why improving features and improving a learning algorithm address different sources of error. It also avoids declaring every current error unavoidable merely because a theoretical irreducible component can exist.

Common misconceptions and mistakes

Bayes risk is always a classification error rate. Its meaning depends on the loss; squared error has different units.
A Bayesian model automatically achieves it. Approximate inference, misspecification and limited data can prevent optimality.
The best tested model reveals the exact Bayes risk. Better untested rules or information may exist.
The benchmark never changes. Changes to available features, population distribution or costs change the problem.

Business use

The concept helps businesses set realistic expectations about predictive performance and distinguish several improvement paths. A model may need more representative training data, a better learning method or information that the current features omit. These should not be treated as interchangeable remedies.

Use Bayes risk carefully in discussion: organizations rarely know the true probability distribution well enough to state its exact value. A claim that an error is unavoidable needs stronger evidence than the fact that several models made it. Likewise, an optimal average decision under a simplified cost model does not automatically satisfy every operational or governance requirement.

Check your understanding

For a feature pattern, class probabilities are 0.7 and 0.3. Under zero-one loss, what prediction and conditional error are optimal?

Answer: Predict the class with probability 0.7. The conditional error is 0.3; the overall risk also depends on other feature patterns and their frequencies.

Related terms

Bayes classifier
Decision theory
Conditional probability
Loss function
expected loss
excess risk
irreducible error

Learn next

Read expected loss and Bayes classifier. Write the outcomes, available information and cost of each mistake for one prediction task. Check whether maximizing accuracy actually represents the decision you need to make.

Bayes classifier

Sources

Aalto Dictionary of Machine Learning, Bayes risk: https://aaltodictionaryofml.github.io/ADictML_Main.pdf. Carnegie Mellon, Predictions and Decision Theory: https://www.stat.cmu.edu/~cshalizi/sml/21/lectures/02/lecture-02.html. Carnegie Mellon, Probability and Mathematical Statistics, decision theory: https://www.stat.cmu.edu/~siva/teaching/700/lec11.pdf. These support the learning and statistical conventions; examples are original illustrations.

Previous: Bayes-Nash equilibrium Next: Bayes theorem