Does rejection require a probabilistic classifier?

No. A selection rule can use other suitable scores or a separately learned error predictor, though its performance still needs validation.

Is rejecting more always better?

No. It reduces coverage and may increase delay or cost; a poor ranking can also reject easy cases while retaining difficult ones.

Does reject mean none of the known classes applies?

Not necessarily. It means the system declines to issue its classification. Detecting an unknown class is a distinct objective.

AI Glossary / Probabilistic machine learning / Intermediate

Classification with rejection

Definition

Classification with rejection allows a classifier to abstain from assigning a class to selected inputs. Instead of forcing a prediction for every case, the system can return a reject decision for further handling. It is also called classification with a reject option or selective classification, and it trades prediction coverage against the risk of incorrect accepted predictions.

What it means

A conventional classifier chooses a label from its known classes. A classifier with rejection also decides whether to issue that label at all.
Rejection is an action, not necessarily a new real-world class. An uncertain document does not become a member of a semantic category called rejected; the system is withholding its classification.
The next step might be additional information, another model, or a human review. That workflow has its own costs and limitations. Rejection does not make a difficult case disappear or establish that the eventual decision will be correct.

How it works

Combine a predictor with a selection rule. The predictor proposes a class, and the selection rule determines whether to accept that proposal. A rule may threshold a score, estimate conditional error risk, or be learned jointly with the predictor.
One formulation assigns costs to misclassification and rejection and minimizes expected cost. Another seeks the lowest selective risk subject to a coverage target, or the greatest coverage subject to a risk constraint.
Coverage is the proportion or probability of inputs receiving an accepted prediction. Selective risk measures loss among those accepted predictions. These quantities must be reported together because rejecting nearly everything can make the remaining cases look easy.
Select and validate the rule using appropriate data. Estimated scores may be miscalibrated, and performance under a changed input distribution may differ from the validation result.

Three practical examples

A document-routing system sends uncertain cases to a review queue while assigning sufficiently supported cases automatically.
An image-labeling pipeline withholds a label when its selection rule indicates that the prediction is unreliable.
A support classifier asks for a missing detail rather than choosing between two plausible issue categories immediately.

Think of it this way

Think of a quiz participant who may pass on a question instead of guessing. Passing can reduce the number of wrong answers submitted, but the number of questions actually answered remains part of the result.
The analogy is limited: an automated system may be confidently wrong, and its score is not a direct measure of self-knowledge. The selection rule needs evidence that it can separate more reliable predictions from less reliable ones.

Worked example

Consider an original evaluation with 100 inputs. A classifier accepts 80 predictions, of which 76 are correct and four are wrong, and rejects 20.
Coverage is 80/100 = 80%. The accepted-set error rate is 4/80 = 5%, so accepted-set accuracy is 95%. It would be misleading to describe the system as correctly handling 95 of all 100 inputs: it has made 76 correct accepted predictions, and twenty cases still require another action.
Now consider a separate idealized decision with correct-classification cost zero, wrong-classification cost one, and rejection cost 0.2. If the true highest class probability is 0.85, predicting that class has expected loss 0.15, below rejection cost. If it is 0.7, expected loss is 0.3, so rejection is preferable under these costs.
The resulting threshold is 0.8, with a tie in expected cost at exactly 0.8. This calculation assumes the stated probabilities and costs; an arbitrary model confidence score cannot automatically be substituted for a true class probability.

Illustrative case study

In a fictional project, Cedar Records adds rejection to an automatic document sorter. A strict rule improves accuracy among accepted documents but sends more cases to a review team than it can process promptly.
The team compares operating points using both accepted-case errors and review volume. It also checks whether particular document types are rejected disproportionately and whether the fallback process resolves them correctly.
This invented case illustrates why rejection is a workflow decision as well as a model feature. It reports no measured saving or production error reduction.

Common misconceptions and mistakes

Do not report selective accuracy without coverage. A model that answers only a small easy subset is not directly comparable with one answering almost everything.
Do not count rejected cases as correct predictions when evaluating the automatic classifier.
Do not assume that the largest softmax score is a calibrated probability of correctness.
Do not confuse rejection with out-of-distribution detection. A selection rule can reject difficult familiar inputs, and it can fail to reject unfamiliar ones.
Finally, do not interpret a validated risk bound as unconditional protection under any future data distribution. Such guarantees depend on the method and its assumptions.

Business use

Classification with rejection can support partial automation when the organization can handle abstentions through a defined alternative process. It gives a team a way to choose where the model is allowed to commit to a label.
Assess the complete process: accepted prediction quality, fraction rejected, review capacity, delay, and the quality of fallback decisions. Monitor whether the risk - coverage relationship changes over time. A useful reject option should improve the intended decision process, not merely produce a more flattering accuracy figure by excluding difficult cases.

Check your understanding

A model accepts 50 of 200 inputs and makes two errors among those accepted. What are coverage and accepted-set accuracy?

Answer: Coverage is 50/200 = 25%, and accepted-set accuracy is 48/50 = 96%. Both figures are needed to describe this operating point; 150 cases remain rejected.

Related terms

selective prediction
abstention
confidence calibration
risk–coverage curve
out-of-distribution detection
human review
learning to defer

Learn next

draw a risk - coverage curve by varying a selection threshold on held-out data. Then add a realistic handling cost for rejected cases and examine which operating points fit the actual workflow.

Sources

Primary sources: Franc, Prusa, and Voracek, Optimal Strategies for Reject Option Classifiers: https://www.jmlr.org/papers/volume24/21-0048/21-0048.pdf Geifman and El-Yaniv, SelectiveNet: https://proceedings.mlr.press/v97/geifman19a/geifman19a.pdf The arithmetic and fictional case are original illustrations, not published benchmark results.

Previous: Classification threshold Next: Classifier chains