Does coverage error need probabilities?

No. It needs comparable ranking scores for labels within each example.

Is a value of two always good?

No. Its meaning depends on the number of true labels, candidate labels and the task's acceptable inspection depth.

Equal scores can place a true label anywhere within a tied group; the documented largest-rank convention handles that ambiguity conservatively.

AI Glossary / Retrieval and knowledge systems / Intermediate

Coverage error

Definition

Coverage error is a measure of how far down a ranked list of labels a system must go to include every correct label for an example. In multilabel prediction, it averages that required depth across examples. Lower values indicate that the correct labels are concentrated nearer the top.

What it means

Here, coverage refers to covering the true labels within a ranking. It does not mean survey coverage, insurance coverage or the percentage of a knowledge base represented in search results. The metric is useful when an item can legitimately have several labels and users need to find all of them.

A single low-ranked correct label can increase the score substantially. This makes coverage error sensitive to whether the ranking leaves an important relevant option far below the others, even when its first suggestion is correct.

How it works

For each example, rank labels by their prediction scores. Find the deepest rank occupied by a true label, and average those depths across the dataset. Scores can be real-valued confidence scores; they do not have to be calibrated probabilities.

The scikit-learn implementation gives tied labels the largest rank within their tied group. Its convention also differs by one from the original reference convention, allowing examples with no true labels to receive zero coverage. State the implementation and convention when comparing results.

Under this depth convention, the best possible score is the average number of true labels per example. That baseline matters because datasets with more labels per item naturally require deeper rankings.

Three practical examples

1. A document-tagging system evaluates how many ranked subject suggestions an editor must inspect to encounter every assigned subject.

2. A support-ticket classifier checks whether secondary issue labels appear near the top rather than being buried below many incorrect categories.

3. A recommendation experiment with several relevant items per query measures the depth needed to include all judged relevant options within a fixed candidate set.

Think of it this way

Think of looking down a shopping checklist ordered by a helper. You stop only when you have found every item you actually need. Finding one useful item immediately is helpful, but your stopping point still depends on the last required item. Coverage error averages that stopping point across many lists, assuming the required items are known.

Worked example

Consider three invented examples with four candidate labels, A, B, C and D. Higher scores rank earlier.

For example one, the true labels are A and C, with scores 0.9, 0.8, 0.7 and 0.1 respectively. Their ranks are one and three, so the coverage depth is three.

For example two, only B is true. Scores are 0.8, 0.8, 0.2 and 0.1. A and B tie for the first two positions. Using the largest rank within the tie, B has rank two, so coverage is two.

For example three, A and B are true and scores are 0.9, 0.8, 0.2 and 0.1. Both appear within the first two positions, giving coverage two.

Average coverage error is (3 + 2 + 2)/3 = 7/3, approximately 2.33. The best possible average for these label sets is (2 + 1 + 2)/3 = 5/3, approximately 1.67. The difference does not establish business value or calibrated confidence; it describes the ranking depth on these three examples.

Illustrative case study

Fictional case: Pine Tagging evaluates an assistant that proposes several labels for each internal document. Editors report that the main subject is usually prominent, but a necessary secondary subject often appears late.

The team adds coverage error to its evaluation alongside precision at the number of suggestions actually displayed. It separates documents with one true label from documents with several, reviews incomplete annotations, and examines the worst rankings. A lower average becomes useful only when the visible suggestions also remain accurate and the labels reflect editors' real needs.

Common misconceptions and mistakes

A score of zero is not the normal target for examples containing true labels under the depth convention. Their minimum required depth depends on how many true labels they have.

Coverage error is not the same as classification error rate or recall at a fixed cutoff. It evaluates an entire scored ranking and does not require a decision threshold. Comparing values across different candidate sets or label densities can be misleading. Missing ground-truth labels also distort evaluation: an unannotated relevant label is not treated as true by the metric.

Business use

Use coverage error when the operational question concerns finding all relevant labels, rather than just making one useful suggestion. Report the candidate-label universe, annotation rules, tie handling and average number of true labels.

Pair it with metrics aligned to the interface, such as precision and recall at the displayed cutoff. Review difficult examples and evaluation slices before selecting a model. A reduction in average depth can reduce inspection effort, but it does not automatically establish faster work, better decisions or acceptable performance on rare labels.

Check your understanding

An example has three true labels, all ranked above every incorrect label. What is its best coverage depth under the depth convention?

Answer: Three. All three required labels are included within the first three positions. A depth of one would not cover the complete true-label set.

Related terms

Precision at k (precision@k)
Recall at k (recall@k)
Ranking loss
multilabel classification
label ranking
precision at k
recall at k
candidate set
confidence score and ground-truth annotation

Learn next

compare coverage error with label ranking average precision on the same small dataset. Then introduce a tied score and an additional true label to see how the required depth changes. Document the metric convention before comparing implementations.

Label ranking average precision

Sources

Primary sources: scikit-learn, coverage_error API documentation, including its depth definition and tie convention: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.coverage_error.html

scikit-learn, Model evaluation guide, coverage error section and mathematical definition: https://scikit-learn.org/stable/modules/model_evaluation.html#coverage-error

Previous: Coverage bias Next: Cox model