AI Glossary / Causal inference and experimentation / Intermediate

Label ranking average precision

Definition

Label ranking average precision, or LRAP, evaluates how well a model ranks the relevant labels for each example above irrelevant labels. For each relevant label, it calculates the fraction of labels scoring at least as highly that are also relevant, then averages those fractions over relevant labels and examples. It is a ranking measure for multilabel predictions. It does not require choosing a probability threshold to turn every score into a yes-or-no decision.

What it means

A document can concern several topics at once. A classifier may assign a score to every topic, producing an ordered list of suggestions. LRAP asks whether the topics that truly apply are near the top of that list, where a person or another system is more likely to encounter them.

The metric works within each example, comparing that example's labels against one another. This differs from ranking documents for a query or evaluating one label across many documents. It also differs from checking whether the complete selected label set exactly matches the reference. A strong ranking can still be converted into poor final selections by an unsuitable threshold.

How it works

Supply a binary reference indicator for every example - label pair and a model score for the corresponding pair. Higher scores must mean a stronger prediction that the label applies.

For each true label, count all labels with scores greater than or equal to its score. Among those, count the true labels. Divide the second count by the first. Average these fractions across the true labels in the example, then average across examples under the stated weighting rule.

Including equal scores defines tie handling: a tied irrelevant label can lower the fraction. Examples without any true labels require an implementation convention because the direct within-example formula has no true labels to average. Document such cases instead of leaving readers to infer their treatment.

Three practical examples

1. Article-topic suggestions: editors receive ranked topic candidates, and LRAP checks whether all applicable topics tend to precede distracting suggestions.

2. Image tagging: a model ranks possible objects for each photograph, while evaluation uses reference sets containing multiple visible object categories.

3. Support classification: a message may concern billing and access simultaneously, so developers evaluate the ranking of both relevant categories before deciding how many categories to display.

Think of it this way

Think of arranging a toolbox so that the tools needed for one job appear before unnecessary ones. Each needed tool is judged by how much clutter appears before or beside it in the ordering. The analogy is limited: LRAP averages mathematical ranking fractions and does not know that one relevant label may be far more important or costly to miss than another.

Worked example

Consider four labels A, B, C, and D. The true labels are A and C. Model scores are A: 0.9, B: 0.8, C: 0.7, and D: 0.1.

For A, only A scores at least 0.9, and it is true, giving 1/1 = 1. For C, three labels score at least 0.7: A, B, and C. Two are true, giving 2/3. The example's LRAP is (1 + 2/3)/2 = 5/6, approximately 0.833.

If C moves above B, both true labels have fractions of one and the score becomes 1. If A and B instead tie at the top while C remains below them, A's fraction is 1/2 and C's is 2/3, averaging about 0.583. These calculations show why tie handling and within-example order matter.

Illustrative case study

Fictional case: Orchard Topics creates a label-suggestion interface for editors. Developers initially evaluate only whether the first suggestion is correct. That overlooks cases where an article has two relevant topics and the second is buried below several irrelevant options.

The team adds LRAP and reviews the ranking for multi-topic articles. It separately tests the actual interface, which displays only three suggestions, and keeps a measure of performance at that display limit. A higher LRAP becomes evidence of improved label ordering, while the interface test answers whether editors receive the right shortlist. The fictional case illustrates complementary questions rather than claiming a measured productivity gain.

Common misconceptions and mistakes

Do not feed selected labels where the evaluation expects ranking scores and assume the information is equivalent. Binary selections create ties and discard ordering detail.

Do not interpret LRAP as probability calibration. Scores can preserve the same order while their numeric values are badly calibrated.

Avoid confusing sample-level label ranking with a class-wise average-precision calculation. State the evaluation axis and aggregation. Also inspect the reference labels: an unrecorded but valid topic may be counted as irrelevant, so incomplete annotation changes what the metric rewards.

Business use

LRAP is useful when several labels may apply and their ordering has practical value. It can support comparisons before a product team settles on thresholds or display counts, especially when relevant labels should be concentrated near the top.

The business decision still needs metrics aligned with the interface or downstream action. A shortlist may have a strict length, or missing a particular category may carry extra cost. Review results by label frequency and number of true labels per example. Document scoring direction, ties, empty-reference cases, and sample weighting so that model comparisons remain interpretable and reproducible.

People also ask

Must the scores be probabilities?

No. They must provide the intended ordering, with higher values indicating greater relevance. Probability calibration is a separate property.

What if each example has one true label?

With distinct scores, its contribution is the reciprocal of that label's rank, linking the average to mean reciprocal rank.

Does a score of one prove the chosen label threshold is correct?

No. It indicates ideal relevant-label ordering under the metric, while the final threshold can still select too many or too few labels.

Check your understanding

A true label has five labels scoring at least as highly, of which three are true. What fraction contributes for that label?

Answer: 3/5 = 0.60. The calculation includes the label itself and any tied labels, rather than counting only labels with strictly higher scores.

Related terms

• Ranking loss
• Coverage error
• Threshold (for decision trees)
• Calibration
• Multilabel classification
• label ranking
• average precision
• mean reciprocal rank
• threshold

Learn next

Create two examples with four candidate labels each and more than one true label. Calculate LRAP by hand, then swap a relevant and irrelevant label in one ranking. Add a tie and explain its effect. Finally, compare this ranking measure with the quality of a fixed two-label shortlist.

More in Causal inference and experimentation

• k-fold cross validation - K-fold cross validation is an evaluation procedure that divides data into k non-overlapping parts, or folds, and repeatedly trains a model on all but one...
• L 1 loss - L1 loss measures prediction error using the absolute difference between a predicted value and its observed target. For one example it is |prediction − target|.
• L 2 loss - L2 loss commonly means squared prediction error: the square of the difference between a prediction and its observed target.
• Labeled precision - Labeled precision is the proportion of predicted structures that match reference structures in both their structural identity and their label.
• Labeled recall - Labeled recall is the proportion of reference structures that a system recovers with the correct structural identity and label.
• Leave-one-out cross-validation - Leave-one-out cross-validation evaluates a learning procedure by repeatedly training on all observations except one and testing on the single omitted...

Sources

Primary implementation source: scikit-learn, Label ranking average precision, https://scikit-learn.org/stable/modules/model_evaluation.html#label-ranking-average-precision . The documented set-and-rank formula and equal-score comparison were consulted. This entry follows that definition and flags empty-reference handling as implementation-specific. Scores, rankings, and the fictional editor project are original illustrations, not benchmark results.

Previous: Label Propagation Next: Label shift