AI Glossary / Causal inference and experimentation / Intermediate

Average precision at k

Definition

Average precision at k, commonly written AP@k, evaluates how early relevant items appear within the first k positions of a ranked result list. It adds the precision measured at positions containing a relevant item and divides by a specified normalization count. Several conventions exist, so the denominator must be stated. This entry uses the number of relevant items capped at k, a convention implemented in Apache Spark's ranking evaluation.

What it means

A search system should ideally place useful results near the top. Precision at k only counts how many of the first k results are relevant. AP@k also responds to their ordering because an early relevant result contributes precision before later irrelevant items dilute the list. For several queries, mean average precision at k, or MAP@k, averages their individual AP@k values. The words average and mean therefore describe different levels of aggregation.

How it works

For one query, define a set of relevant item identifiers and a ranked list of distinct predicted identifiers. At each rank i from 1 through k, compute precision P(i): the number of relevant items encountered so far divided by i. Add P(i) only when the item at i is relevant. Under the capped convention, divide this sum by min(R, k), where R is the total number of relevant items in the reference set. State how empty relevance sets, short lists, duplicate items, and tied scores are handled before comparing results.

Three practical examples

• Search ranking: evaluate whether useful help articles appear early among the first five results.
• Recommendations: compare how well two systems rank a user's held-out relevant items within a limited recommendation panel.
• Retrieval for an assistant: assess the ordering of relevant source passages before separately checking whether the generated answer uses them correctly.

Think of it this way

Think of inspecting the first few folders in a filing tray. Finding the right folders earlier reduces how much irrelevant material precedes them. The analogy captures ranking quality, but AP@k does not measure actual reading time, the varying usefulness of different relevant folders, or whether several folders repeat the same information.

Worked example

Suppose the reference set contains three relevant documents: A, B, and C. A system returns A, X, B, Y, C in that order, where X and Y are irrelevant. With k = 5, the relevant positions are 1, 3, and 5. Their precisions are 1/1 = 1, 2/3, and 3/5. AP@5 under the capped denominator is (1 + 2/3 + 3/5)/3 = 34/45, approximately 0.7556. Precision@5 is simply 3/5 = 0.6.
If the order becomes A, B, C, X, Y, precision@5 stays 0.6, while AP@5 becomes (1 + 1 + 1)/3 = 1. Now suppose there are ten relevant documents in the collection and the top five are all relevant. Capped AP@5 is 5/min(10, 5) = 1. A truncated convention dividing by all ten relevant documents instead gives 5/10 = 0.5. Both calculations are transparent, but they answer different normalization choices. A reported AP@5 without its convention is therefore incomplete.

Illustrative case study

A fictional internal-search team sees its AP@5 improve after a ranking update. Inspection reveals that the test judgments mark only one older policy document relevant, while several valid new documents are unjudged. The team reviews the judgments and checks document versions before interpreting the change. It also separates queries with no relevant document from ordinary ranking failures. The resulting report describes both ranking performance and the limits of the reference set instead of treating every unjudged item as conclusively useless.

Common misconceptions and mistakes

AP@k is not just precision@k averaged over all positions. Contributions come from relevant positions, and the denominator follows the chosen definition. MAP@k is not the same as pooling all returned items into one list. Binary relevance ignores degrees of usefulness, and observed clicks are not automatically reliable relevance labels. AP@k also does not prove that a generated answer is supported: retrieval quality and answer quality require separate assessment. A cutoff change alters the evaluation and should not be hidden.

Business use

Use AP@k when early ordering and multiple relevant results matter for a limited display or retrieval budget. Choose k to reflect the intended workflow, then keep it fixed for comparisons. Report the relevance definition, judgment coverage, denominator, query weighting, and policy for empty cases. Compare systems on the same queries and examine individual wins and losses. If graded relevance, redundancy, or user effort is central, supplement AP@k with measures designed for those properties rather than expecting this one score to capture them.

People also ask

Can AP@k reach 1 when some relevant items lie below k?

Yes, under the capped denominator when at least k relevant items exist and all top-k results are relevant.

What if no relevant items are known?

The formula needs a convention; some implementations assign zero, while exclusion changes the evaluated population and must be documented.

Does moving a relevant item earlier help?

With distinct items and fixed binary relevance, moving it ahead of irrelevant items improves or preserves the score within the evaluated cutoff.

Check your understanding

Two lists contain the same three relevant items in their first five positions. Must their AP@5 match?

Answer: No. Their precision@5 matches, but AP@5 can differ because it rewards relevant items appearing earlier in the ranking.

Related terms

• Precision at k (precision@k)
• Recall at k (recall@k)
• Mean average precision
• Ranked retrieval
• Normalized Discounted Cumulative Gain
• Reciprocal rank
• Precision at k
• recall at k
• relevance judgment
• cutoff
• recommendation evaluation

Learn next

Calculate precision at every position in two short ranked lists, then sum only at relevant positions. Repeat with both capped and uncapped denominators to see the distinction. Next study MAP aggregation and graded relevance measures, and examine how incomplete judgments affect interpretation before using an offline ranking score for a product decision.

More in Causal inference and experimentation

• Augmented inverse probability weighting - Augmented inverse probability weighting, or AIPW, is a causal estimation approach that combines outcome predictions with corrections weighted by treatment...
• Automatic evaluation - Automatic evaluation uses software to assess a model, its outputs, or an AI-enabled process according to specified tests or scoring rules.
• Autorater evaluation - Autorater evaluation uses an automated rating system, often a language model, to judge outputs against defined criteria or compare competing responses.
• Average treatment effect - The average treatment effect, or ATE, is the average difference between the outcome a defined population would experience under one intervention and the...
• Average treatment effect on the treated - The average treatment effect on the treated, or ATT, is the average causal effect of an intervention among the units that actually received it.
• Balanced accuracy score - Balanced accuracy score measures classification performance by averaging the recall of each class with equal class weight.

Sources

Apache Spark, RankingMetrics documentation: https://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/evaluation/RankingMetrics.html Apache Spark, RankingMetrics source implementation: https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala TensorFlow Ranking, MeanAveragePrecisionMetric: https://www.tensorflow.org/ranking/api_docs/python/tfr/keras/metrics/MeanAveragePrecisionMetric

Previous: Average node degree Next: Average reward