Does MAP give more weight to queries with more relevant documents?

Not in the usual unweighted query mean; each query contributes one AP score.

Can an incomplete relevance set distort MAP?

Yes. Incorrect or missing judgments can change which retrieved items receive credit.

AI Glossary / Statistics and probability / Intermediate

Mean average precision

Definition

Mean average precision, often written MAP or mAP, is an average of average-precision scores over evaluation units such as search queries or object categories. In ranked information retrieval, average precision rewards relevant results appearing early and accounts for relevant items that are not retrieved. The exact definition depends on the evaluation protocol, especially the ranking depth, relevance labels, and interpolation rules.

What it means

Average precision summarizes that ranking quality for one query; mean average precision combines the query-level scores.
The two averages serve different purposes. The first considers precision at relevant-result positions. The second combines evaluation units. Reporting only the acronym can hide whether the units are queries, categories, or additional benchmark conditions.

How it works

For the standard noninterpolated retrieval definition, take a query with R known relevant items. Rank the returned items and calculate precision at every rank: the number of relevant items found so far divided by the number of items examined.
Add the precision values at ranks where a relevant item appears, then divide by R. A relevant item not retrieved contributes zero through the denominator. Thus retrieving one relevant item perfectly does not give full credit when several other relevant items are missed.
Average the resulting AP values across queries to obtain MAP. An ordinary unweighted mean gives every query equal importance, irrespective of its number of relevant items or frequency in actual traffic. Weighted variants need their weighting rule stated explicitly.
AP is sensitive to ordering. Moving a relevant result above irrelevant results can increase precision at its position and affect subsequent contributions.
Specify the evaluation depth. AP at a cutoff can use different normalization conventions, and it should not be compared casually with full-ranking AP. Queries with no relevant items also require a documented convention, such as exclusion or a designated score.
Check relevance judgments. Incomplete labels may cause a useful retrieved item to be treated as nonrelevant, while changing the relevance definition changes the task. Binary AP does not directly represent graded relevance in the same way as metrics designed for multiple relevance levels.
Handle tied scores and duplicate results consistently. Arbitrary ordering within a tied group can affect a rank-by-rank calculation. Repeating the same relevant document should not create additional genuine relevant items; use the benchmark's identity and matching rules.
In classification, AP is often calculated as a recall-increment-weighted sum of precision values at score thresholds. Scikit-learn's noninterpolated calculation is not a trapezoidal approximation to the precision-recall area. Different implementations should be compared by definition, not name alone.
In object detection, AP also depends on how predictions match reference objects, including localization thresholds. COCO's primary AP averages over specified intersection-over-union thresholds as well as categories under its protocol. An AP at one localization threshold is not the same metric.
Evaluate uncertainty and variation across units. A small increase in mean AP may reflect only a few queries, and an aggregate score can hide poor performance on important request types.

Three practical examples

A search team compares ranked document retrieval systems using AP for each query and MAP across the query set.
An AI retrieval project checks whether a reranker moves relevant evidence earlier in the returned list.
An object-detection team reports the precise benchmark AP definition, including localization thresholds, rather than an unexplained mAP number.

Think of it this way

Think of rewarding a search list whenever it delivers a relevant item, with a larger reward when few irrelevant items have appeared before it. Missing relevant items reduces the final score. MAP then averages these query-specific reports, so one easy query does not stand in for the whole evaluation.

Worked example

An original example uses two queries with binary relevance and no ties. Query A has three relevant items in total, appearing at ranks 1, 3, and 5 in the returned list. Precision at those positions is 1/1, 2/3, and 3/5.
Its AP is (1 + 2/3 + 3/5)/3 = 34/45, approximately 0.7556. Irrelevant positions do not contribute their own precision terms, but they lower the precision achieved at later relevant positions.
Query B has two relevant items, appearing at ranks 2 and 4. Its AP is (1/2 + 2/4)/2 = 0.5. Giving the queries equal weight produces MAP = (34/45 + 1/2)/2 = 113/180, approximately 0.6278.
If Query A's third relevant item were never returned, its contribution would become zero while the denominator would remain three. With the other relevant positions unchanged, its AP would be (1 + 2/3)/3 = 5/9, approximately 0.5556.
These are full-ranking retrieval calculations under the stated convention, not COCO detection scores or an unspecified AP-at-cutoff variant.

Illustrative case study

In a fictional case, Meadow Search tests a new evidence reranker for an internal assistant. MAP rises on a fixed query set, but the team inspects individual queries before declaring the change useful.
It finds that improvements are concentrated in short factual queries while several complex requests still return weak evidence early. Analysts retain the overall metric, document the query mixture, and expand the evaluation where coverage is thin. The case illustrates diagnosis beyond an average; it does not claim that retrieval improvement guarantees correct generated answers.

Common misconceptions and mistakes

Do not confuse MAP with precision at one cutoff, classification accuracy, mean reciprocal rank, or maximum a posteriori estimation, which shares the acronym MAP.
Do not divide AP only by the relevant items successfully retrieved unless that is explicitly the intended alternative metric. Do not compare detection scores with different matching thresholds or assume all precision-recall area calculations are identical.

Business use

MAP helps compare ranking systems when binary relevance and early retrieval are meaningful. It can support search and evidence-retrieval evaluation, but business relevance depends on the query set and user behavior. Pair it with latency, coverage, and task-success checks. A better evidence ranking does not alone establish that a downstream assistant uses the evidence correctly.

Check your understanding

Why does missing Query A's third relevant item reduce AP even though the other relevant ranks stay unchanged?

Answer: The denominator still counts all three known relevant items, so the missing item contributes zero.

Related terms

Precision at k (precision@k)
Recall
Precision-recall curve
Intersection over union
Average precision
Precision at k
Mean reciprocal rank
Relevance judgement

Learn next

Next, compare AP with reciprocal rank and normalized discounted cumulative gain on the same lists. Then document the ranking, cutoff, tie, and relevance conventions used in your evaluator.

Sources

Manning, Raghavan and Schutze, Introduction to Information Retrieval, Evaluation of ranked retrieval results: https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-ranked-retrieval-results-1.html Scikit-learn, average_precision_score: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html COCO, Detection evaluation: https://cocodataset.org/dataset/detection-eval.htm

Previous: Mean absolute percentage error Next: Mean directional accuracy