AI Glossary / Graphs and symbolic AI / Beginner

Reciprocal rank

Definition

Reciprocal rank is a retrieval-evaluation score equal to one divided by the position of the first relevant or correct result in a ranked list. If no qualifying result is found within the evaluated list, the score is zero. It rewards systems that place a useful first answer near the top, where a user can encounter it quickly.

What it means

A first correct result at rank one gives a score of 1. At rank two, the score is 0.5; at rank four, it is 0.25. Rank positions begin at one, not zero.

The metric focuses on the first success. Once that success is found, later relevant results do not increase the score. This makes reciprocal rank useful for tasks where finding one suitable answer is central, but less informative when users need a complete collection of relevant material. The judgment of what qualifies as relevant remains a separate, essential part of evaluation.

How it works

For each query, inspect the ranked results in order using a defined relevance or correctness rule. Find the first qualifying position r and calculate 1/r. If none qualifies, assign zero.

To summarize several queries, average their individual reciprocal ranks. This is mean reciprocal rank, commonly abbreviated MRR. If evaluation is limited to the first k results, state that cutoff and apply it consistently; a qualifying result below the cutoff receives no credit in that evaluation.

Keep the ranked list, relevance judgments, query set, and handling of ties or missing results reproducible. A score is meaningful only relative to these choices. In particular, a model's own confidence score is not the same quantity as reciprocal rank.

Three practical examples

• Help-center search: a team measures how early the first useful support article appears for each test question.
• Question answering: evaluators examine ranked candidate answers and score the position of the first answer meeting the correctness criteria.
• Retrieval-augmented generation: developers assess whether a retriever places a supporting passage early in its results, while evaluating the generated answer separately.

Think of it this way

Think of looking through a stack of labeled folders until you find the first folder that answers your question. Reciprocal rank rewards finding it near the front. The analogy has limits: the score does not measure actual reading time, the quality of later folders, or whether one folder is enough to complete the wider task.

Worked example

Suppose four test queries have their first relevant result at ranks one, two, four, and nowhere in the evaluated list. Their reciprocal ranks are 1, 0.5, 0.25, and 0.

The mean is (1 + 0.5 + 0.25 + 0) divided by 4 = 0.4375. The query with no relevant result remains in the denominator. Dropping it would instead produce about 0.5833 and overstate performance for the original four-query test.

Now apply a cutoff of three results to the same lists. The rank-four result is outside the evaluated range, so the scores become 1, 0.5, 0, and 0. MRR at three is therefore 1.5 divided by 4 = 0.375.

For the first query, suppose results at ranks two and three also happen to be relevant. Its reciprocal rank remains one because the first relevant result was already at rank one. Conversely, a system that moves a first success from rank two to rank one gains 0.5 on that query, while moving it from rank four to rank three gains only about 0.0833. The scale strongly emphasizes improvements near the top.

These calculations describe ranking quality under the stipulated judgments. They do not establish a probability that a generated answer is true or a percentage of all relevant documents retrieved.

Illustrative case study

Fictional case: Willow Support compares two internal search prototypes. Its team defines which articles answer each test question before examining the competing rankings.

One prototype has a higher MRR, but reviewers discover that it still fails entirely on several questions. They retain the average while also reporting the number of unanswered queries and inspecting those failures. For questions requiring several documents, they add an evaluation that reflects broader coverage.

The case illustrates how to interpret the metric alongside the task. It does not report a measured improvement or claim that one ranking score captures the full support experience.

Common misconceptions and mistakes

Do not average ranks and then take a reciprocal. That is a different calculation from averaging reciprocal ranks.

Do not omit zero-score queries merely because the system failed to return a correct result. Doing so changes the evaluated population.

Avoid assuming extra relevant results improve this metric after the first success. They may improve the product experience, but reciprocal rank does not reward them. Also, incomplete or inconsistent relevance judgments can distort comparisons. A different cutoff, query set, or correctness rule can change the score even when the system itself has not changed.

Business use

For a business, reciprocal rank is useful when the practical question is whether users can find a first satisfactory answer quickly. Before making it a target, verify that a single relevant item really matches the task.

In the worked example, the average conceals a complete failure on the fourth query. Reviewing those failures may be more useful than celebrating a small overall increase. Keep evaluation queries representative, record the cutoff, and distinguish retrieval success from the correctness of any answer later generated from retrieved material.

People also ask

What score does a correct result at rank five receive?

It receives 1/5, or 0.2, if rank five is inside the evaluated range.

Is reciprocal rank the same as MRR?

Reciprocal rank is the per-query score. MRR is the average of those scores across the evaluated queries.

Does a score of 0.5 mean half the returned results are correct?

No. For an individual query it means the first qualifying result is at rank two. It is not precision.

Check your understanding

Why does the four-query MRR fall from 0.4375 to 0.375 at a cutoff of three?

Answer: The third query's first relevant result is at rank four, so it changes from 0.25 to zero when only the first three results count.

Related terms

• Ranking
• Precision
• Recall
• Information retrieval
• mean reciprocal rank
• relevance judgement
• evaluation cutoff
• first relevant result

Learn next

calculate reciprocal rank for several hand-labeled lists, including ties and missing answers under an explicit evaluation policy. Then compare what changes precision, recall, and reciprocal rank reward. Choose a metric set that reflects whether users need one answer or many.

• Precision
• Recall

More in Graphs and symbolic AI

• Real-time dynamic programming - Real-time dynamic programming, or RTDP, is a method for improving decisions by updating value estimates along trajectories through a decision problem.
• Real-time search - Real-time search is a family of search methods that limit the computation performed before an agent commits to its next action.
• Reasoning - Reasoning is the process of using information, assumptions, and relationships to reach conclusions, evaluate possibilities, or decide what follows.
• Reciprocity - Reciprocity measures how often directed connections are matched by connections in the reverse direction.
• Recombination - Recombination is an evolutionary-computation operation that creates candidate solutions by combining information from parent candidates.
• Reconstruction - Reconstruction, in computer vision, is the process of estimating a scene's three-dimensional structure from observations such as photographs.

Sources

Primary source: NIST TREC, Question Answering Collections, including the reciprocal-rank evaluation definition: https://trec.nist.gov/data/qa.html The official historical evaluation description was read. Numerical lists and Willow Support are original illustrations; the historical five-response task is not presented as a universal cutoff.

Previous: Reciprocal human machine learning Next: Reciprocity