AI Glossary / Retrieval and knowledge systems / Intermediate

R-precision

Definition

R-precision is a ranked-retrieval evaluation measure calculated at a cutoff equal to the number of relevant documents for the query. If R documents are known to be relevant, R-precision is the fraction of the first R retrieved documents that are relevant.

What it means

Suppose a query has five relevant documents in the evaluation collection. R-precision examines the first five search results. If three are relevant, the score is three fifths, or 0.60.

The cutoff varies across queries. A query with two relevant documents is evaluated at rank two; a query with twenty relevant documents is evaluated at rank twenty. This differs from precision at a fixed cutoff such as ten, where every query uses the same result-list depth.

The measure requires relevance judgments and a count of relevant documents. In practice, that count may refer to the known relevant set in a benchmark rather than every relevant document that could exist. The interpretation must respect the completeness and quality of those judgments.

How it works

For each query, identify the judged relevant set and let its size be R. Examine the top R unique retrieved documents, count r relevant items and calculate r/R. Under the same relevance set, recall at that cutoff is also r/R because both denominators equal R.

For a query set, evaluators often average the individual R-precision scores. This gives each included query equal weight in the arithmetic mean. Pooling all relevant counts and all cutoff sizes instead produces a different, weighted calculation.

The metric uses binary relevance in its basic form. It does not distinguish a highly useful document from a marginally relevant one, and it ignores the internal ordering of relevant items within the cutoff. Queries with R = 0 require an explicit evaluation policy because division by zero is undefined.

Three practical examples

• A document-search team compares two ranking methods against the same set of judged queries.
• A research benchmark uses a query-specific cutoff to account for variation in the number of relevant documents.
• A retrieval review examines R-precision alongside top-result and graded-relevance metrics to understand different aspects of ranking quality.

Think of it this way

Think of checking the first R places in a display when R is the number of desired items available. The score asks what share of those places contains a desired item.

The analogy assumes you know which items are desired. If that inventory is incomplete, the score is limited by that knowledge. It also does not tell you whether the best item appears first or last within the inspected portion.

Worked example

Two queries and two averaging choices

For an invented query A, the judged relevant set contains five documents: D1, D2, D3, D4 and D5. The top five results are D1, X1, D3, X2 and D5. Three of the five are relevant, so R-precision is 3/5 = 0.60. Precision and recall at rank five are both 0.60.

For query B, there are two relevant documents, and one appears in the first two results. Its R-precision is 1/2 = 0.50. The arithmetic mean across the two queries is (0.60 + 0.50)/2 = 0.55.

Pooling the counts instead gives (3 + 1)/(5 + 2) = 4/7, approximately 0.571. That is not the same averaging rule. Also, moving D1 from rank one to rank five in query A would not change R-precision if the same relevant documents remained inside the top five.

Illustrative case study

The fictional CedarSearch team evaluates an internal policy search engine. A new ranking method improves mean R-precision, but reviewers still complain that a directly useful policy is often below several less helpful relevant documents.

The team recognizes that its binary metric counts all judged relevant documents equally and ignores their order within the cutoff. It retains R-precision for continuity while adding a metric and review process better suited to the first results users actually read.

The case illustrates complementary evaluation needs. It does not claim a measured improvement or imply that one metric can establish overall search quality.

Common misconceptions and mistakes

• R is the number of results the interface happens to show. It is the number of relevant documents in the evaluation's relevance set.
• R-precision rewards every improvement in ranking order. Rearranging documents within the cutoff can leave it unchanged.
• A score based on incomplete judgments measures all real-world relevance perfectly. It measures performance relative to the available judgment set and evaluation policy.

Business use

R-precision can support repeatable comparisons when a judged test collection is available. Its changing cutoff makes it sensitive to the size of each query's relevant set, and a perfect ranking can score one for any query with at least one relevant document.

Use the same query set, judgments and document-identity rules when comparing systems. State how unjudged results, missing results and zero-relevant queries are handled. Inspect per-query differences as well as the average, since one number can hide failures on important information needs. Choose additional metrics according to the user experience, including early precision, graded usefulness or success at finding a specific answer.

People also ask

Why are precision and recall equal at rank R?

Both divide the same count of relevant retrieved documents by R at that cutoff.

Is R-precision the same as average precision?

No. Average precision incorporates relevant hits at multiple ranks; R-precision evaluates one query-dependent cutoff.

Can it be computed without knowing the relevant set?

Not in its standard form. It needs R and relevance judgments for the inspected results.

Check your understanding

A query has eight judged relevant documents. Five of the top eight results are relevant. What is its R-precision?

Answer: 5/8 = 0.625. The relevant cutoff is eight, even if the interface displays ten results per page.

Related terms

• Precision
• Recall
• Precision at k (precision@k)
• Mean average precision
• Normalized Discounted Cumulative Gain
• precision at k
• average precision
• relevance judgment
• test collection

Learn next

Read precision at k and average precision. Construct two rankings with equal R-precision but different first relevant ranks, then explain which ranking better serves a user who reads only the first result.

• Precision

More in Retrieval and knowledge systems

• Proximity weighting - Proximity weighting is a search-ranking technique that gives weight to how close query terms appear to one another in a document.
• Pseudocounts - Pseudocounts are quantities added to observed event counts when estimating probabilities.
• Query likelihood model - The query likelihood model is an information-retrieval approach that ranks documents by how likely each document's language model would be to generate the...
• Ranked retrieval - Ranked retrieval returns search results in an order intended to place the most useful or relevant items first.
• Rating scale - A Rating scale is a defined set of response options used to record a judgment about an item, experience, or characteristic.
• Retrieval-augmented generation - Retrieval-augmented generation, usually shortened to RAG, is a way of giving an AI model relevant information before it generates an answer.

Sources

Christopher Manning, Prabhakar Raghavan and Hinrich Schutze, Introduction to Information Retrieval, Evaluation of ranked retrieval results: https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-ranked-retrieval-results-1.html

The authors' definition, cutoff explanation and discussion of incomplete relevant sets were read. The two-query calculations and fictional policy-search case are original. No benchmark was run, and no unreported relevance judgments are assumed.

Previous: R-learning Next: R-squared