Does the PRP tell us how to estimate relevance?

No. It guides ordering once relevance probabilities are available; estimation requires a model and evidence.

Must every document above probability 0.5 be shown?

Only under a particular equal-cost binary decision setup. Ranked-list size and other cost models can imply different decisions.

Does expected relevance require independent documents?

The additive expected-count calculation does not. Other objectives or probability calculations can require additional assumptions.

AI Glossary / Retrieval and knowledge systems / Intermediate

Probability Ranking Principle

Definition

The Probability Ranking Principle (PRP) says that, under its stated relevance and utility assumptions, documents should be ranked by decreasing probability of relevance to the user's information need. It provides a foundation for probabilistic information retrieval. The principle explains what an ideal ordering should optimize; it does not supply the probabilities, guarantee their accuracy, or solve every search objective involving diversity, cost, or interaction.

What it means

A search system must decide which documents to show first when a user may inspect only a few. If each relevant document has the same value and relevance is treated as binary, placing documents with higher relevance probability earlier maximizes the expected number of relevant documents in any fixed-size prefix.
The probabilities are conditional on the available evidence and the user's need, often represented imperfectly by a query. The ranking principle remains useful as a target, while estimating relevance is a separate modeling and evaluation problem.

How it works

Define relevance and the retrieval objective, estimate a relevance probability for each candidate, and sort in descending order. For an additive count objective, replacing a lower-probability selected document with a higher-probability unselected document improves expected relevance yield.
Returning a ranked list differs from deciding a cutoff. Under equal false-positive and false-negative costs, a binary return-or-do-not-return decision uses probability above one half. Unequal error costs change that threshold. If document values or interactions change the objective, a simple probability ordering may no longer be sufficient.

Three practical examples

A document-search team can use estimated relevance probabilities to organize candidates when users inspect a limited result list.
An evaluator can distinguish errors in probability estimation from errors in sorting those estimates.
A retrieval designer can identify when a product objective, such as avoiding redundant answers, requires more than a basic independently valued document ranking.

Think of it this way

Think of choosing which sealed envelopes to inspect when each useful envelope has equal value and inspection opportunities are limited. Opening the envelopes with the highest estimated chance of usefulness first maximizes expected yield under those rules. The analogy changes if envelopes contain duplicate information, have different opening costs, or alter the value of what remains.

Worked example

Consider an original query with four documents whose relevance probabilities are stipulated as A=0.8, B=0.6, C=0.3, and D=0.1. If the user reads two documents and every relevant document contributes one unit of value, selecting A and B gives an expected relevant count of 0.8 + 0.6 = 1.4. Selecting A and C gives 1.1, and selecting B and C gives 0.9.
The expected precision of the fixed two-document set {A,B} is 1.4/2 = 0.7. This is an expectation, not a promise that a particular search will produce 1.4 relevant documents. An actual two-document result contains zero, one, or two relevant documents. Linearity of expectation makes the sum valid without requiring the relevance events for A and B to be statistically independent.
The ordering also matters if the user reads only one item: A provides expected relevance 0.8, compared with 0.6 for B. Sorting all four by decreasing probability therefore gives the best expected count at every fixed prefix size under the stated objective.
Now consider a different decision: whether to return each document at all. Suppose returning a nonrelevant document costs one unit, while missing a relevant document costs four. Returning a document with relevance probability p has expected error cost 1-p. Omitting it has expected cost 4p. Return it when 1-p is less than 4p, or p is greater than 0.2. A, B, and C qualify; D does not. Equal costs would instead give the one-half threshold and return only A and B.
These constant error costs change the cutoff but preserve descending probability order. If document-specific values differ, or if the user only wants one unique answer and several documents repeat it, the objective changes further. The original top-two argument then cannot be reused without examining those new assumptions.

Illustrative case study

Fictional case: a support-search team trains a relevance model and sorts articles by its output. Offline evaluation suggests good first-result relevance, but users complain that the first few results repeat the same answer.
The team recognizes that its product values coverage as well as individual relevance. It retains relevance estimation but evaluates a separate reranking step for redundancy and coverage. It does not claim that the basic PRP was violated; instead, it documents that the deployed objective includes interactions between selected results.

Common misconceptions and mistakes

Do not interpret the principle as saying every search engine should sort any number called probability without checking its meaning or quality. Poor estimates can produce a poor ranking even when the sorting procedure is flawless.
Do not confuse the PRP with the Binary Independence Model. The latter is one particular probabilistic retrieval model with additional assumptions. The fixed-prefix expected-count argument does not require independent relevance events. Also avoid treating relevance probability as a statement that a document is factually correct in every respect; relevance and factual reliability are distinct judgments.

Business use

The PRP helps teams make ranking objectives explicit. It is useful for deciding whether a system is trying to maximize relevant results, minimize weighted errors, or optimize a more complicated user experience.
Evaluate ranking quality on representative queries, and assess calibration when probability values are used for thresholds or expected-cost decisions. Keep cutoff policies separate from ranking policies. If diversity, reading effort, freshness, or document-specific value matters, include those objectives deliberately rather than assuming a relevance-only ordering automatically captures them.

Check your understanding

Why does a four-to-one cost for missed relevant documents lower the return threshold to 0.2?

Answer: Returning costs 1-p in expectation, while omitting costs 4p. Returning has lower expected cost when 1-p is less than 4p, which gives p greater than 0.2.

Related terms

Bayes risk
Binary independence model
Calibration
Precision at k (precision@k)
Probabilistic retrieval
Relevance probability
Ranking function
Precision at k
Retrieval costs
Search diversification

Learn next

Recalculate the return threshold when a false positive costs three units and a false negative costs two. Apply it to A, B, C, and D. Then explain why the best ranked order can remain unchanged even though the selected set differs.

Sources

Primary textbook sources: Manning, Raghavan, and Schuetze, The 1/0 loss case: https://nlp.stanford.edu/IR-book/html/htmledition/the-10-loss-case-1.html ; and The PRP with retrieval costs: https://nlp.stanford.edu/IR-book/html/htmledition/the-prp-with-retrieval-costs-1.html . Probability examples, expected counts, and threshold derivations are original.

Previous: Probability proportional to size Next: Probability simplex