Is learning to rank only for web search?

No. It can order products, documents, recommendations and other candidates.

What is a pairwise method?

It learns from preferences between pairs of items rather than only isolated labels.

Can a ranking model use neural networks?

Yes. Many model families can be used, provided the training and evaluation suit the ranking task.

AI Glossary / AI and machine learning / Intermediate

Learning to rank

Definition

Learning to rank is the use of machine learning to order items according to relevance, preference or another defined criterion. A ranking model learns from examples so that useful items tend to appear ahead of less useful ones for a particular query or context.

What it means

Search results are a familiar example. A system may retrieve hundreds of documents, but users can inspect only a few. Learning to rank helps decide their order using signals such as query relevance, document properties and observed feedback.

Ranking is different from assigning an independent class to each item. The relative order matters, and errors near the top can be more consequential than errors near the bottom. A model can therefore be evaluated with measures designed for ordered lists.

The training signal needs careful interpretation. Expert relevance judgments can be inconsistent, and clicks reflect what users were shown as well as what they preferred. An item near the top may receive more clicks simply because it is visible. Learning from behavior without accounting for this can reinforce an existing ranking's biases.

How it works

The system represents a query-item pair, or a broader context, using features or learned representations. Training data provide relevance labels, preferences or interaction signals.

Pointwise methods learn a score for each item, pairwise methods learn which of two items should come first, and listwise methods use information about a group or ordered list. These categories describe training formulations, not mutually exclusive product types. At use time, scores or comparisons produce an ordering, often after an initial retrieval stage.

Evaluation may use metrics such as NDCG, mean average precision or reciprocal rank, depending on the task. Data splits should reflect queries, users or time as appropriate. Online evaluation can examine whether the ranking helps users complete their tasks, while monitoring consequences that an offline relevance metric may miss.

Three practical examples

1. An internal search engine ranks policy documents for an employee's query, aiming to place the most relevant version near the top.

2. A shopping site orders candidate products using relevance to the user's request and explicit availability constraints.

3. A support tool ranks suggested help articles, comparing expert judgments with whether users successfully resolve the issue.

Think of it this way

Think of a librarian arranging a short reading list for one question. Several books may be relevant, but the order affects what the reader sees first. The limit is that a ranking model learns from a defined signal; it does not automatically understand all the reasons a librarian might consider an item appropriate.

Worked example

Suppose a toy query has three documents with relevance grades 3, 2 and 0. Using gains of 2 to the relevance power minus 1, the gains are 7, 3 and 0. With discounts of 1 / log2(rank + 1), the ideal order has DCG equal to 7 + 3 / log2(3), approximately 8.893.

If the system puts grade 2 first and grade 3 second, its DCG is 3 + 7 / log2(3), approximately 7.417. Dividing by the ideal gives NDCG of about 0.834.

The calculation shows that swapping two relevant items can matter. The chosen gains, cutoff and relevance labels are part of the evaluation definition, not universal facts about user value.

Illustrative case study

A fictional engineering company trains an internal search ranker from clicks. Frequently opened legacy documents rise to the top, even when newer versions answer the query better.

The team investigates exposure bias and adds explicit document-version handling. It builds a judged evaluation set with realistic queries and checks whether users find the correct current document, rather than counting clicks alone.

The case illustrates that ranking learns from the signal it receives. A convenient behavioral measure can reward familiar or prominently displayed items instead of the result that best serves the user's task.

Common misconceptions and mistakes

The highest click rate proves the best relevance. Position and presentation affect clicks.
Ranking scores are necessarily probabilities. Many scores are meaningful mainly for ordering items.
A good average metric guarantees every query is served well. Rare or difficult queries can remain poor.

Business use

Learning to rank can improve search and recommendation when ordering materially affects user success. Define what a useful result means and identify the stage where ranking will operate.

Use representative relevance data and examine performance across query types. Keep mandatory constraints, such as access permissions and availability, separate from preferences that a score can trade off. Assess task completion and user effort alongside ranking metrics. Maintain awareness that the system's ordering changes the feedback it later collects, creating a loop that requires deliberate evaluation.

Check your understanding

Why might a document's high click count be misleading evidence of relevance?

Answer: The document may have been shown more often or placed higher, so exposure can influence clicks independently of its usefulness.

Related terms

Information retrieval
Relevance
Ranking loss
Recommender System
NDCG
pairwise learning
position bias

Learn next

Read information retrieval and NDCG. Compare two short ranked lists and identify whether the metric's emphasis matches what users actually need near the top.

Information retrieval

Sources

Microsoft Research, LETOR: Learning to Rank for Information Retrieval, benchmark project documentation: https://www.microsoft.com/en-us/research/project/letor-learning-rank-information-retrieval/. Qin, Liu, Xu and Li, LETOR benchmark paper: https://doi.org/10.1007/S10791-009-9123-Y. The relevance calculation and engineering-company case are fictional.

Previous: Learning task Next: Learning vector quantization