Does BM25 need labeled training examples?

The basic formula can rank from collection statistics without supervised relevance labels, although labels can help tune and evaluate a system.

Does it automatically match synonyms?

Basic term matching does not. Query processing or additional retrieval components may supply that ability.

Can BM25 and vector search be combined?

Yes. Hybrid systems can combine candidate sets or rankings, but their combination rule and benefits need evaluation.

AI Glossary / Retrieval and knowledge systems / Intermediate

BM25

Definition

BM25 is a ranking method for text retrieval that scores documents using query-term matches, term rarity, term frequency and document length. It is a lexical method: its basic scoring depends on indexed terms rather than learned semantic similarity between dense vectors. Higher scores indicate a stronger match under the chosen BM25 formulation.

What it means

A document should usually gain credit for containing a query term, especially when that term is uncommon in the collection. Repeating the same word many times should not increase its importance indefinitely. Longer documents also have more opportunities to contain query words simply because they contain more text.

BM25 combines these considerations in a compact scoring rule. It does not understand every synonym or prove that a document answers the question. Search quality also depends on how text is tokenized, indexed and queried.

How it works

For each matching query term, calculate an inverse-document-frequency weight and a term-frequency contribution adjusted for document length. Sum the contributions across query terms. A parameter commonly called k1 controls frequency saturation, while b controls the strength of length normalization.

One common term contribution is IDF × f(k1 + 1)/(f + k1(1 - b + bL/averageL)), where f is the term count and L is document length. Implementations differ in IDF conventions, field treatment and details of length measurement. Scores are ranking quantities, not calibrated probabilities, and should not be assumed comparable across unrelated queries or collections.

Three practical examples

1. A help-center search ranks articles containing the user's product identifier and issue terms, giving useful weight to uncommon identifiers.

2. A retrieval pipeline uses BM25 to gather candidate passages before a separate reranker or language model processes them.

3. A team compares lexical retrieval with an embedding-based method on its own questions. BM25 may retrieve precise names or codes that a semantic method handles differently, while synonym-heavy queries may expose lexical limitations.

Think of it this way

Think of a librarian balancing three clues: which requested words appear, how distinctive those words are, and whether they occupy a short focused note or a long document. Repetition adds evidence, but with diminishing returns. BM25 formalizes that kind of weighting without turning the resulting score into a guarantee of relevance.

Worked example

Use a one-term query and a deliberately specified BM25 variant. There are N = 100 documents, and the term appears in n = 10. Define IDF = log(1 + (N - n + 0.5)/(n + 0.5)), approximately 2.2637. Let k1 = 1.2, b = 0.75 and average length = 100 tokens.

Document A has length 100 and term frequency 2. Its frequency factor is 2 × 2.2/(2 + 1.2) = 1.375, giving score approximately 3.1126.

Document B also contains the term twice but has length 200. Its adjusted denominator is 2 + 1.2 × (0.25 + 0.75 × 2) = 4.1. Its factor is 4.4/4.1, approximately 1.0732, giving score approximately 2.4294.

A ranks above B under these assumptions. The example isolates length normalization; it does not establish that shorter documents are always more relevant. Additional query terms, fields and document contents can change a real ranking.

Illustrative case study

Fictional case: Alder Search builds an internal equipment-document search. Its evaluation set includes exact component codes as well as questions using everyday vocabulary. BM25 retrieves many code-bearing pages, but some vocabulary-mismatched questions return poor results.

The team examines tokenization, approved synonym handling and a possible hybrid retrieval stage. It uses separate evaluation queries to compare changes and checks whether the retrieved passage actually supports an answer. This invented case describes an assessment process, not a measured improvement, a universal recommendation or evidence that one retrieval approach wins on every query type.

Common misconceptions and mistakes

BM25 is not a language model that generates answers, and its score is not a probability that an answer is correct. Lexical matching can retrieve documents that mention a term while denying, criticizing or discussing something unrelated to the user's intent.

Another mistake is assuming that term frequency grows linearly in importance; saturation limits its contribution. Length normalization is configurable, not a rule that every short document must outrank every long one. Finally, default parameters and benchmark results do not replace evaluation on the actual collection, language and query distribution.

Business use

BM25 provides a transparent retrieval baseline and can be a useful component of document search or retrieval-augmented generation. It helps teams inspect which terms contributed to a result and compare a more complex system against a well-defined lexical approach.

Measure retrieval relevance, coverage, latency and downstream answer support separately. Keep access filtering and document freshness in the retrieval design. Tune parameters and preprocessing using development data, then evaluate on held-out queries. A better retrieval score on a training set does not automatically translate into better answers for users.

Check your understanding

Why does document A score higher than B when both contain the query term twice?

Answer: Under the specified formula, B is twice the average length, increasing its normalization denominator. The same two occurrences therefore produce a smaller frequency contribution. This is a property of the chosen scoring rule, not proof of actual relevance.

Related terms

Inverted index
Relevance
Retrieval-augmented generation
Lexical retrieval
inverse document frequency
term frequency
document length normalisation
hybrid search
reranking

Learn next

Understand term frequency and document frequency, then calculate BM25 contributions for a small collection. Inspect your implementation's exact formula and text processing. Compare retrieved results against relevance judgments, paying particular attention to rare identifiers, synonyms and long documents.

Relevance

Sources

Manning, Raghavan and Schutze, Introduction to Information Retrieval, Okapi BM25: https://nlp.stanford.edu/IR-book/html/htmledition/okapi-bm25-a-non-binary-model-1.html

Apache Lucene, BM25Similarity documentation, including its IDF convention: https://lucene.apache.org/core/9_12_1/core/org/apache/lucene/search/similarities/BM25Similarity.html

The numerical ranking and Alder Search case are original hypothetical illustrations. The formula convention is stated explicitly.

Previous: Blocking Next: Boid