BM25

Definition

BM25 is a probabilistic information retrieval ranking function used to score documents against a query based on term occurrence, term rarity and document-length normalisation.

Why it matters

BM25 remains a widely used lexical ranking method because it provides an effective balance between term frequency and specificity. It is commonly used as a baseline or lexical component in modern hybrid search systems.

How it works

Term matching

  • BM25 evaluates query terms that occur in a document or passage.
  • Terms that are rare across the corpus can contribute more to the score than very common terms.

Frequency saturation

  • Additional occurrences of the same term increase the score with diminishing returns rather than indefinitely increasing relevance.

Length normalisation

  • Document length is accounted for so long documents are not automatically favoured simply because they contain more terms.

How Netsleek uses the term

Netsleek uses BM25 as the canonical example of lexical scoring within retrieval architecture. It is useful for explaining how exact terminology can contribute to candidate generation even when an AI system also uses embeddings and semantic retrieval.

Comparisons

  • BM25 vs Lexical Retrieval: Lexical Retrieval is the retrieval family. BM25 is one ranking function used within that family.
  • BM25 vs Cosine Similarity: BM25 scores lexical term evidence. Cosine similarity compares vector direction and is commonly used with embeddings.
  • BM25 vs Reranking: BM25 can generate or rank initial candidates. Reranking applies a later scoring stage to an already retrieved set.

How BM25 Connects Across AI Systems

BM25 does not operate in isolation. It connects to other established concepts across the AI search and retrieval process:

  • Uses inverted indexes: BM25 commonly relies on term statistics stored in an Inverted Index.
  • Supports candidate generation: BM25 can produce an initial ranked candidate set.
  • Can be combined with vectors: Hybrid Search can combine BM25-based lexical scores with dense retrieval.
  • Can feed RRF: Ranked lists from BM25 and vector retrieval can be fused using Reciprocal Rank Fusion.

Related glossary concepts

Common misinterpretations

  • BM25 is not an AI model.
  • BM25 does not understand meaning in the same way as dense semantic models.
  • A high BM25 score does not guarantee that a document will be selected in later AI stages.

Summary

BM25 is an established lexical ranking function that scores query-document relevance using term frequency, rarity and length normalisation. It remains widely useful in conventional and hybrid retrieval systems.