← back to the archiveCover illustration for “BM25: the keyword ranking formula behind search”
TERM6mo ago·Published · Updated ·by Andy Padia

BM25: the keyword ranking formula behind search

In short: BM25 is the classic keyword ranking formula that scores documents by query-term frequency, term rarity and document length, and it still anchors search.

BM25 is a ranking function for keyword search that scores how relevant a document is to a query from three signals: how often the query terms appear in it, how rare those terms are across all documents, and how long the document is. It grew out of probabilistic retrieval research that Robertson and Zaragoza's 2009 review traces to the 1970s and 1980s.

BM25 in plain words

A word that appears often in this document but rarely elsewhere is a strong clue. A long document mentioning it once is weaker evidence than a short one mentioning it three times.

How it works

BM25 combines three ideas. Term frequency with diminishing returns, which the review calls saturation, so the tenth mention adds less than the second. Inverse document frequency, so rare terms count more than common ones. Length normalisation, so long documents do not win just by being long. Two parameters, usually called k1 and b, tune the first and third effects.

Elasticsearch's similarity settings, read in November 2026, list BM25 as the default similarity with k1 at 1.2 and b at 0.75. The 2009 review reports that k1 between 1.2 and 2 and b between 0.5 and 0.8 work reasonably well in many circumstances. Setting b to 0 switches length normalisation off, as Elastic's April 2018 explainer shows.

Example

As an illustration, take the query "GST invoice correction". A short help article mentioning "GST" and "correction" several times outranks a long finance policy that mentions "invoice" thirty times but "GST" once, because "GST" is rarer and the article is focused.

Where you'll meet it

  • Elasticsearch, where it is the default similarity
  • Other full-text search engines and database search features
  • Hybrid search pipelines in RAG
  • Information-retrieval research, where the 2009 review calls it one of the most successful text-retrieval algorithms

What it changes for you

For developers building RAG, BM25 is cheap accuracy to add: no embedding model, no GPU, and strength on exact terms. For search teams, my view is that a tuned BM25 baseline belongs in every comparison before a newer method replaces it.

BM25 vs TF-IDF

TF-IDF also weighs term frequency by rarity. BM25 adds saturation and length normalisation on top.

Common mistake

My view: dropping keyword search when adding embeddings. Codes, names, version numbers and legal clause references are exactly where BM25 shines and embeddings stumble.

Hybrid search · Semantic search · Vector search · Retrieval-augmented generation

From the archive: Why your RAG pipeline returns irrelevant results

Keep keyword search when you add embeddings, because exact codes, names and clause numbers still need it.

#field-guide#search#retrieval#information-retrieval

Sources

  1. staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf
  2. elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables
  3. elastic.co/docs/reference/elasticsearch/index-settings/similarity
← older drop
NIST AI RMF: a framework for managing AI risk
newer drop →
Chunking: splitting documents for retrieval and RAG

related drops

explore all 557 drops →
← back to the archiveday 175