BM25

Overview

BM25 is a ranking function that scores how well a document matches a text query. It counts how often the query words appear in the document, gives more weight to rare words, and stops rewarding repeated words after a point. It also adjusts for document length so that long documents do not win just because they are long. The result is one number per document, and the documents with the highest numbers are shown first.

BM25 has been a default for keyword search for decades, and it is still hard to beat on many tasks. It is fast, needs no training, works on any language that can be split into words, and is easy to explain. Modern retrieval systems rarely drop it. Instead they pair it with embeddings in hybrid search, then pass the shortlist to a cross-encoder reranker. For retrieval-augmented generation (RAG), BM25 is the "keyword half" that catches exact names, codes and rare terms that embeddings often blur.

What it is

BM25 stands for "Best Matching 25". It is a scoring formula, not a model that learns from data. You give it a query, a collection of documents and a few statistics about that collection, and it returns a relevance score for each document.

It belongs to the family of lexical or sparse retrieval methods. "Lexical" means it matches the actual words. "Sparse" means each document is represented as a long vector with one slot per vocabulary word, and almost all slots are zero. Only the words that appear in the document carry a value.

The three ideas inside BM25

• Term frequency with saturation. A word that appears more often in a document is a better sign of relevance, but the first few appearances matter most. Ten mentions are not ten times better than one.
• Inverse document frequency. A word that appears in few documents tells you more than a word that appears in most. The word "the" tells you nothing. A product code tells you a lot.
• Length normalisation. A long document has more chances to contain a word by accident. BM25 scales the score down for documents that are longer than average, and up for shorter ones.

An everyday analogy

Imagine you are looking for a book about "sourdough starter" in a library with no catalogue. You scan the books. A book that says "sourdough" twenty times is likely about sourdough, but a book that says it forty times is not twice as relevant. That is saturation. A book that mentions "starter" is only a small hint, because many books use that word, but "sourdough" is rare and stands out. That is inverse document frequency. A thousand-page encyclopaedia might mention both words once, and that tells you little, while a thin pamphlet that mentions both is more likely to be on topic. That is length normalisation.

Where you meet it

• Search engines and search libraries, where it is the default or a built-in option. See Elasticsearch, Vespa and the Lucene family.
• Vector databases that offer a keyword mode next to vector search, such as Weaviate, Qdrant and Milvus.
• Python packages that compute BM25 in memory for small and medium collections.
• RAG pipelines that need a cheap first-stage retriever.

Where it came from

BM25 comes from the probabilistic retrieval tradition in information retrieval. The core idea is the probability ranking principle: if you rank documents by their estimated probability of being relevant, you get the best possible ordering under that model. Researchers in the 1970s built models of how words are spread over relevant and non-relevant documents.

In the 1990s, a group working on the Okapi retrieval system at a UK university turned these ideas into a practical formula. They added a way to model term frequency that saturates, and a way to account for document length. The formula was tuned on the TREC evaluation campaigns of the early 1990s and quickly became a strong baseline. The number 25 refers to a numbered series of weighting schemes tried along the way, and "Okapi BM25" is the common full name.

Over the years BM25 spread through open source search libraries. In the Lucene library it became the default similarity in the version 6 era, replacing an older TF-IDF style scoring. Variants such as BM25F (for documents with fields like title and body) and BM25+ (which fixes a small issue with long documents) were proposed later.

When neural retrieval rose after the transformer and BERT era, many papers showed that dense models beat BM25 on some benchmarks. Later work showed that BM25 stays very competitive on out-of-domain data, and that dense models can fail on rare words. This is why most production systems now combine both. For more on how model-based retrieval fits with older methods, see research.

How it works

Step 1: Break text into tokens

Both the query and the documents are split into tokens. A basic analyzer lowercases the text, splits on spaces and punctuation, and may remove very common stop words. It may also reduce words to a root form (stemming), so that "running" and "runs" match "run".

This step has a large effect on quality. Two systems using the same BM25 formula but different tokenizers can return different results. For subword schemes used in language models, see SentencePiece, though classic BM25 usually uses whole words.

Step 2: Build an inverted index

An inverted index is like the index at the back of a book. For each word, it lists the documents that contain it, plus how many times the word appears in each. It also stores the length of each document. With this index, the engine never needs to scan every document. It looks up each query word and only touches the documents on those lists.

Step 3: Compute the rarity weight

For each query word, BM25 computes an inverse document frequency (IDF) weight. A common version is:

IDF(q) = ln( 1 + (N - n(q) + 0.5) / (n(q) + 0.5) )

Here N is the number of documents and n(q) is the number of documents that contain the word. Rare words get a large weight. Words that appear in almost every document get a weight near zero. The added 1 inside the logarithm keeps the weight from going negative, which could happen with an older form of the formula.

Step 4: Compute the saturated term frequency

For each word in each document, BM25 turns the raw count f into a bounded number:

tf_part = f * (k1 + 1) / ( f + k1 * (1 - b + b * |D| / avgdl) )

Here |D| is the length of the document and avgdl is the average length over the collection. The two parameters k1 and b control the shape. As f grows, tf_part approaches k1 + 1 and stops growing. That is the saturation.

Step 5: Add up the scores

The final score for a document is the sum over query words:

score(D, Q) = sum over q in Q of IDF(q) * tf_part(q, D)

Words that do not appear in the document add zero. Documents are sorted by score, and the top k are returned.

What the parameters do

• k1 sets how fast term frequency saturates. A small value means a word's first appearance carries almost all the weight. A large value makes the score grow closer to linear with the count. Typical values are between about 1.2 and 2.0.
• b sets how strongly length normalisation applies. With b equal to 0, document length is ignored. With b equal to 1, the score is fully scaled by length relative to the average. A common default is 0.75.

Query word weights

If the same word appears many times in the query, some variants add a saturation step on the query side too. Most search libraries simply treat each query token as a separate clause and sum the scores. For short search queries this makes little difference.

Why it is fast

Because the index is organised by word, a query touches only the lists of its own words. Engines use skip lists, block-max pruning and early termination to avoid scoring documents that cannot reach the top k. This is why BM25 can search millions of documents in milliseconds on modest hardware, long before any neural model is involved.

Variants and extensions

BM25 is a family more than a single fixed formula. Several variants exist, and it helps to know what they change.

• BM25F. Documents often have fields, such as title, headings and body. BM25F first combines the term counts of the fields using field weights, and then applies the saturation step once. This avoids the problem of a word counted separately in each field and over-rewarded.
• BM25+. The basic formula can score a very long document that contains the word lower than a document that does not contain it at all, in extreme cases. BM25+ adds a small constant to the term part so that a match always counts for at least a little.
• BM25L. This variant reduces the penalty on long documents, for collections where long documents are often the relevant ones.
• Query-side saturation. Some versions also saturate the count of a word in the query, which matters when the query is a long paragraph.
• Learned term weights. Newer sparse models keep the same index structure but learn the weight of each word, and may add related words that were not in the text. They still use an inverted index, so they are cheap to serve.

For most projects, the standard formula with default parameters is the right starting point. Switch to a variant only when you can show a measured gain on your own queries.

How the score behaves: a mental model

It helps to picture the term part as a curve. The horizontal axis is the count of a word in a document. The vertical axis is the contribution to the score. The curve rises quickly at first and then flattens near a ceiling. A larger k1 raises the ceiling and flattens the curve more slowly. A larger b lowers the whole curve for documents longer than average and raises it for shorter ones.

Because of this shape, three practical effects follow.

• A document cannot win only by repeating one word many times.
• Matching several different rare query words beats matching one word very often.
• In a field with uniform lengths, such as short titles, the b parameter has little effect, and you can leave it alone.

Common ways to inspect a ranking

When a result looks wrong, do not guess. Ask the engine to explain the score. Most search engines offer an explain mode that shows, for each query word, the IDF, the term count, the length factor and the final contribution. Reading these numbers answers most questions quickly.

• If a good document is missing, check that the query tokens match the indexed tokens. A mismatch in the analyzer is the most common cause.
• If a poor document ranks high, look for a rare word in a short field. Its IDF may be large, and a short field gives a high length boost.
• If two similar documents have very different scores, compare their lengths and the counts of the main words.

Limits of exact statistics

IDF depends on the whole collection. If you add a large batch of documents on a new topic, the weights of old words change. This is normal and usually harmless. In distributed systems, each shard may keep its own document counts. When shards differ in size or topic, the same word can have different weights on different shards, and merged results can look odd. Some engines can gather global statistics before scoring, at some cost in latency.

Multilingual and domain notes

• For languages written without spaces, use a word segmenter or character n-grams. Plain splitting on spaces gives one giant token per sentence.
• For languages with rich word forms, a stemmer or lemmatiser raises recall but may lower precision. Test both.
• For code, logs and part numbers, consider n-gram analyzers or word-delimiter rules so that partial matches work.
• For specialised domains such as law or medicine, a custom synonym list often gives a bigger gain than any parameter change.

Key concepts and terms

A worked example

A tiny collection

Suppose a shop has four product titles and a user searches for "red shoes".

The collection has N = 4 documents and an average length of 4.5 tokens. We use k1 = 1.2 and b = 0.75.

Rarity weights

The word "red" appears in 2 documents. The word "shoes" appears in 3 documents. Using the IDF formula above:

• IDF(red) is about 0.693.
• IDF(shoes) is about 0.357.

"Red" is rarer, so it counts for more. If you search for a word that appears in only one document, its weight would be larger still.

Document scores

For document 0, both words appear once and the length is 5. The length factor is 1 - 0.75 + 0.75 * 5 / 4.5, which is about 1.083. So tf_part is 1 * 2.2 / (1 + 1.2 * 1.083), about 0.957. The score is 0.693 * 0.957 + 0.357 * 0.957, about 1.004.

For document 3, only "red" appears, but the document is short (length 2). Its tf_part is about 1.294, so the score is 0.693 * 1.294, about 0.897.

The final ranking:

Two lessons show up. First, a short document that matches the rare word can beat a longer document that matches the common word twice. Second, repeating "shoes" in document 2 did not help much, because of saturation and because the document is long.

The same idea in Python

import math
from collections import Counter

docs = [
    "red running shoes for men",
    "blue running shoes",
    "leather office shoes for men and women shoes",
    "red dress",
]
tokens = [d.split() for d in docs]
N = len(tokens)
avgdl = sum(len(t) for t in tokens) / N
k1, b = 1.2, 0.75

df = Counter()
for t in tokens:
    for w in set(t):
        df[w] += 1

def idf(w):
    n = df.get(w, 0)
    return math.log(1 + (N - n + 0.5) / (n + 0.5))

def bm25(query, doc):
    score = 0.0
    counts = Counter(doc)
    for w in query.split():
        f = counts.get(w, 0)
        if f == 0:
            continue
        denom = f + k1 * (1 - b + b * len(doc) / avgdl)
        score += idf(w) * f * (k1 + 1) / denom
    return score

ranked = sorted(range(N), key=lambda i: bm25("red shoes", tokens[i]), reverse=True)
print(ranked)

For real collections, use a search engine or a library that builds the inverted index, rather than looping over all documents.

The same idea in a search engine

Most engines let you set k1 and b per field. A configuration might look like this (placeholder names):

index: products
similarity:
  default:
    type: BM25
    k1: 1.2
    b: 0.75
fields:
  title:
    boost: 2.0
  description:
    boost: 1.0

The title field gets a higher boost, because a match in a title is usually a stronger signal than a match in a long description.

A second example: what changing k1 and b does

Take document 2 from the table, the long title where "shoes" appears twice. With k1 = 1.2 and b = 0.75, its term part for "shoes" is about 1.13. Now try other settings on the same document, which has 8 tokens against an average of 4.5.

The values are approximate, and the point is the direction of each change. Notice how small the shifts are for one word. Differences add up across many words and many documents, which is why parameters should be tuned on a query set and not on a single example.

A tiny tuning loop

best = None
for k1 in (0.9, 1.2, 1.5, 2.0):
    for b in (0.4, 0.6, 0.75, 0.9):
        score = evaluate_recall_at_10(k1=k1, b=b, queries=dev_queries)
        if best is None or score > best[0]:
            best = (score, k1, b)
print(best)

Hold out a second set of queries to confirm that the winner still wins there. A grid this small is enough for most projects, and the result is often close to the defaults.

Strengths and limitations

Strengths

• No training needed. You can start with a new collection on day one. There is no labelled data and no GPU.
• Fast and cheap. Inverted indexes scale to very large collections with low latency.
• Exact matching. It is excellent for names, identifiers, error codes, part numbers and rare technical terms.
• Transparent. You can explain exactly why a document ranked where it did, word by word.
• Strong out of domain. It does not depend on a training set, so it does not fail when the data looks new.
• Easy to update. Adding a document means updating the index. No re-training or re-embedding is required.
• Good baseline. A new retrieval idea should beat BM25 before it is taken seriously.

Limitations

• Vocabulary mismatch. If the query says "car" and the document says "automobile", BM25 sees no match. Synonyms and paraphrases are invisible without extra work.
• No understanding of word order or meaning. "Dog bites man" and "man bites dog" look the same to a bag-of-words model.
• Sensitive to tokenization. Languages without spaces, heavy compounding or rich morphology need careful analyzers.
• Weak on long natural-language questions. Many words in a long query add noise, and the key idea may be hidden.
• Parameters matter. The defaults are good, but fields with unusual length patterns may need tuning.
• Weaker on short, repetitive text. If every document is a short label, term frequency carries little information.
• No personalisation or context. The score depends only on words and counts.

How it compares with alternatives

Choosing between them

• If you have no labelled data and need something working today, start with BM25.
• If users ask in natural language and documents use different words, add dense retrieval.
• If precision in the top few results matters, add a reranker.
• If you must find exact strings, keep BM25 in the pipeline even when you add embeddings.

Related index structures

Dense retrieval needs approximate nearest neighbour indexes such as HNSW, FAISS or IVF-PQ. BM25 does not need these. Its inverted index is simpler, smaller and exact. Vector stores such as pgvector and Pinecone are often paired with a keyword index for this reason.

Role in tabular foundation models

BM25 is a text tool, and a tabular foundation model reads tables. Still, the two meet in several concrete places, and it helps to see them separately.

What a tabular foundation model does

A tabular foundation model is pre-trained on many tables, often synthetic ones, so that it can make predictions on a new table without being trained on it. It sees rows and columns, handles mixed types (numbers, categories, dates, text), copes with missing values, and does not need column names to be meaningful. It learns from a few labelled rows given as context, a form of in-context learning. Models like TabPFN, TabICL and TabSTAR follow this style. The architecture page covers how such a model is built.

Where BM25 fits next to it

• Choosing context rows. In-context learning has a limited context size. When a table has millions of rows, you must pick which labelled rows to show the model. Row serialisation to text plus BM25 gives a cheap way to find rows that share rare values with the query row, such as the same product code or the same customer segment. A tabular model then reasons over those neighbours. This is a retrieval-style approach to context selection, and BM25 is a fast, training-free option.
• Text columns inside tables. Many real tables contain free text: product titles, notes, job descriptions, error messages. BM25 can index those columns for lookup, deduplication and candidate generation, while the foundation model handles the numeric and categorical parts.
• Sector and semantic classification. When a row must be mapped to a category in a large taxonomy, BM25 over the category descriptions gives a shortlist of candidates. A model then picks the best one. This keeps cost low and gives exact support for rare labels. See also semantic sector heads.
• Entity matching and deduplication. Matching two records that describe the same thing often starts with a lexical candidate step. BM25 on names and addresses finds likely pairs, and a stronger model scores them. See deduplication.
• Imputation support. To fill a missing text value, such as a missing category, you can retrieve similar rows by their other text fields and let the model propose a value. This sits next to imputation networks and masked-cell approaches like masked cell modeling.
• Anomaly detection. Rare tokens in a text column can be a signal. A row with a code that appears nowhere else gets a high IDF weight, which makes IDF a simple feature for flagging unusual records. See anomaly detection.
• RAG over table documentation. A question-answering assistant for a data team may retrieve column descriptions, data dictionaries and schema notes. BM25 is strong here because column names, table names and codes are exact strings.

What BM25 does not do

BM25 does not understand numeric structure. It cannot tell that 99 is close to 100, that a date falls between two others, or that a column is skewed. It does not learn from labels. It does not use column types. These are the things tabular foundation models are built for. So the clean split is: BM25 narrows and matches text, and the tabular model reasons over structure and values.

A simple combined pattern

1. Serialise each historical row into a short text, such as "plan basic region north status active".
2. Index those texts with BM25 and store the label next to each.
3. For a new row, retrieve the top k similar rows.
4. Give those rows and their labels to a tabular foundation model as context, and ask for a prediction on the new row.
5. Evaluate against a plain random choice of context rows to confirm that retrieval helps.

For measured results of tabular models, see benchmarks. For how tabular models relate to the wider research field, see research.

Practical guidance and best practices

Start with sensible text analysis

• Use a language-specific analyzer where one exists.
• Lowercase text, and consider folding accents if users type without them.
• Keep stop word removal modest. For short queries, removing a word can change the meaning.
• Use stemming or lemmatisation if your users search in a language with many word forms.
• Index identifiers in a field with a minimal analyzer so that "AB-1234" is kept intact.

Tune k1 and b with evidence

• Build a small set of real queries with known good documents.
• Measure a ranking metric such as recall at 10 or nDCG at 10.
• Change one parameter at a time and measure again.
• Check b first when your documents vary widely in length. Check k1 when repeated words matter.
• Do not tune on a handful of queries. Small sets mislead.

Use fields wisely

• Give titles, headings and tags a higher boost than body text.
• Use a combined-field approach (BM25F style) when your engine supports it, rather than summing separate scores blindly.
• Keep very short fields such as codes in their own field with their own settings.

Combine with other signals

• Add a dense retriever and merge the lists. Reciprocal rank fusion is a simple, robust merge that needs no score calibration. See hybrid search.
• Rerank the top 50 to 200 with a cross-encoder for better precision.
• Rewrite vague user questions before searching. See query rewriting and HyDE.
• Use learning to rank when you have click or label data and want to blend many features, with BM25 as one feature.

Chunk documents for RAG

• Index passages, not whole files. A passage of a few hundred words fits the way BM25 treats length.
• Overlap chunks a little so that a key sentence is not cut in half.
• Store the document title with each chunk so that context is not lost. See contextual retrieval.

Evaluate the whole pipeline

• Measure retrieval on its own with recall at k.
• Measure the final answers with a RAG evaluation tool such as RAGAS, TruLens or DeepEval.
• Watch for regressions when you change analyzers, because a small tokenizer change can shift results for many queries. See regression boards.

Operations

• Rebuild or merge index segments regularly so that statistics stay accurate.
• Remember that IDF depends on the collection. In a sharded index, each shard may compute its own statistics unless the engine shares them.
• Monitor query latency and the share of queries with zero results.

Common mistakes

• Treating BM25 as obsolete. It is a strong component in most real systems, and many dense-only systems regress on exact terms.
• Skipping text analysis. Using a default analyzer on a language it was not built for hurts results more than any parameter.
• Over-tuning parameters on few queries. The gain is small and may not hold on new queries.
• Ignoring document length. Mixing very long and very short documents in one field can bury short but good matches, or the reverse. Chunk to similar sizes.
• Removing too many stop words. Phrases like "to be or not to be" or product names with common words can vanish.
• Merging scores from different systems directly. BM25 scores and cosine similarities live on different scales. Use rank-based fusion or normalise carefully.
• Forgetting identifiers. Splitting "AB-1234" into "ab" and "1234" can match the wrong items.
• Judging only by a few examples. Always test on a representative query set.
• Assuming BM25 handles meaning. It does not. Add semantic retrieval if users paraphrase.
• Not updating statistics. An old index with stale document counts will weight words incorrectly after large changes to the data.
• Using BM25 on text that is not text. Numbers, dates and categorical codes in a table need structured filters or a tabular model, not word matching.

Frequently asked questions

What does BM25 stand for?

BM25 stands for "Best Matching 25". It is the name of a ranking function from the Okapi family of probabilistic retrieval models, where the number 25 marks one of a numbered series of weighting schemes tried by the original researchers. The name stuck, and today people say BM25 or Okapi BM25 for the same formula.

Is BM25 a machine learning model?

Not in the usual sense. BM25 is a formula with two tunable parameters, and it uses simple statistics from the collection such as word counts and document lengths. It does not learn weights from labelled data. You can tune k1 and b on examples, but there is no training loop, no gradient and no neural network involved.

What are the best values for k1 and b?

There is no single best value, but k1 between about 1.2 and 2.0 and b near 0.75 work well on many collections, which is why they are common defaults. If your documents differ a lot in length, test a few values of b. If repeated words matter more or less than usual, test k1. Always measure with a set of real queries before changing defaults.

How is BM25 different from TF-IDF?

Both reward words that appear often in a document and are rare in the collection. BM25 improves on plain TF-IDF in two ways. It saturates term frequency so that many repeats stop adding score, and it normalises for document length with a tunable parameter. These changes make rankings more stable and usually more accurate.

Does BM25 still matter now that we have embeddings?

Yes. Embedding models capture meaning but often miss exact strings such as names, codes and rare terms. BM25 captures those well, costs very little, and needs no training. Many strong retrieval systems run both and merge the results, then rerank. BM25 is also a reliable baseline when you test whether a new dense model really helps.

Can BM25 handle synonyms?

Not by itself. It only matches the tokens it sees. You can add synonym lists in the analyzer, expand queries with a rewriting step, or use a dense or learned sparse model to cover meaning. Another option is to combine BM25 with an embedding retriever in a hybrid setup so that each covers the other's gaps.

Can I use BM25 on tables or structured data?

You can use it on the text columns of a table, such as titles, notes and descriptions. It cannot reason about numbers, dates or category structure, so numeric and categorical parts should be handled with filters or with a model built for tables. A common pattern is to serialise rows to text for candidate retrieval, then let a tabular model make the final prediction.

How does BM25 work with a reranker?

BM25 is fast, so it can scan the whole collection and return a shortlist, for example the top 100. A cross-encoder reranker then reads the query and each shortlisted document together and gives a more accurate score. This two-stage design gives you the speed of BM25 and the precision of a heavier model.

Is BM25 good for RAG?

It is a good first-stage retriever for RAG, especially when questions contain specific names, identifiers or technical terms. It tends to be less helpful when questions are long and conversational, or when the documents use different words than the question. In those cases, add dense retrieval and a reranker, and consider rewriting the question before search.

How do I evaluate a BM25 setup?

Build a set of queries with known relevant documents, then measure recall at k and nDCG at k. Compare variants of the analyzer, parameters and field boosts on the same set. For a RAG system, also measure answer quality with an evaluation tool, so you can see whether better retrieval leads to better answers.

Glossary

• Analyzer: The part of a search engine that turns raw text into tokens.
• Average document length: The mean number of tokens per document in the collection.
• b: The BM25 parameter for length normalisation.
• BM25F: A BM25 variant that combines several fields with weights.
• Bag of words: A view of text that keeps word counts but ignores order.
• Boost: A multiplier that raises the importance of a field or term.
• Candidate generation: The first stage that finds a shortlist of possible results.
• Dense retrieval: Retrieval using vectors that capture meaning.
• Document frequency: The number of documents that contain a given word.
• Hybrid search: A combination of lexical and semantic retrieval.
• Inverted index: A map from words to the documents that contain them.
• IDF: Inverse document frequency, a weight that favours rare words.
• k1: The BM25 parameter for term frequency saturation.
• Lexical retrieval: Retrieval based on matching words.
• nDCG: A ranking metric that rewards putting relevant results near the top.
• Posting list: The list of documents for one word in an inverted index.
• Recall at k: The share of relevant items found in the top k results.
• Reciprocal rank fusion: A method to merge ranked lists using only ranks.
• Saturation: The effect that extra repeats of a word add less and less score.
• Sparse vector: A vector with mostly zero values, one slot per word.
• Stemming: Reducing a word to its root form.
• Stop word: A very common word that is often dropped.
• Term frequency: The number of times a word appears in a document.
• Tokenization: The process of splitting text into tokens.

Summary

BM25 is a simple, fast and well tested ranking function for keyword search. It combines three ideas: rare words matter more, repeated words help with diminishing returns, and long documents are scaled down. With two parameters, k1 and b, it gives strong results on many collections without any training.

It does not understand meaning, so it works best next to dense retrieval and a reranker, as in hybrid search and cross-encoder reranking. For tabular foundation models, it is a helpful helper for text columns, context row selection, candidate generation and documentation search, while the model itself handles numbers, categories and missing values. To learn more about how such models are built and measured, see research, architecture and benchmarks.

Related

It belongs to RAG in my stack, under LLMs, retrieval, and evaluation. Others in the same group: