Sparse vs Dense Retrieval: BM25 and Its Place in Modern RAG

Why This Debate Refuses to Die

Every few months, someone on your team asks whether you should "just use embeddings" for search, and every few months a production incident reminds everyone why BM25 is still running quietly in the background of half the world's search engines. If you're building a Retrieval-Augmented Generation (RAG) system, this question isn't academic. It decides whether your chatbot correctly finds the invoice with the exact string INV-2024-0091, or whether it confidently retrieves five semantically related but wrong invoices instead.

Sparse retrieval and dense retrieval are not two versions of the same idea competing for a single winner. They're solving different problems that happen to look similar from the outside: "given a query, find relevant documents." Sparse methods like BM25 count and weight words. Dense methods use neural embeddings to compare meaning in vector space. Modern, serious RAG systems tend to use both, not because it's trendy, but because the failure modes of each method are almost perfectly complementary.

This article walks through how BM25 actually works, how dense retrieval differs mechanically, where each one wins and loses in practice, and how to combine them in a real RAG pipeline without over-engineering your first version.

What Sparse Retrieval Actually Means

"Sparse" refers to the representation, not the amount of data. In sparse retrieval, a document is represented as a vector over the entire vocabulary of your corpus, where almost every entry is zero except for the words that actually appear in that document. If your vocabulary has 50,000 unique terms and a document uses 200 distinct words, you get a vector with 200 non-zero entries and 49,800 zeros — hence "sparse."

The classic sparse retrieval algorithm is TF-IDF (term frequency–inverse document frequency), and its most successful descendant is BM25 (Best Matching 25), which is the default ranking function in Elasticsearch, OpenSearch, and most traditional search engines. BM25 improves on raw TF-IDF in two important ways:

• It saturates term frequency. Mentioning "database" 50 times in a document doesn't make it 50 times more relevant than mentioning it once — BM25's scoring curve flattens out, so extra repetitions matter less and less.
• It normalizes for document length. A 200-word article that mentions "kubernetes" three times is treated very differently from a 20,000-word manual that mentions it three times.

The BM25 formula for a document D and query terms q1...qn is:

score(D, Q) = sum over qi in Q of:
    IDF(qi) * ( f(qi, D) * (k1 + 1) ) /
    ( f(qi, D) + k1 * (1 - b + b * (|D| / avgdl)) )

Where f(qi, D) is how often the term appears in the document, |D| is the document length, avgdl is the average document length across the corpus, and k1 and b are tuning constants (commonly k1 ≈ 1.2 to 2.0, b ≈ 0.75). IDF(qi) down-weights terms that appear in almost every document ("the," "is," "system") and up-weights rare, distinguishing terms.

The key intuition: BM25 rewards exact term overlap between query and document, adjusted for how "surprising" that term is and how long the document is. It has no idea that "car" and "automobile" mean the same thing. It just counts.

A Minimal BM25 Implementation

It helps to actually see BM25 computed, rather than just read the formula. Here's a compact, dependency-light Python implementation you can run against a handful of documents to build intuition:

import math
from collections import Counter

class BM25:
    def __init__(self, documents, k1=1.5, b=0.75):
        self.k1 = k1
        self.b = b
        self.docs = [doc.lower().split() for doc in documents]
        self.doc_lens = [len(doc) for doc in self.docs]
        self.avgdl = sum(self.doc_lens) / len(self.docs)
        self.term_freqs = [Counter(doc) for doc in self.docs]
        self.doc_count = len(self.docs)
        self.idf = self._compute_idf()

    def _compute_idf(self):
        df = Counter()
        for doc in self.docs:
            for term in set(doc):
                df[term] += 1
        idf = {}
        for term, freq in df.items():
            idf[term] = math.log(
                (self.doc_count - freq + 0.5) / (freq + 0.5) + 1
            )
        return idf

    def score(self, query, doc_index):
        query_terms = query.lower().split()
        tf = self.term_freqs[doc_index]
        doc_len = self.doc_lens[doc_index]
        total = 0.0
        for term in query_terms:
            if term not in tf:
                continue
            freq = tf[term]
            idf = self.idf.get(term, 0)
            numerator = freq * (self.k1 + 1)
            denominator = freq + self.k1 * (
                1 - self.b + self.b * (doc_len / self.avgdl)
            )
            total += idf * (numerator / denominator)
        return total

    def rank(self, query):
        scores = [
            (i, self.score(query, i)) for i in range(self.doc_count)
        ]
        return sorted(scores, key=lambda x: x[1], reverse=True)


documents = [
    "Kubernetes deployment failed due to insufficient memory limits",
    "How to bake sourdough bread with a wet dough starter",
    "Setting resource limits and requests in a Kubernetes pod spec",
    "Memory leaks in Node.js can be traced with heap snapshots",
]

bm25 = BM25(documents)
results = bm25.rank("kubernetes memory limits")
for idx, score in results:
    print(f"{score:.3f}  {documents[idx]}")

Run this and you'll see documents 0 and 2 dominate because they share exact tokens with the query ("kubernetes," "memory," "limits"), while document 3 gets a small partial score for "memory," and the bread recipe scores near zero. This is exactly the behavior you want for keyword-heavy queries — and exactly where it breaks down for meaning-heavy queries, which brings us to dense retrieval.

What Dense Retrieval Actually Means

Dense retrieval represents documents and queries as fixed-length vectors — typically 384, 768, or 1536 dimensions — where almost every dimension has a non-zero value. These vectors come from a neural network (an embedding model) trained so that semantically similar texts land close together in vector space, measured by cosine similarity or dot product.

The pipeline looks like this:

1. Chunk your documents into passages (usually a few hundred tokens each).
2. Pass each chunk through an embedding model to get a vector.
3. Store the vectors in a vector index (HNSW, IVF, or similar) inside a vector database.
4. At query time, embed the query with the same model and find the nearest vectors.

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")

documents = [
    "Kubernetes deployment failed due to insufficient memory limits",
    "How to bake sourdough bread with a wet dough starter",
    "Setting resource limits and requests in a Kubernetes pod spec",
    "Memory leaks in Node.js can be traced with heap snapshots",
]

doc_embeddings = model.encode(documents, normalize_embeddings=True)

def search(query, top_k=4):
    query_embedding = model.encode([query], normalize_embeddings=True)[0]
    scores = doc_embeddings @ query_embedding
    ranked = np.argsort(scores)[::-1][:top_k]
    return [(documents[i], float(scores[i])) for i in ranked]

for doc, score in search("why is my pod running out of RAM"):
    print(f"{score:.3f}  {doc}")

Notice the query here: "why is my pod running out of RAM" shares almost no exact vocabulary with "Kubernetes deployment failed due to insufficient memory limits" — no "kubernetes," no "memory," no "limits." A pure BM25 search would likely miss this connection or rank it low. But a good embedding model has learned that "pod," "RAM," and "running out" are semantically tied to Kubernetes memory limits, and it will surface that document near the top. This is the entire value proposition of dense retrieval: it retrieves based on meaning, not surface tokens.

Where BM25 Wins and Dense Retrieval Loses

It's tempting to treat dense retrieval as a strict upgrade over BM25, since it "understands meaning." In practice, dense retrieval loses badly in several common scenarios:

• Exact identifiers and codes. Product SKUs, error codes like ECONNREFUSED, ticket numbers, part numbers, legal citations, and API method names are exactly the kind of tokens embedding models were never trained to distinguish precisely. Two different error codes that differ by one character can embed nearly identically, while BM25 treats them as completely different tokens.
• Rare or out-of-domain vocabulary. If your corpus is full of internal jargon, acronyms, or a specialized domain (say, pharmaceutical compound names or legacy COBOL variable names) that the embedding model never saw much of during training, the embeddings for those terms can be poorly separated. BM25 doesn't care what the token means — it just needs it to appear.
• Negation and precise constraints. "Contracts that do NOT include an indemnification clause" is something embeddings frequently mishandle, because the embedding for that sentence still sits close to embeddings about contracts that DO include indemnification clauses. Term-based matching combined with structured filters handles this more reliably.
• Low-resource languages or code search. Many open-source embedding models are trained overwhelmingly on English natural language. Searching source code, config files, or less-represented languages often still favors lexical matching.
• Cold-start with no training data. BM25 requires zero training, zero fine-tuning, and zero GPU. You can stand up a working sparse retriever with pip install rank_bm25 in five minutes. Dense retrieval quality depends heavily on choosing (or fine-tuning) a good embedding model for your domain, which is a nontrivial undertaking.

Where Dense Retrieval Wins and BM25 Loses

The complementary set of dense retrieval's advantages is just as real:

• Paraphrase and synonym handling. "How do I cancel my subscription" and "steps to terminate my membership plan" share almost no vocabulary but mean nearly the same thing. Dense retrieval handles this natively; BM25 needs a synonym dictionary or query expansion layered on top.
• Cross-lingual retrieval. Multilingual embedding models can match a French query against an English document because the meaning representations align across languages, something no amount of TF-IDF tuning will achieve.
• Conceptual, vague, or exploratory queries. "Ways to make my Python code run faster" needs to match documents about caching, algorithmic complexity, vectorization, and multiprocessing — none of which necessarily share the words "make," "run," or "faster."
• Robustness to phrasing noise. Typos, informal phrasing, and voice-transcribed queries degrade BM25 sharply (a missing letter is a missing token) but degrade dense retrieval much more gracefully, since embeddings are somewhat tolerant of minor surface variation.

Hybrid Retrieval: Getting Both

Given these complementary failure modes, most production RAG systems in 2024 and beyond don't pick one — they combine sparse and dense scores into a single ranked list, commonly called hybrid search. There are two popular ways to combine them.

1. Weighted score fusion. Normalize both BM25 and cosine similarity scores to a common scale (e.g., min-max normalization) and combine them linearly:

def normalize(scores):
    lo, hi = min(scores), max(scores)
    if hi == lo:
        return [0.0 for _ in scores]
    return [(s - lo) / (hi - lo) for s in scores]

def hybrid_score(bm25_scores, dense_scores, alpha=0.5):
    norm_bm25 = normalize(bm25_scores)
    norm_dense = normalize(dense_scores)
    return [
        alpha * d + (1 - alpha) * b
        for b, d in zip(norm_bm25, norm_dense)
    ]

The alpha parameter controls how much weight you give to dense vs. sparse. Many teams start at 0.5 and tune it against a labeled evaluation set.

2. Reciprocal Rank Fusion (RRF). Instead of normalizing raw scores (which can be unstable across query types), RRF combines rankings using only each document's rank position:

def reciprocal_rank_fusion(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

bm25_ranking = ["doc_2", "doc_0", "doc_3", "doc_1"]
dense_ranking = ["doc_0", "doc_3", "doc_2", "doc_1"]

fused = reciprocal_rank_fusion([bm25_ranking, dense_ranking])
for doc_id, score in fused:
    print(doc_id, round(score, 4))

RRF is popular precisely because it sidesteps the awkward problem of normalizing two scores that live on entirely different scales (BM25 scores are unbounded and corpus-dependent; cosine similarity is bounded between -1 and 1). It's simple, robust, and used internally by systems like Elasticsearch's hybrid search and OpenSearch's neural search plugin.

In practice, a typical hybrid retrieval pipeline for RAG looks like this:

1. Run BM25 over the full corpus, get the top 50-100 candidates.
2. Run dense vector search over the full corpus (or the same candidate pool), get the top 50-100 candidates.
3. Fuse the two rankings with RRF or weighted fusion.
4. Optionally, run a cross-encoder reranker over the fused top 20-30 to get a final, more expensive but more accurate ordering.
5. Pass the top 3-8 chunks into the LLM's context window.

Practical Guidance: Picking a Starting Point

If you're building a RAG system today and want a pragmatic default rather than a research project, here's what tends to work:

• If your corpus is small (a few hundred documents) and mostly narrative text — support docs, blog posts, policy documents — start with dense retrieval alone using a solid open embedding model, and add BM25 later if you notice exact-match failures (product codes, ticket IDs, names).
• If your corpus is heavy on structured or technical content — API references, log files, legal contracts, medical records — start with hybrid search from day one. The exact-match failures will show up immediately and be embarrassing in front of users.
• Always keep a small evaluation set of 30-50 realistic queries with known correct answers. Run both BM25-only, dense-only, and hybrid against it and measure recall@k. This turns "which is better" from a philosophical debate into a five-minute experiment.
• Don't skip chunking strategy just because you added hybrid search. Bad chunking (too large, too small, or splitting mid-sentence) will hurt both sparse and dense retrieval regardless of how clever your fusion logic is.
• Reranking with a cross-encoder after fusion usually produces a bigger quality jump than tuning the fusion weights themselves. If you only have budget for one additional component, budget for the reranker.

How Chunk Size Interacts With Retrieval Choice

One detail that gets lost in the sparse-vs-dense debate is that your chunking strategy changes which retrieval method looks better in benchmarks, independent of the retrieval algorithm itself. This is worth understanding before you conclude one method is "winning" in your evaluation.

BM25 tends to reward smaller, tightly-scoped chunks. Because its scoring depends on term frequency relative to document length, a short chunk that contains your exact query terms two or three times will score very highly, while the same terms diluted across a long chunk full of unrelated content score lower. If you chunk at the paragraph level, BM25 usually looks stronger than it would at the whole-document level.

Dense retrieval interacts with chunk size differently. Embedding models compress an entire chunk into one fixed-length vector, so if a chunk covers multiple topics, the resulting vector is a blurry average of all of them, and it may not match strongly with a query about any single topic. Very long chunks (multiple pages) tend to hurt dense retrieval more than they hurt BM25, because BM25 can still find the exact term needle in the haystack, while the embedding has already blurred that needle into the average.

A practical takeaway: when you're comparing BM25 against dense retrieval, run both methods against the *same* chunking scheme, and test more than one chunk size (say, 150 tokens, 400 tokens, and 800 tokens) before drawing conclusions. It's common for teams to conclude "dense retrieval underperforms for us" when the real issue was chunks that were too long and topically diverse, not a weakness of embeddings themselves.

Cost and Latency Tradeoffs

Retrieval quality is only half the engineering decision — cost and latency shape what's actually feasible at scale, and the two approaches differ substantially here too.

BM25 indexes are cheap to build and cheap to query. Building an inverted index over a few million documents takes minutes on a single machine, requires no GPU, and query latency is typically single-digit milliseconds because you're doing sorted-list intersections, not floating-point vector math. This is why search engines that need to serve billions of queries a day, like classic web search, still lean heavily on inverted-index techniques even after decades of neural retrieval research.

Dense retrieval has real infrastructure costs. Embedding your entire corpus requires running every chunk through a neural network once at index time, which is fine for a few thousand documents but becomes a meaningful GPU or API bill at millions of documents. Query-time cost is usually small (one embedding call per query, plus an approximate nearest-neighbor lookup), but that nearest-neighbor index itself — HNSW graphs in particular — consumes considerably more memory than an inverted index for the same corpus size, since you're storing dense float vectors instead of sparse term postings.

For a RAG system with a modest corpus (under a few hundred thousand chunks), this difference is rarely the deciding factor — you can comfortably afford both. But if you're planning to scale retrieval to tens of millions of documents, it's worth prototyping the cost curve for embedding and re-embedding (which you'll need every time you update your embedding model) well before you're locked into a specific vector database.

Common Mistakes Teams Make

A few patterns show up repeatedly when teams adopt hybrid retrieval without fully understanding the mechanics:

• Treating BM25 as legacy and removing it entirely after adopting embeddings, then getting surprised months later when users search for exact order numbers and get irrelevant semantic neighbors instead.
• Using the same top-k for both retrievers without considering that BM25 and dense retrieval often need different candidate pool sizes to have comparable recall, especially on skewed corpora.
• Never re-tuning `k1` and `b` for BM25 on their own corpus. The Elasticsearch defaults are reasonable, but a corpus of very short chat messages behaves very differently from a corpus of long PDF manuals, and the length-normalization parameter b matters a lot in that difference.
• Assuming a bigger embedding model always wins. Larger embedding models are slower and more expensive to run at index time and query time, and the quality gain over a well-chosen smaller model is often marginal for a specific domain. Benchmark on your own data before assuming bigger is better.
• Ignoring metadata filtering. A lot of "retrieval failures" attributed to sparse vs. dense choice are actually solved more cheaply by filtering on structured fields (date ranges, document type, department) before either retrieval method even runs.

Wrapping Up

BM25 and dense retrieval aren't rivals fighting for the same job — they're specialists with different blind spots. BM25 is fast, interpretable, requires no training, and nails exact-token matching; it fails on paraphrase, synonymy, and conceptual queries. Dense retrieval captures meaning and handles messy, natural-language queries gracefully; it fails on exact identifiers, rare vocabulary, and negation. The pragmatic answer for most production RAG systems isn't "pick one" — it's hybrid retrieval with score fusion (weighted or RRF), often followed by a reranking stage.

If retrieval quality is the bottleneck in your RAG pipeline — and for most teams, it is, far more often than the LLM's generation quality — investing an afternoon in setting up hybrid search and measuring recall@k on a real evaluation set will pay off more than any amount of prompt tweaking downstream.

If you want the fuller picture of how retrieval fits into the rest of a RAG system — chunking strategy, embedding model selection, vector database tradeoffs, and evaluation methodology — that's exactly what we cover in depth in our Introduction to RAG course, where these same BM25 and dense retrieval concepts get built out into a complete, production-grade pipeline from scratch.