Semantic vs Keyword Search: What RAG Actually Needs

The semantic vs keyword search debate gets framed as a winner-take-all contest, but that framing is wrong for RAG (retrieval-augmented generation). Semantic search, powered by vector embeddings, is good at matching meaning and paraphrase. Keyword search, typically BM25, is good at matching exact terms: product codes, error messages, names, acronyms, negations. Most retrieval failures in production RAG systems trace back to using only one of these when the query needed the other. This article walks through what each approach actually does under the hood, where each one breaks, and how to combine them with hybrid search and reranking so your retrieval layer stops silently failing on the query types that matter most.

How keyword search works

Keyword search, in its modern form, means BM25 (Best Matching 25), a ranking function built on top of an inverted index. An inverted index maps every term in your corpus to the list of documents containing it. When a query comes in, BM25 scores each candidate document by:

• Term frequency: how often the query term appears in the document, with diminishing returns for repetition
• Inverse document frequency: rare terms across the corpus count for more than common ones
• Document length normalization: a 200-word document matching a term scores differently than a 5,000-word document matching the same term

BM25 is implemented in Elasticsearch, OpenSearch, Postgres full-text search (via tsvector), Typesense, and Meilisearch, among others. It requires no model inference at query time, it's fast, it's interpretable (you can see exactly why a document scored high), and it does not hallucinate matches. If a user searches "ORA-01555 snapshot too old," BM25 will find documents containing that exact string. A vector embedding model, unless it happens to have seen that error code during training, has no idea what to do with it.

The weakness of keyword search is equally sharp: it does not understand synonymy or paraphrase. A document about "canceling a subscription" will not match a query for "how do I stop my recurring payment" unless both phrasings happen to share enough overlapping terms. BM25 also struggles with queries that are conceptually related but lexically distant, and it has no notion of semantic similarity between "car" and "automobile."

How semantic search works

Semantic search encodes text into dense vectors using an embedding model, then finds nearby vectors using a similarity metric, usually cosine similarity or dot product. The pipeline looks like this:

• Chunk your documents into passages (typically 200 to 800 tokens depending on the embedding model's context window and your retrieval granularity)
• Embed each chunk with a model like OpenAI's text-embedding-3-large, Cohere's embed-v4, or an open model such as bge-large or nomic-embed-text
• Store the resulting vectors in a vector index (HNSW is the dominant algorithm) inside a vector database like Pinecone, Qdrant, Weaviate, or pgvector on Postgres
• At query time, embed the user's query with the same model and run an approximate nearest neighbor search to retrieve the top-k closest chunks

The strength here is exactly what BM25 lacks: semantic search finds documents that mean the same thing even when the words differ. "How do I stop my recurring payment" and "canceling a subscription" will land close together in embedding space because the model has learned that these concepts are related from its training data.

The weakness is subtler and easy to miss until it costs you a support ticket. Embedding models are trained to capture general semantic similarity, not exact lexical matching, and they can be surprisingly bad at:

• Exact identifiers: SKUs, order numbers, error codes, version numbers, function names. The string "v2.3.1" and "v2.3.0" will often embed as nearly identical vectors because the model treats them as semantically similar, even though for your use case they're completely different documents.
• Negation: "how to enable two-factor authentication" and "how to disable two-factor authentication" frequently embed close together because the surrounding context is nearly identical. The embedding model captures topical similarity, not the polarity flip.
• Rare or out-of-vocabulary terms: a newly coined internal product name, a rarely-used API parameter, a typo the user made. If the embedding model never saw anything like it during training, the vector it produces is not meaningfully anchored to anything.
• Short, keyword-like queries: a query like "invoice PDF export bug" is really three keywords stapled together, not a natural-language question. Embedding models are typically trained on more sentence-like text and can underperform on this query shape.

Why RAG breaks when you pick only one

A RAG pipeline is only as good as the chunks it retrieves. If retrieval misses the relevant passage, the language model generating the answer never sees it, and no amount of prompt engineering fixes that: the model will either hallucinate an answer or say it doesn't know. This is why the semantic vs keyword search decision is not academic for RAG builders. It directly determines your answer accuracy.

Consider a support-bot RAG system built on pure semantic search over a knowledge base. A user asks "what does error E1042 mean." The embedding for that query sits somewhere in a general "error codes and troubleshooting" region of vector space, but nothing in your embedding model's training data taught it that "E1042" specifically means "payment gateway timeout." The nearest neighbors returned might be generic troubleshooting articles that don't mention E1042 at all. Pure keyword search, by contrast, finds the one document containing the literal string "E1042" instantly, because BM25 doesn't need to understand what the code means, it just needs the string to match.

Now flip it. A user asks "why did my payment fail," and the only document in your knowledge base that answers this is titled "Resolving Declined Transactions" and never uses the word "fail" once. Pure keyword search returns nothing useful. Semantic search finds it immediately because "payment fail" and "declined transaction" are close in meaning.

Neither failure mode is rare. In practice, real user queries mix identifier-heavy lookups, natural-language questions, and everything in between, often within the same session. A retrieval layer tuned for only one query shape will silently underperform on the other, and because RAG failures usually look like "the model gave a vague or wrong answer" rather than an obvious error, these gaps are hard to catch without deliberate evaluation.

Hybrid search: running both and merging results

The practical fix used by most production RAG systems today is hybrid search: run BM25 and vector search in parallel over the same query, then merge the two ranked lists into one.

The most common merging technique is Reciprocal Rank Fusion (RRF), which doesn't require normalizing scores across two different scoring systems (BM25 scores and cosine similarities are not on comparable scales). RRF instead uses each result's rank position:

RRF_score(d) = sum over each ranked list r of 1 / (k + rank_r(d))

where k is a constant (commonly 60) that dampens the influence of lower-ranked results. A document that ranks highly in both the BM25 list and the vector list gets a high combined score. A document that ranks highly in only one list still gets credit, just less of it. This is simple to implement and doesn't require tuning a weighted blend between the two systems.

A minimal hybrid retrieval function looks like this:

def reciprocal_rank_fusion(bm25_results, vector_results, k=60):
    scores = {}
    for rank, doc_id in enumerate(bm25_results):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    for rank, doc_id in enumerate(vector_results):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

bm25_results and vector_results are each a ranked list of document IDs from their respective retrieval systems, already run independently against the same query. You take the top-k from each (commonly 20 to 50) before fusing, then take the top-k of the fused result to pass downstream.

Several vector databases now support hybrid search natively, which saves you from hand-rolling the fusion step:

• Weaviate has a built-in hybrid query type with an alpha parameter to weight vector vs keyword contribution
• Qdrant supports sparse-dense hybrid search using sparse vectors (e.g., SPLADE) alongside dense embeddings, fused server-side
• Elasticsearch and OpenSearch support combining their native BM25 scoring with k-NN vector fields in a single query
• Pinecone supports sparse-dense hybrid indexes using a single vector combining both representations
• Postgres with pgvector can run a hybrid query by combining tsvector full-text ranking and vector cosine distance in one SQL query, then blending with RRF in application code or a UNION-based CTE

If you're running Postgres already, this is often the lowest-friction way to add hybrid search without introducing a second database:

WITH keyword_search AS (
  SELECT id, ts_rank(content_tsv, query) AS score,
         row_number() OVER (ORDER BY ts_rank(content_tsv, query) DESC) AS rank
  FROM documents, plainto_tsquery('english', 'payment gateway timeout') query
  WHERE content_tsv @@ query
  LIMIT 50
),
vector_search AS (
  SELECT id, 1 - (embedding <=> $1) AS score,
         row_number() OVER (ORDER BY embedding <=> $1) AS rank
  FROM documents
  ORDER BY embedding <=> $1
  LIMIT 50
)
SELECT COALESCE(k.id, v.id) AS id,
       COALESCE(1.0 / (60 + k.rank), 0) + COALESCE(1.0 / (60 + v.rank), 0) AS rrf_score
FROM keyword_search k
FULL OUTER JOIN vector_search v ON k.id = v.id
ORDER BY rrf_score DESC
LIMIT 10;

Reranking: the step most teams skip

Hybrid search improves recall (finding the right documents somewhere in your candidate set) but it doesn't guarantee precision at the top of the list. The fused ranking is a mechanical combination of two independent scoring systems, and it doesn't actually read the query and the document together to judge relevance.

A cross-encoder reranker does. Unlike embedding models, which encode the query and document separately and then compare vectors (a bi-encoder architecture), a cross-encoder takes the query and document as a single input and outputs a relevance score directly. This is more expensive per comparison, which is why you only run it on the top 20 to 50 candidates from hybrid search rather than your whole corpus, but it materially improves final ranking quality.

Common rerankers as of 2026 include Cohere's rerank-v3.5, Voyage AI's rerank models, and open options like BAAI's bge-reranker family that you can self-host. The typical pipeline is:

1. Retrieve top 50 with BM25
2. Retrieve top 50 with vector search
3. Fuse with RRF into a candidate set of roughly 30 to 50 unique documents
4. Rerank the candidate set with a cross-encoder
5. Pass the top 5 to 10 reranked chunks into the language model's context window

This three-stage funnel (recall-oriented hybrid retrieval, then precision-oriented reranking) is the architecture behind most serious RAG systems in production today. Skipping reranking is the most common reason teams see decent recall in evaluation but mediocre answer quality in the actual product: the right chunk was in the candidate set, it just wasn't ranked high enough to make the final context window.

When you can skip hybrid and use just one

Hybrid search adds infrastructure complexity: two indexes to maintain, a fusion step, and usually a reranker on top. It's not always worth it.

Pure semantic search is often fine when your corpus is narrative or conversational (support transcripts, long-form documentation, internal wikis) and your users ask natural-language questions without relying on exact codes or identifiers. If nobody is searching for SKUs or error strings, the keyword-matching weakness of embeddings never gets exercised.

Pure keyword search is often fine, or even preferable, for structured or code-heavy corpora: API references, log search, codebases, or catalogs where users search by exact identifier almost exclusively. Adding vector search here mostly adds noise and latency without meaningfully improving results, since BM25 already handles the dominant query pattern well.

The signal to watch for is your query log. If you see a mix of natural-language questions and exact-term lookups, which is the common case for customer support, internal knowledge bases, and technical documentation, you need hybrid search. If your query distribution is heavily skewed to one shape, you can often ship with a single retrieval method and revisit later.

Evaluating retrieval quality directly

Don't infer retrieval quality from downstream answer quality alone, because a good language model can sometimes compensate for weak context, and a bad model can produce a poor answer even from perfect context. Evaluate retrieval on its own using a held-out set of query-to-relevant-document pairs, and track:

• Recall@k: of the documents actually relevant to the query, what fraction appear in your top-k retrieved results
• MRR (Mean Reciprocal Rank): how high up the first relevant result appears, averaged across your evaluation set
• NDCG (Normalized Discounted Cumulative Gain): accounts for graded relevance and rewards putting the most relevant results at the very top

Build this evaluation set from real queries, not synthetic ones, if you can. Pull a sample of actual user questions, manually label which chunks in your corpus are relevant to each, and run that set against every retrieval configuration change you make. This turns "does hybrid search help here" from a guess into a measurement, and it's the only reliable way to know whether adding a reranker, switching embedding models, or tuning your RRF constant actually moved the needle for your specific corpus and query mix.

FAQ

Is semantic search always better than keyword search for RAG? No. Semantic search is better for paraphrased, conceptual, or natural-language queries. Keyword search is better for exact identifiers, error codes, negations, and rare terms. Most production RAG systems need both, combined through hybrid search, because real user queries mix both shapes.

What is BM25 and why is it still used alongside vector search? BM25 is a term-frequency-based ranking algorithm built on inverted indexes. It's still used because it matches exact strings reliably, requires no model inference, is fast and interpretable, and covers the identifier-heavy and negation-sensitive queries where embedding models tend to struggle.

Do I need a vector database, or can I use Postgres for hybrid search? Postgres with the pgvector extension can handle both keyword search (via tsvector and ts_rank) and vector search in the same database, which is often enough for small to mid-sized corpora. Dedicated vector databases like Qdrant, Weaviate, or Pinecone become more attractive as your corpus grows or you need features like native hybrid fusion, filtering at scale, or multi-tenant isolation.

What's the difference between hybrid search and reranking? Hybrid search combines two retrieval methods (BM25 and vector search) to build a better candidate set, optimizing for recall. Reranking takes that candidate set and reorders it using a more expensive model that scores the query and each document together, optimizing for precision at the top of the list. They solve different problems and are typically used together, not as alternatives.

How do I choose the RRF constant k when fusing BM25 and vector results? A value of 60 is the common default from the original RRF paper and works reasonably well across most corpora without tuning. If you have a labeled evaluation set, you can sweep values (commonly between 10 and 100) and measure the effect on Recall@k or NDCG, but in practice the choice of k matters far less than whether you're fusing at all versus using a single retrieval method.

Does a bigger embedding model fix the exact-match weakness of semantic search? Not reliably. Larger embedding models generally improve semantic similarity judgments, but they are still trained to capture meaning, not exact string matching, so identifiers, codes, and negation-sensitive queries remain a weak spot regardless of model size. This is a structural property of how embeddings are trained, not something that scales away.

Should I chunk documents differently for keyword search versus semantic search? You can use the same chunks for both if you're running hybrid search over one corpus, which is simpler to maintain. Some teams do use larger chunks for keyword indexes (since BM25 handles longer documents reasonably well) and smaller, more focused chunks for embeddings (since dense vectors lose precision as passages get longer and cover multiple topics). Start with one chunking strategy shared across both, and only split them if evaluation shows a clear benefit.