Hybrid Search in Vector Databases

Vector hybrid search runs two retrievers at once, a sparse keyword scorer like BM25 and a dense embedding search, then fuses their results into one ranked list. You reach for it when pure vector search keeps missing exact matches (part numbers, error codes, rare names) and pure keyword search keeps missing paraphrases. This article shows how vector hybrid search works, how to build it in Postgres and in a dedicated vector database, and how to tune the fusion so retrieval quality actually goes up instead of getting noisier.

Why pure vector search is not enough

Dense embeddings are good at meaning. Ask for "how do I reset my password" and a vector search will happily return a chunk titled "recovering account access" even though not one word overlaps. That is the whole reason embeddings took over retrieval.

The failure mode shows up the moment a query contains a token that carries meaning precisely because it is literal. Consider these queries:

• ERR_CONN_REFUSED in a logs knowledge base
• SKU-4471-B in a product catalog
• ibuprofen 400mg in a medical corpus
• useEffect cleanup in code docs

Embedding models compress text into a few hundred or a few thousand floats. That compression blurs exactly the rare, high-signal tokens you needed. SKU-4471-B and SKU-4471-C land almost on top of each other in vector space, so the nearest-neighbor search cannot tell them apart. A keyword index treats them as completely different terms, which is correct here.

So the two methods fail in opposite directions. Dense search loses on lexical precision; sparse search loses on semantics. Vector hybrid search is the pragmatic answer: run both, then combine. You do not have to pick which failure you can live with.

What sparse and dense actually mean

A dense vector is what most people mean by "embedding": a fixed-length array of floats from a model such as OpenAI text-embedding-3-large, Cohere embed-v4, or an open model like bge-m3. Similarity is cosine or dot product. Every dimension is populated.

A sparse vector is keyword-shaped. Think of a dictionary of tens of thousands of terms where a given document only lights up the handful of terms it actually contains. BM25 is the classic scoring function over this representation. It rewards documents that contain the query terms, discounts terms that appear in almost every document (like "the"), and dampens the effect of a term appearing many times in one long document.

There is a middle category worth knowing: learned sparse models such as SPLADE. They still produce a sparse term-weight vector, so you can store it in an inverted index, but the weights come from a transformer that does light query and document expansion. That means car can activate the term automobile with a smaller weight, giving you some semantic recall without giving up the exact-match backbone. If your vector database supports sparse vectors natively, SPLADE-style vectors are a strong drop-in upgrade over raw BM25.

For this article, "sparse" means BM25 or SPLADE and "dense" means a normal embedding. Vector hybrid search fuses the two.

Fusion: how you actually combine the two lists

You get two ranked lists back. Now what. There are two mainstream fusion strategies and you should understand both because their failure modes differ.

Reciprocal Rank Fusion (RRF) ignores the raw scores and only looks at the rank position of each document in each list. A document at rank 1 in the dense list and rank 3 in the sparse list gets a combined score built from those positions. The formula for one document is the sum over every list of 1 / (k + rank), where rank starts at 1 and k is a smoothing constant usually set to 60.

RRF is popular for a good reason: it needs no score normalization. Dense cosine scores live in one range, BM25 scores live in a totally different and unbounded range, and trying to add them directly is a mess. RRF sidesteps that entirely by throwing away the magnitudes and keeping only order.

Here is a self-contained RRF implementation:

def rrf_fuse(dense_hits, sparse_hits, k=60, top_n=10):
    # dense_hits, sparse_hits: lists of doc ids in ranked order (best first)
    scores = {}
    for ranked_list in (dense_hits, sparse_hits):
        for rank, doc_id in enumerate(ranked_list, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    ranked = sorted(scores.items(), key=lambda kv: kv[1], reverse=True)
    return [doc_id for doc_id, _ in ranked[:top_n]]

The second strategy is weighted score fusion. You normalize each list's scores to a common range (min-max to 0..1 is typical), then take a weighted sum: alpha * dense_score + (1 - alpha) * sparse_score. This keeps magnitude information, so a document that is a runaway top dense match can dominate even if it is only mid-pack on keywords. The cost is that you now own the normalization and the alpha, and both need tuning per corpus.

Rule of thumb: start with RRF because it is robust and parameter-light. Move to weighted fusion only when you have an evaluation set and can measure that the extra control buys you recall.

Building vector hybrid search in Postgres with pgvector

You do not need a separate vector database to start. Postgres with the pgvector extension plus its built-in full-text search covers real workloads, and keeping retrieval in the same database you already run is a genuine operational win.

Set up a table with both a dense column and a text-search column:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE docs (
    id        bigserial PRIMARY KEY,
    content   text NOT NULL,
    embedding vector(1024),
    ts        tsvector GENERATED ALWAYS AS (to_tsvector('english', content)) STORED
);

CREATE INDEX docs_embedding_idx ON docs
    USING hnsw (embedding vector_cosine_ops);

CREATE INDEX docs_ts_idx ON docs USING gin (ts);

The ts column is a generated tsvector, so keyword search stays in sync automatically. The HNSW index accelerates the dense side; the GIN index accelerates the keyword side.

Now run both retrievers and fuse them in one query using RRF. You pass in the query embedding and the raw query text:

WITH dense AS (
    SELECT id,
           row_number() OVER (ORDER BY embedding <=> $1) AS rank
    FROM docs
    ORDER BY embedding <=> $1
    LIMIT 50
),
sparse AS (
    SELECT id,
           row_number() OVER (
               ORDER BY ts_rank_cd(ts, plainto_tsquery('english', $2)) DESC
           ) AS rank
    FROM docs
    WHERE ts @@ plainto_tsquery('english', $2)
    LIMIT 50
)
SELECT COALESCE(d.id, s.id) AS id,
       COALESCE(1.0 / (60 + d.rank), 0.0)
     + COALESCE(1.0 / (60 + s.rank), 0.0) AS score
FROM dense d
FULL OUTER JOIN sparse s ON d.id = s.id
ORDER BY score DESC
LIMIT 10;

The <=> operator is cosine distance in pgvector. The FULL OUTER JOIN matters: a document that shows up in only one of the two lists still gets scored, it just gets a contribution from that one side. That is the correct behavior. An exact keyword hit that the embedding missed should still surface.

A couple of practical notes. Pull 50 candidates from each side (the LIMIT 50) before fusing down to 10, because a document ranked 30th on one side and 2nd on the other can win overall, and you would miss it if you only pulled the top 10 per retriever. Also, plainto_tsquery handles user input safely and does the stemming; do not hand-build tsquery strings from raw user text.

Building it in a dedicated vector database

Once your corpus grows past what a single Postgres box wants to hold, or you need sparse SPLADE vectors and score-based fusion out of the box, a purpose-built vector database earns its keep. Qdrant, Weaviate, Milvus, and Elasticsearch all ship native hybrid search now, and the managed embedding databases like Pinecone support sparse-dense vectors too.

Here is vector hybrid search in Qdrant using its query API, which does the fusion server-side. You upload both a dense vector and a sparse vector per point, then ask for both and let Qdrant fuse with RRF:

from qdrant_client import QdrantClient
from qdrant_client import models

client = QdrantClient(url="http://localhost:6333")

client.query_points(
    collection_name="docs",
    prefetch=[
        models.Prefetch(
            query=dense_vector,          # list[float] from your embedder
            using="dense",
            limit=50,
        ),
        models.Prefetch(
            query=models.SparseVector(   # from BM25 or SPLADE
                indices=sparse_indices,
                values=sparse_values,
            ),
            using="sparse",
            limit=50,
        ),
    ],
    query=models.FusionQuery(fusion=models.Fusion.RRF),
    limit=10,
)

The shape is the same idea as the Postgres query: two prefetches, each capped at 50, then a fusion step that returns the top 10. The difference is that the database owns the fusion and the sparse index, so you are not writing SQL windows by hand. Weaviate exposes the same pattern through an alpha parameter on its hybrid query, where alpha=1 is pure dense, alpha=0 is pure keyword, and 0.5 is an even blend. Elasticsearch calls it an rrf retriever that wraps a standard (BM25) retriever and a knn retriever.

Whichever engine you pick, the mental model does not change: two retrievers, capped candidate pools, one fusion step.

Add a reranker for the last mile

Fusion gives you a good top 10, but it is still a coarse combination of two approximate signals. If quality matters, put a cross-encoder reranker after fusion. A reranker takes the query and each candidate document together, jointly, and scores how well they match. That joint attention is far more accurate than comparing two independently produced vectors, which is why it moves the needle on precision at the top.

The pattern is retrieve wide, rerank narrow:

# 1. hybrid search returns ~50 candidates
candidates = hybrid_search(query, limit=50)

# 2. rerank with a cross-encoder (Cohere rerank, bge-reranker, etc.)
import cohere
co = cohere.ClientV2()
reranked = co.rerank(
    model="rerank-v3.5",
    query=query,
    documents=[c["content"] for c in candidates],
    top_n=5,
)

# 3. keep the top 5 the reranker chose
final = [candidates[r.index] for r in reranked.results]

The reason this works well on top of hybrid search: hybrid gives you high recall (the right document is somewhere in the 50), and the reranker gives you high precision (the right document rises to the top of the 5 you actually feed the LLM). Rerankers are heavier per document, so you only run them on the fused candidate set, never on the whole corpus. Fifty candidates is a common sweet spot; measure your own latency budget.

Chunking and indexing choices that decide the outcome

Vector hybrid search cannot rescue bad chunks. Two things quietly determine whether it works.

First, chunk size. Chunks that are too large dilute both signals: the embedding averages over too many topics, and BM25 term frequency gets muddied. Chunks that are too small lose context and the LLM gets fragments. Start around 200 to 500 tokens with a small overlap of 10 to 15 percent, and adjust based on how your documents are structured. Structured docs (API references, FAQs) chunk cleanly on their existing boundaries; prose needs more care.

Second, what text you embed versus what text you keyword-index. It is often worth embedding a slightly cleaned or summarized version of a chunk while keyword-indexing the raw text, so the sparse side keeps every literal token (codes, IDs, exact phrasing) while the dense side sees cleaner semantics. In Postgres that means the ts column stays on content while your embedding is computed from a preprocessed field.

One more indexing note for the dense side: HNSW has build-time parameters (m, ef_construction) and a query-time ef_search that trades recall for latency. For hybrid search you generally want the dense retriever tuned for high recall, because the fusion and reranker downstream will handle precision. A dense list that silently drops the right document cannot be repaired later.

Measure it, do not vibe it

The single biggest mistake teams make is shipping vector hybrid search because it "feels better" without a number attached. Build a small evaluation set: 50 to 100 real queries, each with the document ids that should be retrieved. Then compute recall at k and a rank-aware metric like nDCG or MRR for three configurations: dense only, sparse only, and hybrid.

You are looking for two things. One, does hybrid beat both single retrievers on recall at 10. It almost always should; if it does not, your fusion or candidate limits are off. Two, where does the win come from. Bucket your queries into "keyword-heavy" (codes, names) and "semantic" (paraphrases, questions) and confirm hybrid holds up on both while each single method sags on one. That bucketed view tells you whether to lean the alpha toward dense or sparse for your specific traffic.

Rerun this eval whenever you change the embedding model, the chunker, or the fusion weights. Retrieval quality is not a set-and-forget property.

FAQ

When should I not bother with hybrid search? If your queries are almost entirely natural-language questions with no literal identifiers, and your evaluation shows dense-only already hitting your recall target, hybrid adds cost for little gain. Hybrid earns its complexity when your traffic mixes exact-match terms (product codes, error strings, proper nouns) with conceptual questions. Always confirm with an eval set rather than adding it reflexively.

Is RRF or weighted score fusion better? Start with RRF. It needs no score normalization, has one sane default (k=60), and rarely does anything catastrophic. Weighted fusion can beat it when you have an evaluation set and can tune alpha per corpus, because it preserves score magnitude that RRF discards. In practice many production systems ship RRF and never move off it.

Do I need a dedicated vector database, or is Postgres enough? Postgres with pgvector plus built-in full-text search handles real hybrid workloads and keeps everything in one system you already operate. Move to a dedicated engine like Qdrant, Weaviate, or Milvus when your corpus outgrows a single node, when you need native SPLADE sparse vectors, or when server-side fusion and reranking pipelines save you meaningful code. Do not reach for a new datastore before you have measured that Postgres is the bottleneck.

How does SPLADE differ from BM25 in hybrid search? BM25 scores documents purely on the literal terms they contain. SPLADE is a learned sparse model: a transformer expands both query and document with related terms at learned weights, so laptop can lightly activate notebook. You still store it in an inverted index and fuse it the same way, but you get some semantic recall on the sparse side. If your vector database supports sparse vectors, SPLADE is usually a better keyword leg than raw BM25.

Where does the reranker fit relative to fusion? After fusion, not instead of it. Fusion combines the two retrievers into one high-recall candidate list (say 50 documents). The cross-encoder reranker then re-scores those candidates by looking at query and document jointly, and you keep the top 5. Fusion buys recall, reranking buys precision. Running a reranker over the entire corpus instead of a candidate set is too slow and unnecessary.

What candidate count should each retriever return before fusion? A common default is 50 per retriever fused down to 10, or fused to 50 then reranked to 5. The key point is to pull more candidates per side than your final answer count, because a document ranked low on one retriever and high on the other can win after fusion and you would lose it if you truncated each list too early. Tune the number against your latency budget and eval scores.

Does hybrid search fix bad chunking? No. If chunks are too large, both the embedding and the BM25 term frequencies get diluted and no fusion strategy recovers the signal. Get chunking right first (roughly 200 to 500 tokens with light overlap, respecting document structure), then add hybrid search on top. Retrieval quality is a stack, and chunking is the foundation.