RAG

Multi-Query RAG: Generating Multiple Retrieval Queries Per Question

Pramod Dutta · May 22, 2026 · 15 min read

You ship a RAG pipeline, it works great in the demo, and then a real user types something like "why did our churn spike after the pricing change" and your retriever comes back with three irrelevant chunks about a completely different quarter. The embedding for that question doesn't sit close to the embeddings of the passages that actually answer it, because the question is really three questions stacked on top of each other — churn definition, the specific pricing change, and the timing correlation. A single vector search can't cover all three angles at once. This is the exact failure mode multi-query RAG was built to fix, and once you've seen it in production logs, you can't unsee it.

Multi-query RAG is a deceptively simple idea: instead of embedding the user's question once and running one retrieval pass, you use an LLM to generate several reformulations of that question, retrieve for each one, and merge the results before generation. It's one of the highest-leverage upgrades you can make to a naive RAG system, and it usually takes an afternoon to implement. This article walks through why it works, how to build it properly, where it breaks, and how to tune it so it doesn't quietly triple your latency and your vector database bill.

Why a single query isn't enough

Embedding models compress meaning into a fixed-size vector, and that compression is lossy in a specific way: it tends to capture the dominant topic of a sentence, not every sub-claim inside it. Ask "what are the tax implications of converting a traditional IRA to a Roth IRA for someone in the 32% bracket" and the embedding will likely land somewhere near "Roth conversion" documents. It may miss a chunk that's specifically about marginal tax brackets, because that chunk's dominant topic, as far as the embedding model is concerned, is tax brackets in general, not conversions.

There's also a vocabulary mismatch problem. Users don't phrase questions the way source documents phrase answers. A user asks "how do I get my API key" and your documentation says "generate a personal access token from the developer console." Lexically these overlap in almost no words, and even semantically, cosine similarity between the two embeddings might be mediocre because "API key" and "personal access token" are treated as different concepts unless your embedding model was fine-tuned on your domain.

Multi-query RAG attacks both problems at once. By generating multiple phrasings — some more literal, some more abstract, some focused on sub-parts of a compound question — you widen the net that gets cast into your vector index. Each reformulated query retrieves a slightly different set of chunks, and the union of those sets is almost always a better retrieval set than what any single query would have pulled.

The core architecture

The pipeline has four stages: query expansion, parallel retrieval, deduplication/merging, and generation. Here's the shape of it in Python using the OpenAI API and a generic vector store interface — swap in whatever retriever you're using (Pinecone, Weaviate, pgvector, Qdrant).

from openai import OpenAI

client = OpenAI()

QUERY_EXPANSION_PROMPT = """You are an assistant that generates alternate
search queries for a retrieval system. Given a user question, produce
{n} different queries that capture different angles, sub-questions, or
phrasings of the same information need. Do not answer the question.
Return one query per line, no numbering, no extra text.

User question: {question}
"""

def generate_queries(question: str, n: int = 4) -> list[str]:
    prompt = QUERY_EXPANSION_PROMPT.format(n=n, question=question)
    response = client.chat.completions.create(
        model="gpt-4.1-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.7,
    )
    lines = response.choices[0].message.content.strip().split("\n")
    queries = [line.strip("- ").strip() for line in lines if line.strip()]
    # Always keep the original question as one of the retrieval queries
    return [question] + queries[:n]

The important detail here is that the original question stays in the mix. It's tempting to replace it entirely with "better" reformulations, but the original phrasing sometimes matches a chunk exactly (especially for named entities, error codes, or product names), and you don't want to lose that by over-engineering the query set.

Fan-out retrieval and merging results

Once you have your list of queries, you retrieve against your vector store for each one, independently, and then merge. The naive approach is to just concatenate everything and dedupe by chunk ID, but that throws away useful signal about how strongly each chunk matched across queries.

import asyncio

async def retrieve_for_query(query: str, vector_store, top_k: int = 5):
    embedding = await embed(query)
    results = await vector_store.search(embedding, top_k=top_k)
    return results

async def multi_query_retrieve(question: str, vector_store, n_queries: int = 4):
    queries = generate_queries(question, n=n_queries)
    all_results = await asyncio.gather(
        *[retrieve_for_query(q, vector_store) for q in queries]
    )

    # Merge with reciprocal rank fusion
    scores = {}
    chunks = {}
    for result_set in all_results:
        for rank, chunk in enumerate(result_set):
            chunk_id = chunk["id"]
            chunks[chunk_id] = chunk
            rrf_score = 1 / (60 + rank)
            scores[chunk_id] = scores.get(chunk_id, 0) + rrf_score

    ranked_ids = sorted(scores, key=lambda cid: scores[cid], reverse=True)
    return [chunks[cid] for cid in ranked_ids]

That merge step uses reciprocal rank fusion (RRF), which is the standard way to combine multiple ranked lists without needing the raw similarity scores to be on the same scale — a real concern when different queries retrieve chunks with very different score distributions. A chunk that shows up at rank 1 for two separate queries will outscore a chunk that only showed up once at rank 1, which is usually exactly the behavior you want: consistency across reformulations is a strong signal of relevance.

Run this asynchronously. If you fire off four queries sequentially against your vector store, you've just added three round trips of latency to every single request. asyncio.gather (or your language's equivalent concurrency primitive) keeps the added latency close to the cost of your slowest single retrieval, not the sum of all of them.

A worked example end to end

It helps to trace one question through the whole pipeline concretely. Say you're running RAG over a company's internal engineering wiki, and someone asks: "why does our deploy pipeline retry three times before failing and can I change that."

A naive single-query system embeds that whole sentence and searches. Because the dominant topic looks like "deploy pipeline retries," it might return two chunks about the CI retry mechanism and completely miss the configuration file where the retry count is actually set, because that file is titled pipeline-config-reference.md and never uses the word "retry" anywhere near the word "three."

Run it through query expansion instead, and a reasonable model produces something like:

deploy pipeline retry count configuration
why does CI retry failed deployments
how to change number of deployment retry attempts
pipeline-config-reference retry settings

Four retrieval passes go out concurrently. The first two queries pull back the same explanatory chunks the naive search would have found — the "why" side of the question. The third and fourth queries, phrased around changing a setting rather than understanding behavior, are the ones that actually surface the configuration reference chunk, because that chunk's embedding sits much closer to "change number of retry attempts" than to the original compound question. Reciprocal rank fusion then puts the explanatory chunk and the config chunk both near the top of the merged list, because each was a strong match for at least one query, and the reranker confirms both are relevant to the original question before they get passed to the generation step.

This is the mechanism in miniature: the compound question had two distinct information needs (explain the behavior, tell me how to change it), and no single embedding was going to serve both. Multi-query expansion turned one lossy search into two well-targeted ones and let fusion recombine them.

How many queries is too many

Four is a reasonable default, but the right number depends on your corpus and your latency budget. I've tested this across a few different document sets, and the pattern that shows up consistently: going from one query to three or four queries produces a large jump in recall, but going from four to eight produces diminishing and sometimes negative returns. Past a certain point you start injecting queries that are just noisy variations of each other, and each one costs you an embedding call, a vector search, and a slice of your context window budget when you eventually stuff results into the LLM prompt.

A practical way to think about it: generate queries that cover distinct sub-intents rather than distinct phrasings of the same intent. If a user asks a compound question, decompose it into its actual parts instead of generating four paraphrases of the whole thing. For "what are the tax implications of converting a traditional IRA to a Roth IRA for someone in the 32% bracket," you'd want queries like:

"Roth IRA conversion tax rules"
"how conversions are taxed as ordinary income"
"32% marginal tax bracket thresholds"
"traditional IRA to Roth IRA conversion process"

That's decomposition, not paraphrasing, and it's the version of multi-query RAG that actually earns its keep. You can steer the expansion prompt toward this behavior explicitly by asking the model to identify sub-questions first, then generate a query per sub-question.

Combining with hybrid search

Multi-query RAG solves the semantic-coverage problem, but it doesn't solve the lexical-matching problem on its own. If a user's question contains a specific product SKU, an error code, or a person's name, pure embedding search — even across multiple reformulations — can still miss the exact chunk that contains that literal string, because embeddings are bad at treating rare tokens as load-bearing.

The fix is to pair multi-query expansion with hybrid search: run each generated query through both a dense vector search and a sparse keyword search (BM25 is the standard here), then fuse all of those result lists together with the same RRF approach.

async def hybrid_retrieve_for_query(query: str, vector_store, bm25_index, top_k: int = 5):
    dense_task = retrieve_for_query(query, vector_store, top_k)
    sparse_task = asyncio.to_thread(bm25_index.search, query, top_k)
    dense_results, sparse_results = await asyncio.gather(dense_task, sparse_task)
    return dense_results, sparse_results

This roughly doubles your retrieval calls per query, so at four generated queries you're now doing eight retrieval operations per user question. That's fine if your vector store and keyword index are both fast, but it's a real cost to budget for, especially if you're paying per-query on a managed vector database.

Reranking after the merge

RRF gives you a reasonable ordering for free, but it's a heuristic based on rank position, not actual relevance to the original question. Once you've merged and deduplicated your candidate chunks — typically you'll end up with 15-30 unique chunks from a four-query fan-out — it's worth running them through a cross-encoder reranker before deciding what goes into the LLM's context window.

def rerank(question: str, candidates: list[dict], reranker_model, top_k: int = 6):
    pairs = [(question, c["text"]) for c in candidates]
    scores = reranker_model.predict(pairs)
    scored = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [c for c, _ in scored[:top_k]]

The reranker sees the *original* question, not the expanded queries — its job is to judge each candidate chunk against what the user actually asked, using the expanded queries only to have widened the candidate pool. This two-stage design (cheap fan-out retrieval, then expensive precise reranking on a smaller set) is the same pattern used in traditional search engines, and it holds up well in RAG: cast a wide net cheaply, then spend your expensive compute narrowing it down.

Handling query generation failures gracefully

LLM-generated queries occasionally go sideways. The model might generate a query that's actually an answer to the question instead of a search query ("The tax implications depend on your bracket..."), or it might hallucinate a sub-topic that isn't actually in the question at all. Don't trust the raw output blindly.

A few guardrails that are worth the extra fifteen minutes of code:

Filter out generated queries that are longer than roughly 2x the original question — these are usually the model rambling into an answer rather than staying in query mode.
Deduplicate near-identical generated queries before spending retrieval calls on them (a cheap Jaccard similarity check on token sets is enough).
Always fall back to single-query retrieval if the expansion call fails or times out, rather than blocking the whole request on it.

def sanitize_queries(question: str, generated: list[str]) -> list[str]:
    max_len = len(question.split()) * 3
    seen = set()
    cleaned = []
    for q in generated:
        tokens = frozenset(q.lower().split())
        if len(q.split()) > max_len:
            continue
        if any(len(tokens & s) / max(len(tokens | s), 1) > 0.8 for s in seen):
            continue
        seen.add(tokens)
        cleaned.append(q)
    return cleaned

This kind of defensive coding matters more in multi-query RAG than in single-query RAG, precisely because you've introduced a new LLM call into the retrieval path, and that call can fail or misbehave independently of your generation call.

Measuring whether it's actually helping

It's easy to ship multi-query RAG, see that answers "feel" better, and move on — but you should actually measure it before committing to the added latency and cost. Build a small evaluation set of 30-50 real questions with known relevant chunk IDs (you can bootstrap this from support tickets or a documentation FAQ), and compare recall@k between single-query and multi-query retrieval.

def recall_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:
    retrieved_top_k = set(retrieved_ids[:k])
    if not relevant_ids:
        return 0.0
    return len(retrieved_top_k & relevant_ids) / len(relevant_ids)

Run this across your eval set for both pipelines and look at the delta. In practice, the gain from multi-query expansion is largest on compound and vaguely-worded questions, and close to zero on short, single-fact lookups ("what's the refund policy"). If your traffic skews heavily toward the latter, the added latency and cost of query expansion may not be worth it for every request — you can route to multi-query conditionally based on question length or a cheap classifier that flags compound questions.

Cost and latency tradeoffs in production

Every generated query adds one embedding call and one retrieval call, and the query generation step itself is an LLM call with its own latency (typically 300-800ms with a fast model like a mini-tier chat model). At four queries, you're looking at roughly 4-5x the retrieval cost of naive RAG, plus the query generation overhead, before you even get to the final generation call.

A few ways teams manage this in practice:

Cache generated queries for repeated or near-duplicate questions, since users often ask semantically similar things in slightly different words.
Use a small, fast model purely for query expansion — you don't need your most capable model to reformulate a question, you need it to be fast and reasonably diverse.
Set n_queries dynamically: short factual questions get 1-2 expansions, long or multi-clause questions get 4-5.
Batch the embedding calls for all generated queries into a single API request when your embedding provider supports batching, instead of firing them one at a time.

None of these are exotic, but skipping them is the difference between a multi-query RAG system that costs a reasonable amount more than naive RAG and one that quietly 5x's your inference bill because every user question, regardless of complexity, triggers the full expansion pipeline.

Debugging a multi-query pipeline that isn't helping

If you've implemented multi-query RAG and the recall numbers barely move, the bug is almost always in one of three places, and it's worth checking them in this order before you assume the technique itself doesn't work for your corpus.

First, look at the actual generated queries in your logs, not just the final retrieved chunks. It's common to find that the expansion prompt is producing four queries that are lexically different but semantically identical — four ways of saying the same thing rather than four angles on the question. If that's happening, your expansion prompt needs to explicitly ask for sub-question decomposition rather than paraphrasing, as covered above, and you may need a stronger model for the expansion step specifically, even if you keep a cheaper model for final generation.

Second, check whether your chunking strategy is fighting against you. Multi-query RAG widens the net for which chunks get retrieved, but if your chunks are poorly bounded — say, 2,000-token chunks that each cover four unrelated subtopics — then widening the net doesn't help, because every query keeps pulling back the same oversized, unfocused chunks. Multi-query expansion amplifies the benefit of good chunking; it doesn't substitute for it.

Third, verify your merge step isn't accidentally collapsing diversity back down. A bug I've seen more than once: teams implement the fan-out correctly, then take the top-k from *each* query and simply concatenate-and-truncate to an overall top-k, discarding the RRF step. That approach tends to just return the results of whichever query happened to run first or produce the highest raw similarity scores, throwing away exactly the cross-query signal that made the fan-out worthwhile in the first place. Log the merged, scored candidate list before reranking and manually eyeball it against a handful of known-good answers to confirm fusion is doing what you think it's doing.

Putting it together

The full request flow looks like this end to end: a question comes in, you generate and sanitize a handful of alternate queries, you fan those out concurrently against your vector store (and optionally a keyword index for hybrid search), you merge everything with reciprocal rank fusion, you rerank the merged candidates against the original question with a cross-encoder, and you take the top handful of chunks into your generation prompt. Each stage is independently swappable and independently testable, which is what makes this pattern maintainable rather than a black box you're afraid to touch.

The biggest mindset shift is treating retrieval as a search problem with its own optimization surface, not as a single fixed step bolted onto your LLM call. Naive RAG treats the embedding of the user's question as ground truth for what's relevant. Multi-query RAG accepts that a single embedding is an imperfect proxy for intent, and spends a small amount of extra compute to triangulate on the real information need from several angles instead of trusting one shot at it.

If you're building out a RAG pipeline from scratch and haven't yet nailed down the fundamentals of chunking, embeddings, and retrieval evaluation, it's worth backing up to the basics before layering on query expansion — our Introduction to RAG course covers that foundation in depth, and multi-query techniques like this one build directly on top of it once you're comfortable with the core loop.