Query Rewriting for RAG: HyDE, Step-Back and Decomposition

Query rewriting is the highest-leverage fix for a RAG pipeline that keeps retrieving the wrong chunks: instead of embedding the user's raw question, you transform it into one or more queries that match how your documents are actually written. The three techniques that cover most real failure modes are HyDE (embed a hypothetical answer instead of the question), step-back prompting (retrieve on a more general version of the question), and query decomposition (split multi-part questions into sub-queries and merge the results). This guide implements all three in plain Python, adds reciprocal rank fusion to combine result lists, and finishes with a routing heuristic plus an evaluation loop so you can prove which rewrite strategy helps on your own data instead of guessing.

Everything here slots in front of an existing vector search call. No framework required, though we will name the equivalents in LangChain and LlamaIndex where they exist so you can map the concepts.

Why query rewriting fixes RAG retrieval

By 2026 the generation half of RAG is rarely the problem. Current models synthesize a correct answer almost every time the right chunks are in the context window. When a RAG answer is wrong, tracing it back usually lands on the same root cause: retrieval never surfaced the passage that contains the answer. And retrieval failed because the query vector and the document vector were never going to be close.

Users and documents speak different languages. Users write short, typo-ridden, jargon-mismatched, context-dependent questions. Documents are declarative, complete, and written in the vocabulary of whoever authored them. Consider a support corpus for a Kubernetes platform. The docs say "configuring memory limits and requests for containers". The user asks "how do I stop my pods from getting OOMKilled". There is topical overlap, but a bi-encoder embedding model has to bridge slang, an error string, and a negated goal ("stop") against neutral reference prose. Sometimes it manages. At the margin, it does not, and the margin is where your bad answers live.

Embedding models trained on query-passage pairs (the asymmetric retrieval setup used by text-embedding-3-small, voyage-3.5, bge-m3 and friends) close part of this gap, but they were trained on general web data. Your corpus has its own dialect: internal product names, acronyms, error codes, legal phrasing. Query rewriting is how you translate user language into corpus language at runtime, without retraining anything.

The failure modes map cleanly onto techniques:

• Vocabulary mismatch (user words differ from document words): multi-query rewriting and HyDE.
• Specificity mismatch (hyper-specific question, docs explain general principles): step-back prompting.
• Multi-hop or multi-topic questions (one vector cannot represent two intents): query decomposition.
• Context-dependent follow-ups in chat ("does it scale?"): conversational condensing.

If you remember one thing from this article: a reranker cannot fix any of these. Reranking reorders what retrieval already returned. If the right chunk never entered the candidate set, no post-retrieval step recovers it. Query rewriting raises the recall ceiling; everything downstream only spends the recall you already bought.

Where the rewrite step sits in the pipeline

The rewritten pipeline looks like this:

user query
  -> rewrite step (one small LLM call)
  -> 1..N search queries
  -> embed + vector/hybrid search per query (parallel)
  -> fuse result lists (reciprocal rank fusion)
  -> optional cross-encoder rerank
  -> generate

All the code below assumes three primitives. Wire them to whatever you run in production (pgvector, Qdrant, Pinecone, OpenSearch):

from openai import OpenAI

client = OpenAI()
FAST_MODEL = "gpt-5-mini"  # any small, fast model works; Claude Haiku 4.5 or a local 8B are fine too

def llm(prompt: str) -> str:
    resp = client.chat.completions.create(
        model=FAST_MODEL,
        messages=[{"role": "user", "content": prompt}],
    )
    return resp.choices[0].message.content.strip()

def embed(text: str) -> list[float]:
    resp = client.embeddings.create(model="text-embedding-3-small", input=text)
    return resp.data[0].embedding

def search(vector: list[float], k: int = 8) -> list:
    # your vector store call; must return objects with .id and .text
    ...

One structural decision matters more than any individual technique: always search the original query too. Every rewrite occasionally loses intent, and the cheapest insurance is to run the raw query in parallel and fuse. This single habit prevents nearly all rewriting regressions.

Baseline query rewriting: one cheap LLM call

Before HyDE or anything fancy, get the boring version working: fix typos, expand acronyms, strip filler, surface implied terms. This alone repairs a surprising share of failed retrievals, because so many production queries arrive as fragments like "sso login broken after upg 2.3".

REWRITE_PROMPT = """Rewrite the search query for a documentation search engine.
Fix typos, expand abbreviations and acronyms, remove greetings and filler,
and add the formal term for any slang. Keep every constraint from the
original (versions, product names, error codes). Output only the rewritten
query, nothing else.

Query: {query}
Rewritten:"""

def rewrite(query: str) -> str:
    return llm(REWRITE_PROMPT.format(query=query))

def rewrite_search(query: str, k: int = 8) -> list:
    original_hits = search(embed(query), k=k)
    rewritten_hits = search(embed(rewrite(query)), k=k)
    return fuse([original_hits, rewritten_hits], top_n=k)

Input: "sso login broken after upg 2.3". A decent rewrite: "SSO single sign-on login failure after upgrading to version 2.3". The error-prone slang is gone, the acronym is expanded next to its full form, and the version constraint survived. Note the explicit instruction to preserve constraints: without it, small models happily drop version numbers, which silently changes what the query means.

HyDE: hypothetical document embeddings

HyDE comes from the paper "Precise Zero-Shot Dense Retrieval without Relevance Labels" (Gao et al.). The observation: document-to-document similarity is easier for embedding models than question-to-document similarity, because two documents about the same topic share vocabulary, register, and structure. So instead of embedding the question, ask an LLM to write a fake answer, embed the fake answer, and search with that vector.

The counterintuitive part is that hallucination does not matter much here. The hypothetical document is never shown to anyone and never treated as truth. Its job is purely to be shaped like the real answer: same terminology, same sentence structure, same density. The vector store supplies the actual facts. A fabricated passage that says the wrong default value in the right vocabulary still lands next to the chunk that states the correct one.

HYDE_PROMPT = """Write a short passage (under 120 words) that directly answers
the question below. Write it in the style of technical documentation:
declarative, specific, no hedging, no preamble. If you are unsure of exact
details, state plausible ones; this passage is used only as a search probe
and is never shown to a user.

Question: {question}

Passage:"""

def hyde_search(question: str, k: int = 8, n_docs: int = 3) -> list:
    result_lists = [search(embed(question), k=k)]  # keep the raw query
    for _ in range(n_docs):
        fake_doc = llm(HYDE_PROMPT.format(question=question))
        result_lists.append(search(embed(fake_doc), k=k))
    return fuse(result_lists, top_n=k)

The original paper averages the embeddings of several hypothetical documents together with the query embedding into a single vector; searching each and fusing ranks, as above, is an equivalent and simpler variant when your store bills per query cheaply. Three hypothetical documents is a reasonable default; beyond five you pay latency for noise.

Two practical tuning notes. First, cap the passage length in the prompt. Long hypothetical documents drift into generic filler, and the filler drags the embedding toward the center of your corpus. Second, tell the model what your corpus sounds like. "In the style of a Django REST Framework changelog entry" produces a far better probe against a changelog corpus than generic documentation prose.

When HyDE helps and when it hurts

HyDE is not a default-on technique, and knowing its failure profile matters as much as the implementation.

HyDE tends to help when:

• Questions are conversational but documents are formal reference material, the classic register gap.
• You are in a zero-shot setting with no query-passage training data and no reranker yet.
• The corpus is dense technical prose where answer-shaped text clusters tightly (API docs, runbooks, legal clauses).

HyDE tends to hurt when:

• The model knows nothing about your domain. Ask it to write a passage about your internal service "Kestrel" and it writes generic bird facts or generic infrastructure prose. The probe lands in the wrong neighborhood entirely, and results get worse than the raw query. This is the most common HyDE failure in enterprise settings.
• The query is a short, exact lookup: an error code, a function name, a ticket ID. The entity is the entire signal, and wrapping it in a fluent paragraph dilutes it. This is especially damaging in hybrid setups where the BM25 leg rewarded the exact match.
• You are on a tight latency budget. HyDE is a full generation call, typically several hundred tokens, which is noticeably slower than a one-line rewrite.

Also be honest about the era you are operating in. HyDE was proposed against 2022 retrieval models. Current embedding models trained heavily on query-passage pairs have narrowed the asymmetry it exploits, so on some corpora HyDE now adds little over multi-query rewriting. That is not an argument against it, it is an argument for the evaluation harness at the end of this article.

Step-back prompting for questions that are too specific

Step-back prompting comes from "Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models" (Zheng et al., Google DeepMind). The retrieval version of the idea: users ask about instances, documents explain principles. Nobody writes a doc titled "why does my Postgres 16 planner skip the BRIN index on created_at when the table has 40M rows". Somebody did write "how the query planner estimates index costs". The step-back question is the bridge.

The implementation is a few-shot prompt that abstracts the question one level, then searches both the specific and the general question:

STEP_BACK_PROMPT = """Given a specific question, write one more general
step-back question about the underlying concept or mechanism needed to
answer it. Output only the step-back question.

Question: Why does my Postgres 16 planner skip the BRIN index on created_at?
Step-back: How does the Postgres query planner decide whether to use an index?

Question: Can I retry a Stripe PaymentIntent after a card_declined error?
Step-back: How do Stripe PaymentIntent states and retry rules work?

Question: {question}
Step-back:"""

def step_back_search(question: str, k: int = 8) -> list:
    general = llm(STEP_BACK_PROMPT.format(question=question))
    specific_hits = search(embed(question), k=k)
    general_hits = search(embed(general), k=k)
    return fuse([specific_hits, general_hits], top_n=k)

The few-shot examples are load-bearing. Write two or three from your own domain, because the right level of abstraction is domain-specific: too small a step and you retrieve the same chunks twice, too large ("how do databases work") and the general leg retrieves noise.

Step-back shines on debugging questions, policy and compliance questions ("can I store EU customer data in the Mumbai region" steps back to "what are the data residency rules per region"), and API misuse questions where the fix lives in a concepts page rather than the endpoint reference. Skip it for simple factual lookups, where the extra general query mostly adds latency.

Query decomposition for multi-hop questions

Single-vector retrieval quietly assumes one query has one intent. Comparison and aggregation questions break that assumption: "Compare pgvector and Qdrant for filtered vector search at 10M scale" produces an embedding that averages two products and a workload, and the average is closest to generic comparison content rather than the two specific documentation pages you actually need. Multi-hop questions ("what did the team that built our billing service choose for idempotency keys") are worse: the answer requires chaining two lookups.

Decomposition splits the question into self-contained sub-questions, retrieves for each independently, and fuses:

DECOMPOSE_PROMPT = """Break the question into 2 to 4 self-contained
sub-questions that can each be answered from documentation independently.
Every sub-question must make sense on its own with no pronouns referring
to the others. Output one sub-question per line, no numbering, no extras.

Question: {question}"""

def decompose_search(question: str, k_per_sub: int = 4, k: int = 8) -> list:
    raw = llm(DECOMPOSE_PROMPT.format(question=question))
    subs = [line.strip() for line in raw.splitlines() if line.strip()]
    result_lists = [search(embed(question), k=k)]
    for sub in subs:
        result_lists.append(search(embed(sub), k=k_per_sub))
    return fuse(result_lists, top_n=k)

For the pgvector versus Qdrant example, a good decomposition yields "How does pgvector handle metadata filtering with vector search?", "How does Qdrant handle filtered vector search?", and "What are the performance characteristics of pgvector and Qdrant at tens of millions of vectors?". Each sub-question is retrievable on its own, and the fused set contains chunks about both systems instead of whichever one the averaged vector happened to favor.

Two upgrades worth knowing. First, sequential decomposition: when sub-question two depends on the answer to sub-question one, you retrieve and answer them in order, feeding earlier answers into later queries. This is the pattern behind interleaved retrieval approaches like IRCoT and LlamaIndex's SubQuestionQueryEngine; implement it as a loop that alternates retrieve -> answer -> formulate next query. It is powerful and slow, so gate it behind a router rather than running it on everything. Second, answer-then-aggregate: answer each sub-question separately with its own retrieved context, then synthesize a final answer from the sub-answers. This keeps each generation call focused and avoids stuffing one context window with chunks about four different things.

Multi-query rewriting and reciprocal rank fusion

The workhorse pattern, and the one to implement first, is multi-query rewriting: generate a handful of paraphrases that vary the vocabulary, search all of them plus the original, and fuse. LangChain ships this as MultiQueryRetriever; the variant that popularized rank fusion for RAG is RAG-Fusion (Raudaschl).

MULTI_QUERY_PROMPT = """Generate 3 alternative search queries for the question
below. Vary the vocabulary: use synonyms, expand acronyms, and phrase one
version the way official documentation would phrase it. Keep all concrete
constraints (versions, names, error codes) in every variant.
Output one query per line, no numbering.

Question: {question}"""

def multi_query_search(question: str, k: int = 8) -> list:
    raw = llm(MULTI_QUERY_PROMPT.format(question=question))
    variants = [line.strip() for line in raw.splitlines() if line.strip()]
    result_lists = [search(embed(question), k=k)]
    result_lists += [search(embed(v), k=k) for v in variants]
    return fuse(result_lists, top_n=k)

All roads above end in fuse, so here it is. Reciprocal rank fusion (RRF) merges ranked lists using only ranks, no scores, which means you never have to normalize cosine similarities against BM25 scores or across differently-phrased queries:

def fuse(result_lists: list[list], k: int = 60, top_n: int = 8) -> list:
    scores: dict[str, float] = {}
    docs: dict[str, object] = {}
    for results in result_lists:
        for rank, doc in enumerate(results):
            docs[doc.id] = doc
            scores[doc.id] = scores.get(doc.id, 0.0) + 1.0 / (k + rank + 1)
    ranked = sorted(scores, key=scores.get, reverse=True)
    return [docs[doc_id] for doc_id in ranked[:top_n]]

The constant k=60 comes from the original RRF paper and is a fine default; it dampens the advantage of rank one over rank three so that a chunk appearing at moderate rank in several lists beats a chunk appearing once at the top. That property is exactly what you want for query variants: the chunks that survive every phrasing are the ones that are actually about the question.

RRF also merges heterogeneous backends. Fusing a BM25 list with a dense list gives you hybrid search with the same ten lines. And after fusion, a cross-encoder reranker (Cohere Rerank 3.5, or an open model like bge-reranker-v2) is the natural precision step: rewriting plus fusion buys recall, the reranker converts it into a clean top five.

Conversational query rewriting for chat RAG

Chat breaks retrieval in a way none of the techniques above address: the query is not even a question. After "How does your rate limiting work?" the user sends "and on the enterprise plan?". Embed that follow-up raw and you retrieve chunks about enterprise pricing, SSO, or nothing useful. The fix is condensing: rewrite the follow-up into a standalone question using the conversation history.

CONDENSE_PROMPT = """Given the conversation and a follow-up message, rewrite
the follow-up as one fully self-contained question. Resolve every pronoun
and implicit reference using the conversation. Keep entity names exactly as
written. If the follow-up starts a new topic, ignore the conversation and
return the follow-up cleaned up. Output only the rewritten question.

Conversation:
{history}

Follow-up: {question}
Rewritten:"""

def condense(question: str, history: list[tuple[str, str]], max_turns: int = 6) -> str:
    recent = history[-max_turns:]
    formatted = "\n".join(f"{role}: {text}" for role, text in recent)
    return llm(CONDENSE_PROMPT.format(history=formatted, question=question))

"and on the enterprise plan?" becomes "How does rate limiting work on the enterprise plan?", which retrieves correctly. Three gotchas from production. Limit history to the last few turns; feeding a fifty-turn transcript makes the rewriter latch onto stale topics and costs tokens for nothing. Instruct the model to detect topic switches, or every new question inherits ghosts of the old one. And run condensing before any other rewrite: it is the normalization step that makes HyDE, step-back and decomposition receive a well-formed question in the first place.

Choosing a strategy: route, do not stack

Stacking every technique on every query is the tempting failure mode: five LLM calls, fifteen vector searches, two seconds of added latency, and most of it wasted because most queries only need one transformation. The better shape is a router that classifies the query and picks one strategy.

Rules get you most of the way: if there is chat history, condense first. If the query contains an exact identifier (error code, function name, quoted string), skip semantic rewrites and lean on hybrid exact match. If it contains "compare", "vs", "difference between", or multiple question marks, decompose. Otherwise a tiny classifier call decides between lookup, how-to, and conceptual:

ROUTE_PROMPT = """Classify the search query as exactly one of:
lookup (a specific fact, value, name, or error code)
howto (how to accomplish a task)
conceptual (why something works or behaves the way it does)
multi (contains several distinct sub-questions)
Output only the label.

Query: {query}"""

A reasonable mapping: lookup -> original query plus one baseline rewrite over hybrid search; howto -> multi-query with RRF; conceptual -> step-back (and HyDE if your evals endorse it); multi -> decomposition. When unsure, default to multi-query plus RRF, which has the best cost-to-benefit ratio of the whole family and degrades gracefully.

Routing on a small model adds one cheap call, and you can skip even that by caching route decisions for repeated queries, which in support and docs traffic are a large fraction of volume.

Measuring query rewriting in a RAG pipeline

Every claim above is corpus-dependent, which is why the last component is not optional. HyDE genuinely wins on one corpus and genuinely loses on another; without measurement you are cargo-culting blog posts, including this one.

Build a golden retrieval set: 50 to 200 real queries pulled from logs, each labeled with the chunk IDs that answer it. Labeling is tedious and worth it; an afternoon of labeling pays for months of confident iteration. Then evaluate retrieval in isolation, before any generation metric, with recall at k and MRR:

def evaluate(strategy, golden: list[dict], k: int = 8) -> dict:
    recall_hits, rr_total = 0, 0.0
    for item in golden:
        results = strategy(item["query"], k=k)
        ids = [doc.id for doc in results]
        gold = set(item["relevant_chunk_ids"])
        if gold & set(ids):
            recall_hits += 1
            first = min(ids.index(i) for i in gold if i in ids)
            rr_total += 1.0 / (first + 1)
    n = len(golden)
    return {"recall@k": recall_hits / n, "mrr": rr_total / n}

for name, strat in [("raw", raw_search), ("multi", multi_query_search),
                    ("hyde", hyde_search), ("stepback", step_back_search)]:
    print(name, evaluate(strat, golden))

Read the failures, not just the aggregate. Sort by which queries each strategy uniquely fixes and uniquely breaks; that per-query diff is what tells you HyDE is losing exact-match lookups while winning conceptual questions, which is a routing insight, not a verdict.

If you have no labels yet, LLM-judged metrics like RAGAS context recall and context precision give a directional signal: weaker than labels, fine for choosing between two strategies, not fine for chasing single-point improvements. Graduate to labels as soon as the pipeline matters.

Production notes: latency, cost and caching

A few operational lessons that do not fit anywhere else:

• Use a small, fast model for rewrites. Rewrite quality saturates early; gpt-5-mini, Claude Haiku 4.5, or a self-hosted 8B model all produce fine rewrites, and the latency difference against a frontier model is user-visible.
• Parallelize against the rewrite. Fire the raw-query search immediately, generate rewrites concurrently, search the variants as they arrive, fuse whatever completed within your budget. On rewrite timeout, serve raw results. Rewriting should be a progressive enhancement, never a hard dependency.
• Cache rewrites keyed on the normalized query (plus last turn for chat). Docs and support traffic is heavily repeated, and a rewrite cache converts your most common queries to zero added latency.
• Log every rewrite next to its query and the retrieved IDs. Most retrieval bugs become obvious the moment you can read what the rewriter actually produced; it is the single most useful debug artifact in the pipeline.
• Guard against constraint drift. Rewriters add or drop versions, dates and product names when not explicitly told to preserve them. Keep the preservation instruction in every prompt and add a couple of eval cases that check for it.
• Remember the rewriter reads untrusted input. User text flows into your rewrite prompt, so keep that call toolless, treat its output as data, and never execute anything it produces.
• Prefer structured output (a JSON array of strings) for multi-query and decomposition once you are past prototyping; line-splitting is fine until a model decides to number its answers.

Start with multi-query plus RRF: it is twenty lines, cheap, and the most reliable single win. Add condensing the day you ship chat. Bring in HyDE, step-back and decomposition behind a router, each one justified by your own recall numbers. Keep the original query in every search, and keep the golden set growing from real logs. That is the whole discipline of query rewriting: translate the user into the corpus, measure, and let the evidence pick your techniques.

FAQ

What is query rewriting in RAG?

Query rewriting is a pre-retrieval step where an LLM transforms the user's raw query into one or more search queries that better match the corpus: fixing typos, expanding acronyms, generating paraphrases, writing a hypothetical answer (HyDE), abstracting the question (step-back), or splitting it into sub-questions (decomposition). The retrieved results from all query variants are merged, typically with reciprocal rank fusion.

Does HyDE still help with modern embedding models?

Sometimes. HyDE was designed when embedding models handled question-to-document similarity poorly; current retrieval-tuned models have narrowed that gap, so HyDE's edge is smaller and corpus-dependent. It still tends to help on formal reference corpora and conversational queries, and tends to hurt on exact-match lookups and domains the LLM knows nothing about. Measure recall at k on your own golden set before adopting it.

Should I use HyDE or multi-query rewriting?

Default to multi-query with reciprocal rank fusion: it is cheaper (short generations), more robust, and rarely regresses. Reach for HyDE when evals show a persistent register gap between casual questions and formal documents that paraphrasing does not close. Many production systems route: multi-query for how-to questions, HyDE or step-back for conceptual ones.

How much latency does query rewriting add?

One small-model call, so typically a few hundred milliseconds for short rewrites and more for HyDE's longer generations, plus the extra parallel vector searches, which are usually negligible next to the LLM call. You can hide most of it by searching the raw query immediately while rewrites generate, fusing what arrives in time, and caching rewrites for repeated queries.

Do I still need query rewriting if I use a reranker?

Yes, they fix different stages. A reranker improves precision by reordering the candidates retrieval returned; it cannot recover a document that was never retrieved. Query rewriting improves recall by changing what enters the candidate set. The strongest pipelines do both: rewrite and fuse for recall, then rerank the fused list for precision.

Can I fine-tune an embedding model instead of rewriting queries?

Fine-tuning on your own query-passage pairs attacks the same vocabulary mismatch at the model level and can reduce how much rewriting you need. It requires training data, an offline pipeline, and re-embedding your corpus on every model update, so most teams ship query rewriting first (it is runtime-only and reversible) and consider fine-tuning once query logs provide abundant training pairs. Decomposition and conversational condensing remain useful even with a fine-tuned model, since no embedding model fixes multi-intent or context-dependent queries.