RAG Query Rewriting and HyDE: Improving Retrieval Before You Search

The Query You Get Is Rarely The Query You Need

Every RAG tutorial starts the same way: embed the documents, embed the query, do a cosine similarity search, stuff the top-k chunks into a prompt, call it a day. It works great in the demo. Then you ship it, real users start typing real questions, and retrieval quality falls off a cliff. The user types "why is my bill so high this month," and your vector store — full of neatly written documentation chunks about "usage-based pricing tiers" and "overage calculation methodology" — returns nothing useful, because the *words* don't match even though the *intent* does.

This is the dirty secret of most RAG pipelines that underperform in production: the retrieval step is not broken because the embedding model is bad or the chunking strategy is wrong. It's broken because nobody thought about the query side of the equation. We obsess over chunk size, overlap, and reranking, but the query — the thing that actually drives what gets retrieved — gets treated as immutable, sacred input that must be embedded exactly as typed.

It doesn't have to be. Query rewriting and HyDE (Hypothetical Document Embeddings) are two techniques that treat the user's query as a draft, not a final answer, and transform it into something that retrieves better *before* it ever touches your vector index. Neither technique requires retraining anything. Both can be added to an existing RAG pipeline in an afternoon. And both consistently move the needle on retrieval quality in ways that better rerankers alone cannot.

This article walks through why raw queries fail, how query rewriting and HyDE work mechanically, when to use each, and how to combine them into a single pre-retrieval pipeline you can drop into production.

Why Raw User Queries Make Bad Search Queries

Think about the mismatch between how users ask questions and how your source documents are written. Users are conversational, underspecified, and often assume context you don't have. Documents are declarative, technical, and self-contained. A few concrete failure patterns:

• Vocabulary mismatch. The user says "it crashed," the docs say "the process terminated with a non-zero exit code." Semantically related, but embedding models — especially smaller, cheaper ones — don't always bridge that gap well, particularly for short queries with little surrounding context.
• Underspecified intent. "How do I fix the auth error" could mean a dozen different things depending on which auth provider, which SDK version, which error code. The query alone doesn't carry enough signal.
• Compound questions. "What's the difference between the Pro and Enterprise plans and can I downgrade later?" is really two or three retrieval queries jammed into one sentence. A single embedding of that whole string retrieves mediocre results for both sub-questions.
• Conversational reference. In a multi-turn chat, "what about the second one?" is meaningless to a retriever without resolving what "the second one" refers to from prior turns.
• Query length asymmetry. Short queries (3-6 words) produce weak, diffuse embeddings. Long, well-formed documents produce dense, specific embeddings. Comparing a 4-word query against a 400-word chunk in the same vector space is inherently lossy — this asymmetry is one of the core motivations behind HyDE, which we'll get to shortly.

None of these are retriever bugs. They're a mismatch between the query distribution and the document distribution. The fix isn't a better embedding model — it's transforming the query so it looks more like the thing you're trying to find.

Query Rewriting: Let an LLM Clean Up the Question First

Query rewriting is exactly what it sounds like: before embedding the user's query and hitting the vector store, you pass it through an LLM (usually a small, fast, cheap one) that rewrites it into a form better suited for retrieval. This can mean several different things depending on what's broken:

1. Disambiguation and context resolution — resolving pronouns and implicit references using conversation history, so "what about the second one" becomes "what is the refund policy for the Enterprise plan."

2. Expansion — adding likely synonyms or related terminology so vocabulary mismatch matters less. "it crashed" might become "application crash error termination exception."

3. Decomposition — splitting a compound question into multiple independent sub-queries that each retrieve cleanly, then merging results.

4. Normalization — stripping conversational filler ("hey so I was wondering, kind of urgently, whether...") down to the actual informational need.

Here's a minimal implementation using an LLM call for rewriting before retrieval:

from openai import OpenAI

client = OpenAI()

REWRITE_PROMPT = """You rewrite user questions into clear, standalone search queries
for a document retrieval system. Rules:
- Resolve any pronouns or vague references using the conversation history.
- If the question contains multiple distinct asks, split into separate queries.
- Remove conversational filler, keep only the informational content.
- Output ONLY the rewritten query/queries, one per line. No explanations.

Conversation history:
{history}

User question: {question}
"""

def rewrite_query(question: str, history: str = "") -> list[str]:
    prompt = REWRITE_PROMPT.format(history=history or "None", question=question)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
    )
    lines = response.choices[0].message.content.strip().split("\n")
    return [line.strip("- ").strip() for line in lines if line.strip()]


# Example
history = "User: I'm on the Pro plan.\nAssistant: Got it, Pro plan noted."
question = "what about the second one and can I downgrade later?"

queries = rewrite_query(question, history)
print(queries)
# ['What is the Enterprise plan compared to the Pro plan?',
#  'Can I downgrade from the Pro plan to a lower plan later?']

Notice what happened here: a single vague, conversationally-dependent question turned into two clean, standalone retrieval queries. You now run retrieval twice (or in parallel), get candidate chunks for each, and merge/dedupe before passing to the generation step. This alone tends to fix a huge fraction of "the chatbot didn't understand what I was asking" complaints in production RAG systems, because most of those complaints are actually retrieval failures wearing a generation-failure costume.

A cheaper variant that doesn't need an LLM call at all: maintain a small synonym/expansion dictionary for domain-specific jargon and do rule-based expansion. This is less flexible but adds zero latency and zero cost, and it's worth doing for high-frequency query patterns you can identify from your logs.

HyDE: Search With the Answer, Not the Question

Query rewriting fixes the query. HyDE takes a stranger, more powerful approach: it doesn't try to make the query better — it replaces the query with something else entirely for the purposes of embedding.

The idea, from the paper "Precise Zero-Shot Dense Retrieval without Relevance Labels" (Gao et al.), is disarmingly simple. Instead of embedding the user's question and searching for similar document chunks, you:

1. Ask an LLM to write a hypothetical answer to the question — a fake, possibly factually wrong, document-shaped piece of text that *looks like* what a real answer would look like.
2. Embed that hypothetical answer instead of the original question.
3. Use that embedding to search your vector store.

Why does this work? Remember the length/density asymmetry problem from earlier — short queries embed poorly against long documents because they don't share enough structural and lexical similarity. A hypothetical answer, even a wrong one, is written in the same register, length, and style as your actual documents. It uses the same kind of vocabulary a real answer would use, because the LLM has seen enough text in this domain (or is told to write in this domain's style) to approximate it. So instead of comparing "why is my bill so high" against "usage-based pricing tiers and overage calculation," you're comparing an LLM-generated paragraph about billing overages against your actual documentation — document-to-document similarity, which embedding models are generally much better at than query-to-document similarity.

Here's a working HyDE implementation:

from openai import OpenAI
import numpy as np

client = OpenAI()

HYDE_PROMPT = """Write a short, factual passage that would answer the following
question, as if it were an excerpt from official product documentation.
Do not mention that this is hypothetical. Just write the passage.

Question: {question}
"""

def generate_hypothetical_document(question: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": HYDE_PROMPT.format(question=question)}],
        temperature=0.3,
    )
    return response.choices[0].message.content.strip()


def embed(text: str) -> list[float]:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text,
    )
    return response.data[0].embedding


def hyde_retrieve(question: str, vector_store, top_k: int = 5):
    hypothetical_doc = generate_hypothetical_document(question)
    query_vector = embed(hypothetical_doc)
    results = vector_store.similarity_search_by_vector(query_vector, k=top_k)
    return results, hypothetical_doc


# Example
question = "why is my bill so high this month?"
hypothetical_doc = generate_hypothetical_document(question)
print(hypothetical_doc)
# "Your monthly bill may increase due to usage-based overage charges when
# your account exceeds the included quota for API calls, storage, or
# compute minutes in your current billing cycle. Overage rates are applied
# per unit beyond the plan's included allowance, and are itemized separately
# on your invoice under 'Additional Usage Charges.'"

Notice that the hypothetical document doesn't need to be *correct* — it just needs to sound like the kind of text that would live in your knowledge base. If it happens to hallucinate a specific number or a wrong policy detail, that's fine, because you never show this hypothetical document to the user. It's a retrieval instrument, not an answer. It gets embedded, used to find real chunks, and then discarded. The real chunks it retrieves go on to ground the actual generation step.

You can even combine multiple hypothetical documents for the same question (generate 3-5 with higher temperature, average their embeddings) to make retrieval more robust to any single hallucination steering the search in a bad direction — this is literally what the original HyDE paper does, calling it an ensemble of hypothetical answers.

When to Use Query Rewriting vs. HyDE

These two techniques solve overlapping but distinct problems, and the right call depends on your failure mode:

• Use query rewriting when your main issue is conversational context, ambiguity, or compound questions. If users are chatting with your system in a multi-turn interface and your retrieval quality degrades specifically after turn 2 or 3, that's a rewriting problem, not a HyDE problem.
• Use HyDE when your main issue is vocabulary/register mismatch between short factual questions and long, formal documents. If single-turn factual questions ("what's the refund window") consistently miss chunks that clearly contain the answer, that's an embedding asymmetry problem, and HyDE targets it directly.
• Use both when you have a conversational assistant over a large, jargon-heavy knowledge base — which describes the majority of production RAG systems people actually build. Rewrite first to resolve context and split compound asks, then run HyDE on each resolved sub-query before embedding.

A practical combined pipeline looks like this:

def rag_pretrieval_pipeline(question: str, history: str, vector_store):
    # Step 1: rewrite ambiguous/compound queries into clean sub-queries
    sub_queries = rewrite_query(question, history)

    all_results = []
    for sub_q in sub_queries:
        # Step 2: generate a hypothetical answer for each sub-query
        hypothetical_doc = generate_hypothetical_document(sub_q)

        # Step 3: embed the hypothetical doc, not the raw sub-query
        query_vector = embed(hypothetical_doc)

        # Step 4: retrieve using that embedding
        results = vector_store.similarity_search_by_vector(query_vector, k=5)
        all_results.extend(results)

    # Step 5: dedupe and rerank the merged candidate pool
    return deduplicate_and_rerank(all_results, question)

This adds two LLM calls per sub-query before you've even hit the generation step. That's a real cost — both in latency and in dollars — so it's worth being deliberate about where you apply it.

The Cost and Latency Tradeoff Nobody Mentions

Every pre-retrieval transformation is an extra network round-trip to an LLM before you even start the "real" work of generation. If your baseline RAG latency is 2 seconds, adding a rewriting call and a HyDE call in sequence can easily add another 1-2 seconds, especially if you're chaining them rather than running them concurrently where possible.

A few practical mitigations:

• Use small, fast models for the pre-retrieval steps. You don't need your most capable model to rewrite a query or draft a hypothetical paragraph. A distilled or mini-tier model does this well and cheaply — save your frontier model budget for the final generation step where quality actually matters most to the user.
• Cache aggressively. Many user questions repeat, especially in support and internal-tools contexts. Cache the rewritten query and the hypothetical document keyed on a normalized version of the input question, so repeat questions skip the LLM calls entirely.
• Skip HyDE for queries that are already long and well-formed. If the incoming query is already 20+ words and reads like a document excerpt, HyDE adds cost without much benefit — the asymmetry problem it solves mostly doesn't apply. A simple length/heuristic check before invoking HyDE saves a meaningful fraction of calls.
• Run rewriting and embedding of the original query in parallel as a fallback. Retrieve using both the raw query and the transformed query, then merge and rerank. This hedges against the rewriting or HyDE step making things worse for a given edge case, which does happen occasionally — HyDE's hypothetical document can sometimes drift the embedding toward an irrelevant cluster if the LLM badly misunderstands the question.

None of this is free, and that's the honest tradeoff: you're spending extra inference cost and latency to buy retrieval precision. For high-stakes RAG systems — legal research, medical information, internal support tools where a wrong answer is expensive — that trade is almost always worth it. For a low-stakes FAQ bot, it might not be.

Evaluating Whether It's Actually Helping

Don't ship query rewriting or HyDE on faith. Build a small retrieval evaluation set: 30-50 real user questions (pulled from logs if you have them) paired with the chunk IDs that should be retrieved for each. Then measure retrieval metrics — recall@k and mean reciprocal rank are the two most useful — with and without each transformation:

def evaluate_recall_at_k(eval_set, retrieve_fn, k=5):
    hits = 0
    for question, expected_chunk_ids in eval_set:
        retrieved = retrieve_fn(question, top_k=k)
        retrieved_ids = {r.chunk_id for r in retrieved}
        if retrieved_ids & set(expected_chunk_ids):
            hits += 1
    return hits / len(eval_set)

baseline_recall = evaluate_recall_at_k(eval_set, plain_retrieve, k=5)
rewrite_recall = evaluate_recall_at_k(eval_set, rewrite_then_retrieve, k=5)
hyde_recall = evaluate_recall_at_k(eval_set, hyde_retrieve, k=5)

print(f"Baseline: {baseline_recall:.2f}")
print(f"With rewriting: {rewrite_recall:.2f}")
print(f"With HyDE: {hyde_recall:.2f}")

This is the only way to know whether these techniques are actually earning their latency and cost budget on *your* corpus and *your* query distribution, rather than assuming the paper results transfer directly. HyDE, in particular, was validated on broad web-scale retrieval benchmarks — it doesn't automatically follow that it helps on a narrow, highly technical internal knowledge base where the vocabulary gap between questions and documents is smaller to begin with. Measure before you commit compute budget to it in production.

Also watch for silent failure modes: if your hypothetical document generator starts producing answers that contradict your actual documentation on edge cases, and those contradictions happen to sit closer in embedding space to the *wrong* chunks, HyDE can actively hurt retrieval for those questions. This is rare but worth spot-checking on your eval set rather than assuming monotonic improvement.

Putting It Together in a Production Pipeline

The pattern that tends to work best in real systems isn't "always rewrite, always HyDE" — it's a lightweight router in front of the retrieval step:

1. Classify the incoming query as single-turn/simple, multi-turn/ambiguous, or compound.
2. Route ambiguous and compound queries through rewriting; route already-clean single-turn queries straight to retrieval.
3. For any query shorter than a threshold (say, under 12 words) or that scored poorly on a previous retrieval attempt, apply HyDE before embedding.
4. Merge results from the raw-query retrieval and the transformed-query retrieval, rerank with a cross-encoder, and take the top-k for generation.
5. Log every transformation (original query, rewritten query, hypothetical document, retrieved chunk IDs) so you can build your eval set from real production traffic over time.

This keeps costs proportional to actual difficulty instead of paying the rewriting-plus-HyDE tax on every single query, including the easy ones that would have retrieved fine anyway.

Wrapping Up

Retrieval quality is the ceiling on everything a RAG system can do — no amount of prompt engineering on the generation side fixes a system that retrieved the wrong chunks in the first place. Query rewriting and HyDE both attack the problem at its actual source: the mismatch between how people ask questions and how your documents are written. Rewriting cleans up ambiguity, context, and compound asks. HyDE closes the query-document asymmetry gap by searching with a hypothetical answer instead of a bare question. Neither is exotic — both are a single LLM call away from your existing pipeline, and both are measurable with a straightforward recall@k eval you can build in an afternoon.

If you're new to these concepts or want the fuller picture of how retrieval, chunking, embeddings, and generation fit together before diving deeper into pre-retrieval optimization, our Introduction to RAG course on teachyou.ai covers the foundations this article builds on, with hands-on labs you can run alongside the lessons.