Multi-Hop RAG: Answering Questions That Need Several Documents

Multi-hop RAG is the set of techniques that let a retrieval-augmented generation system answer questions whose evidence is spread across multiple documents, where the second document you need only becomes obvious after reading the first. A single vector search returns the top-k chunks for one query, but "which university did the CEO of the company that acquired Instagram attend" requires resolving "who acquired Instagram," then "who is the CEO," then "where did they study," each step depending on the result of the last. Plain RAG cannot do this because it treats retrieval as one shot, not a chain. This article covers query decomposition, iterative (agentic) retrieval, graph-augmented retrieval, and how to evaluate whether your multi-hop pipeline is actually hopping correctly.

Why single-shot RAG fails on multi-hop questions

A standard RAG pipeline embeds the user's question, does one nearest-neighbor search over a vector index, stuffs the top-k chunks into a prompt, and asks the LLM to answer. This works well for questions where the answer sits in one chunk or one document. It fails on multi-hop questions for three concrete reasons.

First, the query and the final evidence chunk often share almost no vocabulary. If the question is "what programming language did the founder of the company that built the first iPhone jailbreak tool use," the chunk that actually contains the answer might just say "she wrote most of the tool in Objective-C." Nothing in that sentence matches "iPhone jailbreak" well enough to rank in the top-k against a large corpus, because the embedding of the question is dominated by "iPhone" and "founder," not the intermediate entity.

Second, single-shot retrieval has a fixed k. If the answer needs three separate facts from three separate documents, and your retriever returns eight chunks total, there is no guarantee all three needed chunks make the cut, especially when they compete against near-duplicates and tangentially related passages.

Third, and most importantly, multi-hop questions have an implicit dependency graph. You cannot even formulate the search query for hop two until you know the answer to hop one. Single-shot retrieval has no mechanism for that at all, it treats the original question as static.

You can see this fail in a quick test. Take any RAG system built with a naive "embed question, retrieve top-k, generate" loop and ask it a bridging question like "what award did the director of the movie that won best picture in [a specific year] win for their next film." If your corpus has the movie's Wikipedia-style page and the director's filmography page as separate documents, single-shot retrieval will usually only surface one of them, and the model either hallucinates the missing hop or gives a partial answer with false confidence.

Two families of multi-hop questions

Before building anything, it helps to separate the two shapes of multi-hop questions, because they call for different retrieval strategies.

Bridge questions need an intermediate entity resolved before the final search makes sense. "What year was the company that makes the operating system on the device Steve Jobs demoed in 2007 founded" is a bridge question: you must resolve "iPhone" as the device, then "iOS" or the manufacturer, before you can search for a founding year.

Comparison questions need facts from two or more independent documents that are then combined, without one depending on the other's identity. "Which is older, PostgreSQL or MySQL" needs two independent lookups and a comparison, not a chain.

Bridge questions require sequential, dependent retrieval. Comparison questions can often be parallelized: decompose into independent sub-queries, retrieve for each concurrently, then combine. Getting this distinction right up front changes your architecture, so the decomposition step described below should classify the question type, not just split it.

Approach 1: query decomposition with sequential retrieval

The most common production pattern is to have an LLM break the multi-hop question into an ordered list of sub-questions, then retrieve and answer each in sequence, feeding each answer forward into the next sub-question's context.

from openai import OpenAI

client = OpenAI()

DECOMPOSE_PROMPT = """Break the following question into an ordered list of
sub-questions needed to answer it. Each sub-question should be answerable
by a single document lookup. If a sub-question depends on the answer to
a previous one, write it using a placeholder like {answer_1}.

Question: {question}

Return a JSON list of strings, ordered by dependency."""

def decompose(question: str) -> list[str]:
    resp = client.chat.completions.create(
        model="gpt-4.1",
        messages=[{"role": "user", "content": DECOMPOSE_PROMPT.format(question=question)}],
        response_format={"type": "json_object"},
    )
    import json
    return json.loads(resp.choices[0].message.content)["sub_questions"]

With the sub-questions in hand, run them sequentially, substituting each resolved answer into the next sub-question before retrieving:

def answer_subquestion(sub_q: str, retriever, llm_answer_fn) -> str:
    chunks = retriever.search(sub_q, top_k=5)
    context = "\n\n".join(c.text for c in chunks)
    return llm_answer_fn(sub_q, context)

def multi_hop_answer(question: str, retriever, llm_answer_fn) -> str:
    sub_questions = decompose(question)
    answers = {}
    for i, sub_q in enumerate(sub_questions, start=1):
        filled = sub_q.format(**{f"answer_{j}": answers[j] for j in range(1, i)})
        answers[i] = answer_subquestion(filled, retriever, llm_answer_fn)

    final_prompt = (
        f"Original question: {question}\n\n"
        f"Resolved facts: {answers}\n\n"
        "Using only the resolved facts above, give the final answer."
    )
    return llm_answer_fn(final_prompt, context="")

The key detail is the placeholder substitution ({answer_1}). Without it, sub-question two is retrieved using the abstract phrasing from decomposition ("the company that acquired the app") instead of the concrete resolved entity ("Meta"), and you are back to the vocabulary mismatch problem from single-shot retrieval.

This pattern works well for bridge questions with two to four hops. It breaks down past four or five hops because decomposition errors compound: if hop two's sub-question is slightly wrong, every downstream hop inherits the mistake with no way to recover.

Approach 2: iterative agentic retrieval (retrieve, reason, retrieve again)

Instead of committing to a fixed decomposition up front, an agentic loop lets the model decide, after seeing each retrieval result, whether it has enough information or needs another search. This is closer to how a person actually researches: read something, notice a new name or term, search for that, repeat until satisfied.

def agentic_multi_hop(question: str, retriever, llm, max_hops: int = 5) -> str:
    history = []
    current_query = question

    for hop in range(max_hops):
        chunks = retriever.search(current_query, top_k=5)
        context = "\n\n".join(c.text for c in chunks)
        history.append({"query": current_query, "context": context})

        decision_prompt = f"""Original question: {question}

Retrieved so far:
{format_history(history)}

Do you have enough information to answer the original question fully?
Reply with JSON: {{"done": true, "answer": "..."}} if yes, or
{{"done": false, "next_query": "..."}} if you need another search.
The next_query should target the specific missing fact, using concrete
entity names you have already discovered, not vague references."""

        decision = llm.structured_call(decision_prompt)
        if decision["done"]:
            return decision["answer"]
        current_query = decision["next_query"]

    return llm.call(f"Answer as best you can from this research:\n{format_history(history)}\n\nQuestion: {question}")

The next_query instruction to use "concrete entity names you have already discovered" is doing the same job as the placeholder substitution in the decomposition approach, it forces the retrieval query to use terms that actually exist in the corpus rather than the abstract phrasing of the original question.

Agentic retrieval handles variable-depth questions better than fixed decomposition, because it does not have to guess the hop count up front. It costs more (one LLM call per hop for the decision step, plus the final synthesis call) and needs a hard max_hops cap, otherwise a model that is unsure will keep issuing searches indefinitely. In practice, cap at 4 to 6 hops for most knowledge-base use cases; anything needing more than that usually indicates the corpus is too fragmented and needs better chunking or a knowledge graph instead of pure vector hopping.

Approach 3: graph-augmented retrieval for structured multi-hop

When your corpus has clear entities and relationships (people, organizations, products, events), building a lightweight knowledge graph alongside your vector index lets you do multi-hop traversal directly, without relying on the LLM to guess the right search query at each step.

The pattern: extract (entity, relation, entity) triples during ingestion, store them in a graph store or even a simple adjacency table in Postgres, and use graph traversal for the bridge hops while falling back to vector search for the free-text answer at the final hop.

import networkx as nx

def build_graph_from_docs(docs, extract_triples_fn) -> nx.DiGraph:
    g = nx.DiGraph()
    for doc in docs:
        triples = extract_triples_fn(doc.text)
        for subj, relation, obj in triples:
            g.add_edge(subj, obj, relation=relation, source_doc=doc.id)
    return g

def graph_hop(g: nx.DiGraph, start_entity: str, relation: str) -> list[str]:
    results = []
    for _, target, data in g.out_edges(start_entity, data=True):
        if data["relation"] == relation:
            results.append(target)
    return results

A concrete flow for "what year was the company that makes the iPhone's OS founded": resolve iPhone -> runs_os -> iOS via the graph, then iOS -> made_by -> Apple, then either read Apple -> founded_year -> 1976 straight off the graph if you extracted that triple, or fall back to a vector search scoped to documents about Apple for "founding year" if you did not.

Graph-augmented retrieval is the most accurate approach for bridge questions because it sidesteps the vocabulary mismatch problem entirely: you are following an explicit edge, not hoping an embedding model notices the connection. The cost is upfront: triple extraction at ingestion time is another LLM pass over every document, and keeping the graph in sync with document updates is genuinely more engineering than a vector index alone. Use this approach when your domain has a small, well-defined entity schema (support tickets referencing customers and products, legal documents referencing parties and clauses, internal wikis referencing people and teams). Skip it for open-ended, loosely structured corpora where entity extraction would be too noisy to trust.

Combining approaches in practice

Production multi-hop RAG systems rarely pick just one of these. A workable default:

1. Classify the incoming question as single-hop, bridge, or comparison using a small, cheap LLM call before doing any retrieval at all. Route single-hop questions straight to standard RAG, skip the multi-hop machinery, and save latency and cost.
2. For comparison questions, decompose into independent sub-queries and retrieve them concurrently (they do not depend on each other, so there is no reason to serialize).
3. For bridge questions, use graph traversal where a graph exists for the entity type involved, and fall back to agentic iterative retrieval where it does not.
4. Always cap hop count, and always log the full chain of sub-queries and retrieved chunks. When a multi-hop answer is wrong, the chain is the only way to tell whether decomposition, retrieval, or final synthesis broke.

def route_and_answer(question: str, retriever, graph, llm) -> str:
    q_type = llm.classify(question, labels=["single_hop", "bridge", "comparison"])

    if q_type == "single_hop":
        chunks = retriever.search(question, top_k=5)
        return llm.answer(question, chunks)

    if q_type == "comparison":
        sub_qs = decompose(question)
        results = [answer_subquestion(sq, retriever, llm.answer) for sq in sub_qs]
        return llm.synthesize(question, results)

    return agentic_multi_hop(question, retriever, llm)

Evaluating a multi-hop RAG pipeline

Standard RAG evaluation metrics like answer relevance or single-chunk retrieval precision do not catch multi-hop failures, because a pipeline can retrieve one correct chunk out of three needed ones and still score reasonably on those metrics while producing a wrong final answer. Evaluate multi-hop RAG on the chain, not just the output.

Hop-level retrieval recall: for each sub-question in a labeled test set, check whether the retriever actually surfaced the document containing the needed fact, independent of whether the final answer was right. This isolates retrieval failures from reasoning failures.

Decomposition accuracy: for bridge questions, check whether the intermediate entity resolved at each hop matches ground truth. A wrong entity at hop one guarantees a wrong final answer even if every retrieval afterward is technically correct given the (wrong) query.

End-to-end exact match / F1: standard QA metrics on the final answer, but only meaningful once the two metrics above are passing, otherwise you cannot tell which stage to fix.

A public benchmark worth building your own eval harness around is HotpotQA, which was designed specifically for bridge and comparison multi-hop questions and ships with supporting-fact annotations, meaning you get ground truth for exactly which sentences in which documents were needed, not just the final answer string. Running your pipeline against a HotpotQA-style eval set with supporting-fact labels, before shipping, is the fastest way to catch a decomposition step that silently drops a hop.

def eval_hop_recall(pipeline, test_set) -> float:
    hits = 0
    total = 0
    for example in test_set:
        result = pipeline.run_with_trace(example["question"])
        retrieved_doc_ids = {c.doc_id for hop in result.hops for c in hop.chunks}
        for needed_doc_id in example["supporting_doc_ids"]:
            total += 1
            if needed_doc_id in retrieved_doc_ids:
                hits += 1
    return hits / total

FAQ

What is the difference between multi-hop RAG and agentic RAG? Agentic RAG is the broader pattern where an LLM decides when and what to retrieve, rather than retrieval being a fixed pre-generation step. Multi-hop RAG is a specific problem that agentic RAG (along with decomposition and graph traversal) is used to solve. Not all agentic RAG is multi-hop, an agent might just decide whether to retrieve at all for a single-hop question, and not all multi-hop RAG is agentic, fixed decomposition pipelines are multi-hop without any autonomous looping.

How many hops should I plan for? Most real-world knowledge-base questions that need more than one document top out at two or three hops. Design for a hard cap of four to six and treat anything that seems to need more as a sign the corpus needs restructuring (better cross-document linking, or a graph layer) rather than a sign you need a smarter agent loop.

Can I do multi-hop RAG without an LLM-based decomposition step? Yes, if your corpus has structured entity links (a graph, a citation network, explicit cross-references), you can traverse those directly without an LLM deciding the hops. This is faster and more reliable than LLM-driven decomposition wherever the structure already exists, use the LLM only for the free-text final synthesis.

Does a bigger context window make multi-hop RAG unnecessary? No. A larger context window lets you stuff more retrieved chunks into one prompt, but it does not solve the retrieval problem, the model still needs to find the right ten documents out of ten thousand before that budget is useful. Multi-hop techniques are about search strategy, not the eventual size of what you feed the model.

How do I handle multi-hop questions where a hop returns multiple valid candidates? Carry a small set forward instead of collapsing to one answer at each hop, then let the final synthesis step reconcile which candidate is consistent with the rest of the chain. For example, if hop one for "companies founded by [name]" returns three companies, retrieve for hop two against all three and let the last step filter using the constraint from the original question, rather than guessing wrong at hop one and never recovering.

Is graph-based retrieval always more accurate than agentic iterative retrieval? Only when your triple extraction is accurate and your schema actually covers the relations the questions need. A graph with sparse or noisy edges performs worse than a well-tuned agentic loop, because a missing edge is a silent dead end while an agentic loop can still fall back to a broader vector search. Treat the graph as an accelerator for well-covered entity types, not a universal replacement for vector-based hopping.