Ragas for Multi-Hop Question Answering Evaluation

Why Multi-Hop QA Breaks Ordinary RAG Evaluation

Most RAG evaluation tutorials quietly assume a single-hop world: one question, one relevant chunk, one clean answer. Ask "What is the capital of France?" and a retriever fetches a chunk that says Paris, the generator repeats it, and every metric lights up green. That world does not match how people actually use knowledge assistants in production.

Real users ask questions like "Which supplier delivered the component that caused the Q3 recall, and did that supplier also work on the Q2 batch?" Answering that requires connecting at least two separate facts pulled from different documents, reasoning across them, and then producing a single coherent answer. This is multi-hop question answering, and it is where the majority of RAG systems quietly fail even while their single-hop benchmarks look great.

The failure mode is subtle because it does not look like a crash. The pipeline returns fluent, confident-sounding text. Retrieval "succeeds" in the sense that some relevant documents come back. The generator produces something plausible. But if the retriever only fetched the chunk about the Q3 recall and missed the chunk linking the supplier to Q2, the model either hallucinates the connection or quietly ignores half the question. Standard single-hop metrics — did we retrieve *a* relevant chunk, is the answer *faithful* to *some* context — do not catch this, because technically a relevant chunk was retrieved and the answer is faithful to it. The failure is structural: the reasoning chain broke, not any single retrieval or generation step.

This is exactly the gap Ragas is built to address once you move past its default single-hop assumptions. Ragas gives you the primitives — a knowledge-graph-driven test set generator, faithfulness and context metrics that decompose claims, and an extensible metric framework — to actually measure whether your pipeline is stitching together evidence correctly across hops, not just whether it "answered the question" in some superficial sense. In this article we will build a multi-hop evaluation harness end to end: constructing multi-hop test data, running a small multi-hop RAG pipeline, scoring it with both Ragas' built-in metrics and a purpose-built custom metric for hop-chain integrity, and interpreting what the scores actually tell you about where your system breaks.

What Makes a Question "Multi-Hop" in the First Place

Before writing any evaluation code, it's worth being precise about the taxonomy, because "multi-hop" gets used loosely and that looseness leaks into bad test sets.

• Bridge questions: the answer to sub-question A is required as an input to sub-question B. Example: "What year did the founder of Company X start the company, and what other company did they found the same year?" You must first resolve "founder of Company X," then use that entity to search for "same year, other company."
• Comparison questions: two independent facts are retrieved separately and then compared or combined. Example: "Did Product A or Product B ship first?" Here the two retrievals are parallel, not sequential, but the answer still requires synthesis across both.
• Intersection questions: the answer must satisfy constraints drawn from multiple documents simultaneously. Example: "Which employee worked on both the mobile app redesign and the payments migration?"
• Compositional questions: a single natural-language question that decomposes into a chain of three or more sub-questions, common in research and technical support domains.

Each of these stresses a RAG pipeline differently. Bridge questions stress the retriever's ability to do a "second pass" retrieval using an intermediate answer. Comparison and intersection questions stress the retriever's recall (you need *both* relevant chunks, not just one) and the generator's ability to hold multiple facts in context without conflating them. Compositional questions stress everything at once, plus the orchestration logic if you're using an agentic or iterative retrieval loop.

This matters for evaluation because a generic "multi-hop accuracy" number hides which failure mode you actually have. If your bridge questions score low but comparisons score fine, the problem is almost certainly your query reformulation or second-hop retrieval step. If comparisons score low, it's more likely a context window or generator synthesis problem. Building your test set with this taxonomy tagged from the start pays off enormously during error analysis later.

Building a Multi-Hop Test Set with Ragas

Ragas' testset generation module supports multi-hop synthesis through its knowledge graph abstraction. Instead of sampling a single document chunk and writing a question about it, the generator builds a graph of entities and relationships across your corpus and then synthesizes questions that require traversing more than one node.

Here's a minimal setup that builds a knowledge graph from a document set and generates multi-hop test cases:

from ragas.testset.graph import KnowledgeGraph, Node, NodeType
from ragas.testset.transforms import default_transforms, apply_transforms
from ragas.testset.synthesizers.multi_hop import (
    MultiHopAbstractQuerySynthesizer,
    MultiHopSpecificQuerySynthesizer,
)
from ragas.testset import TestsetGenerator
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

# Wrap your LLM and embedding model for Ragas
generator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))
generator_embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())

# Load your corpus as LangChain documents (loaders omitted for brevity)
documents = load_your_documents()

# Build a knowledge graph and enrich it with entity/relationship transforms
kg = KnowledgeGraph()
for doc in documents:
    kg.nodes.append(
        Node(type=NodeType.DOCUMENT, properties={"page_content": doc.page_content})
    )

transforms = default_transforms(
    documents=documents, llm=generator_llm, embedding_model=generator_embeddings
)
apply_transforms(kg, transforms)

# Configure the generator to favor multi-hop synthesizers
generator = TestsetGenerator(
    llm=generator_llm,
    embedding_model=generator_embeddings,
    knowledge_graph=kg,
)

query_distribution = [
    (MultiHopAbstractQuerySynthesizer(llm=generator_llm), 0.5),
    (MultiHopSpecificQuerySynthesizer(llm=generator_llm), 0.5),
]

testset = generator.generate(
    testset_size=50,
    query_distribution=query_distribution,
)

df = testset.to_pandas()
print(df[["user_input", "reference", "reference_contexts"]].head())

A few details matter here. The knowledge graph transforms (default_transforms) run entity extraction and relationship-building passes over your document set — this is what lets the synthesizer find pairs of nodes that share an entity or a semantic link, which is the raw material for a bridge or comparison question. MultiHopSpecificQuerySynthesizer tends to produce questions anchored to concrete named entities (people, products, dates), while MultiHopAbstractQuerySynthesizer produces more conceptual, thematic questions that span documents without needing a single shared entity. Mixing both gives you a test set that stresses your retriever differently — specific queries reward precise entity linking, abstract queries reward semantic recall.

Critically, each generated row includes reference_contexts — the actual source chunks the synthesizer used to construct the question and reference answer. This is gold data you would otherwise have to hand-label, and it becomes the ground truth against which you measure whether your pipeline's retrieval actually found the right hops.

Running Your Pipeline Against the Test Set

With test data in hand, the next step is running it through your actual RAG pipeline so you have real retrieved contexts and real generated answers to evaluate — not the synthetic reference ones. Ragas evaluation always compares your system's actual output against the reference, not the other way around.

from datasets import Dataset

def run_pipeline_on_testset(testset_df, rag_pipeline):
    records = []
    for _, row in testset_df.iterrows():
        question = row["user_input"]
        result = rag_pipeline.answer(question)  # your own retrieve+generate call

        records.append({
            "user_input": question,
            "retrieved_contexts": result["contexts"],
            "response": result["answer"],
            "reference": row["reference"],
            "reference_contexts": row["reference_contexts"],
        })
    return Dataset.from_list(records)

eval_dataset = run_pipeline_on_testset(df, my_multi_hop_rag_pipeline)

For a multi-hop pipeline specifically, rag_pipeline.answer() is usually doing more work than a single retrieve-then-generate call — it might be running an initial retrieval, extracting an intermediate entity, issuing a second retrieval query, and only then generating. Whatever that internal process looks like, make sure retrieved_contexts captures the union of everything actually fetched across all hops, in the order it was fetched. Losing the hop order or truncating to only the final retrieval will make faithfulness scoring misleading, because low-hop-order evidence often supplies the "connector" fact that lets the second hop's evidence make sense at all.

Core Ragas Metrics That Apply to Multi-Hop

Ragas' standard metric suite still applies to multi-hop QA, but they need to be read differently.

• Faithfulness decomposes the generated answer into individual claims and checks each against the retrieved context. In multi-hop answers, a single sentence often bundles two claims sourced from two different hops. A faithfulness score that dips slightly (rather than collapsing entirely) on multi-hop questions is expected and healthy — it means the metric is correctly catching that not every claim traces to a single chunk, without falsely flagging correctly-synthesized answers as hallucinated.
• Context Precision measures whether the retrieved contexts relevant to the reference are ranked highly. For multi-hop questions this is where you catch a retriever that fetches the first hop's evidence but buries or drops the second hop's evidence below your top-k cutoff.
• Context Recall measures whether all the reference-supporting context actually made it into the retrieved set at all. This is the single most important metric for multi-hop diagnosis: low context recall paired with a low overall answer score almost always means the retriever, not the generator, is your bottleneck.
• Answer Relevancy checks whether the generated answer actually addresses the question asked, independent of factual correctness. Multi-hop questions are long and compound, so it's common for a generator to answer only the first clause and drift off; Answer Relevancy catches partial answers that would otherwise look fine.

Here's a full evaluation run using these metrics:

from ragas import evaluate
from ragas.metrics import (
    Faithfulness,
    ContextPrecision,
    ContextRecall,
    AnswerRelevancy,
)
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))

results = evaluate(
    dataset=eval_dataset,
    metrics=[
        Faithfulness(llm=evaluator_llm),
        ContextPrecision(llm=evaluator_llm),
        ContextRecall(llm=evaluator_llm),
        AnswerRelevancy(llm=evaluator_llm),
    ],
)

scores_df = results.to_pandas()
print(scores_df[["user_input", "faithfulness", "context_recall", "context_precision"]])

Once you have scores_df, the analysis move that matters most is segmenting by question type. If you tagged your test set with the bridge/comparison/intersection/compositional taxonomy from earlier, group by that tag and look at mean context_recall per group. A pipeline that scores 0.91 recall on single-source questions but 0.58 on bridge questions has a specific, fixable problem: it isn't performing a second retrieval pass conditioned on the first hop's answer.

Writing a Custom Metric for Hop-Chain Integrity

The built-in metrics tell you *whether* evidence was retrieved and *whether* the answer is grounded in it, but they don't directly tell you whether the reasoning *chain* between hops is intact — whether the model correctly used the output of hop one as the input to hop two, rather than getting lucky and stating both facts independently without actually connecting them.

Ragas makes it straightforward to write a custom metric using its MetricWithLLM and prompt-based scoring pattern. Here's a custom hop-chain-integrity metric that asks an LLM judge to verify the logical connection explicitly, rather than just checking factual grounding:

from dataclasses import dataclass, field
from typing import Dict, Optional
from ragas.metrics.base import MetricWithLLM, SingleTurnMetric
from ragas.prompt import PydanticPrompt
from pydantic import BaseModel

class HopChainInput(BaseModel):
    question: str
    contexts: str
    answer: str

class HopChainOutput(BaseModel):
    chain_intact: bool
    broken_hop: Optional[str] = None
    reasoning: str

class HopChainPrompt(PydanticPrompt[HopChainInput, HopChainOutput]):
    instruction = (
        "You are auditing a multi-hop answer. Given the question, the retrieved "
        "contexts, and the generated answer, determine whether the answer correctly "
        "chains evidence across hops: does it use an intermediate fact from one "
        "piece of context as a bridge to reach a fact in another piece of context? "
        "If the answer states two facts side by side without actually connecting "
        "them the way the question requires, mark chain_intact as false and name "
        "the broken hop."
    )
    input_model = HopChainInput
    output_model = HopChainOutput

@dataclass
class HopChainIntegrity(MetricWithLLM, SingleTurnMetric):
    name: str = "hop_chain_integrity"
    _required_columns: Dict[str, set] = field(
        default_factory=lambda: {"user_input", "retrieved_contexts", "response"}
    )

    async def _single_turn_ascore(self, sample, callbacks) -> float:
        prompt = HopChainPrompt()
        result = await prompt.generate(
            data=HopChainInput(
                question=sample.user_input,
                contexts="\n\n".join(sample.retrieved_contexts),
                answer=sample.response,
            ),
            llm=self.llm,
            callbacks=callbacks,
        )
        return 1.0 if result.chain_intact else 0.0

    async def _ascore(self, row, callbacks) -> float:
        return await self._single_turn_ascore(row, callbacks)

Wire it into the same evaluate() call as the built-ins:

hop_chain_metric = HopChainIntegrity(llm=evaluator_llm)

results = evaluate(
    dataset=eval_dataset,
    metrics=[Faithfulness(llm=evaluator_llm), ContextRecall(llm=evaluator_llm), hop_chain_metric],
)

The value of this custom metric is that it separates two failures that faithfulness alone conflates. A pipeline can score high faithfulness (every claim traces to some retrieved chunk) while scoring low hop-chain integrity (the claims are never actually linked the way the question demands). That gap is the clearest signal you can get that your generator is pattern-matching fluent text rather than genuinely reasoning across the retrieved evidence — and it's invisible if you only run the default metric set.

Diagnosing Where the Chain Breaks

Once you have scores across the built-in metrics plus hop-chain integrity, segmented by question type, a diagnostic pattern usually emerges. It helps to think of it as a decision tree:

1. Low context recall on bridge questions, everything else fine — your first retrieval finds the right starting chunk, but nothing triggers a second retrieval using an intermediate entity. Fix: add an explicit query-decomposition or re-query step after the first hop, rather than relying on one-shot retrieval with a large top-k.
2. High context recall, low faithfulness — both hops' evidence made it into context, but the generator is still hallucinating connections not supported by either chunk. Fix: tighten the generation prompt to require citing which context each clause came from, or reduce context window clutter so the model isn't drowning the relevant chunks in noise.
3. High context recall, high faithfulness, low hop-chain integrity — this is the subtle one. The model states both facts correctly and doesn't hallucinate, but never actually performs the connecting inference the question asked for (e.g., it answers "the supplier is Acme Corp" and "Acme Corp also worked on Q2" as two disjoint statements without confirming they're the same causal chain). Fix: this is a prompting and possibly a reasoning-scaffold problem — consider chain-of-thought prompting that explicitly names the bridge entity, or an intermediate verification step before final answer synthesis.
4. Low answer relevancy on compositional questions specifically — the model is answering only part of a multi-part question. Fix: decompose compositional questions programmatically before generation, answer each sub-question, then synthesize, rather than asking the generator to handle decomposition and synthesis in a single pass.

Running this segmented analysis regularly, ideally as part of CI whenever you change retrieval logic, prompt templates, or the underlying model, turns multi-hop evaluation from a one-off audit into a regression safety net. It's easy for a change that improves single-hop accuracy (say, a reranker tuned on single-fact queries) to quietly regress multi-hop recall, because the two objectives are not perfectly aligned. Without a segmented multi-hop suite, that regression ships silently.

Practical Tips for Production Multi-Hop Evaluation

A few operational lessons that make this workflow sustainable rather than a one-time exercise:

• Keep hop counts explicit in your metadata. When you generate a multi-hop test set, store how many hops the synthesizer used (Ragas surfaces this via the node relationships in the knowledge graph). Aggregate scores by hop count (2-hop vs. 3-hop vs. 4-hop) in addition to question type — degradation is rarely linear, and 3+ hop questions often reveal problems 2-hop questions hide.
• Version your knowledge graph, not just your document corpus. If the underlying documents change, the entity and relationship graph Ragas builds changes too, which changes what multi-hop questions are even generatable. Re-running testset generation without pinning the graph version can make historical score comparisons meaningless.
• Use a stronger evaluator LLM than your production generator. Judging whether a hop chain is logically intact is a harder task than answering the question in the first place. If your evaluator model is weaker than or equal to your generator, it will systematically under-detect broken chains that a stronger judge would catch.
• Sample real user queries into your test set, not just synthetic ones. Synthetic multi-hop questions from Ragas' synthesizer are excellent for structural coverage, but real user phrasing is messier and often implicitly multi-hop without stating it as cleanly. Periodically mine production logs for multi-part questions and add them as a held-out set.
• Don't discard low-scoring traces — bucket them. A hop-chain integrity score of 0 with the reasoning field populated by the LLM judge is a debugging goldmine. Store the judge's reasoning text alongside the score so you can read *why* it thought the chain broke, not just that it did.

Closing Thoughts

Multi-hop question answering is where RAG systems earn or lose user trust, because these are exactly the questions users ask when a single search would have been too easy to bother with an AI assistant in the first place. Evaluating them well means moving past single-hop faithfulness and relevancy checks and building a test methodology that reflects the actual structure of the reasoning being demanded — bridge, comparison, intersection, and compositional questions each stress a different part of your pipeline, and a single aggregate score will hide which one is failing.

Ragas gives you the building blocks to do this properly: knowledge-graph-driven multi-hop test set synthesis so you're not hand-writing hundreds of bridge questions, a metric suite that, read correctly, separates retrieval failures from generation failures, and an extensible custom metric framework so you can measure things the framework doesn't ship with out of the box, like hop-chain integrity. The combination turns "our RAG demo looked good" into a defensible, repeatable claim about exactly which reasoning patterns your system handles and which ones it doesn't yet.

If you want to go deeper into building these evaluation harnesses hands-on, with real multi-hop datasets, custom metric design, and CI integration patterns, our Ragas Tutorial course on teachyou.ai walks through this exact workflow step by step, from knowledge graph construction to production-grade regression testing for RAG pipelines.