Testing RAG Pipelines End to End: A Practical Guide

Testing RAG pipelines is different from testing a normal API because two independent failure surfaces sit inside a single request: the retriever can fetch the wrong chunks, and the generator can hallucinate even when the right chunks are sitting right in front of it. A pipeline that "looks fine" in a demo can silently regress the moment you change an embedding model, re-chunk your documents, or bump a prompt template. This guide walks through a layered testing strategy for retrieval-augmented generation systems: unit tests for the retriever, unit tests for the generator, and end-to-end tests that score the full answer against a golden dataset, all wired into a CI pipeline so regressions get caught before they reach users.

Why RAG pipelines need a different testing approach

A typical web service test asserts on a deterministic output: given input X, expect output Y. RAG pipelines break that assumption twice over. First, the retriever returns a ranked list of chunks from a vector store, and "correct" often means "the right chunk is somewhere in the top K," not an exact match. Second, the generator is a language model, so the same prompt with the same context can produce differently worded (but equally correct) answers across runs.

This means testing RAG pipelines needs two kinds of assertions:

• Deterministic assertions for anything that doesn't touch the LLM: chunk counts, metadata filters, retrieval latency, whether a specific document ID appears in the top K results.
• Semantic assertions for anything downstream of the LLM: does the answer address the question, is it grounded in the retrieved context, does it avoid contradicting the source documents.

Treating both classes the same way is the single most common mistake teams make. Trying to string-match LLM output produces flaky tests that fail on harmless rewording. Trying to eyeball retrieval quality by reading transcripts doesn't scale past a handful of test cases. You need a golden dataset, a retrieval metric suite, and a generation metric suite, kept as three separate concerns.

Building a golden dataset for RAG evaluation

Before writing a single test, build a golden dataset: a set of question, expected-answer, expected-source pairs pulled from your real corpus. Fifty to a hundred examples is enough to start; you can grow it as you find edge cases in production.

Each row needs:

• question: the user query
• expected_chunk_ids: the document chunk IDs that should be retrieved (ground truth for retrieval tests)
• expected_answer: a reference answer (ground truth for generation tests)
• category: a tag like factual, multi-hop, out-of-scope, ambiguous so you can slice results later

# golden_dataset.py
GOLDEN_SET = [
    {
        "question": "What is the refund window for annual plans?",
        "expected_chunk_ids": ["billing-policy-004", "billing-policy-005"],
        "expected_answer": "Annual plans can be refunded within 30 days of purchase.",
        "category": "factual",
    },
    {
        "question": "Can I use the API if I'm on the free tier and also invited to a team workspace?",
        "expected_chunk_ids": ["pricing-002", "teams-011"],
        "expected_answer": "Free tier users get API access, and joining a team workspace does not change the free tier's API limits unless the team owner upgrades the workspace plan.",
        "category": "multi-hop",
    },
    {
        "question": "What's the weather in Tokyo today?",
        "expected_chunk_ids": [],
        "expected_answer": "I don't have information about that in the provided documents.",
        "category": "out-of-scope",
    },
]

Keep this dataset under version control next to your pipeline code. Every time a production query gets a wrong or embarrassing answer, add it here as a regression case. That's how the dataset earns its keep over time.

Testing the retriever in isolation

Retrieval tests should never touch the LLM. Call the retriever directly, get back a ranked list of chunk IDs, and score it against expected_chunk_ids using standard information-retrieval metrics.

# test_retriever.py
import pytest
from golden_dataset import GOLDEN_SET
from pipeline.retriever import Retriever

retriever = Retriever(collection="docs-prod-mirror", top_k=5)


def recall_at_k(retrieved_ids, expected_ids, k):
    if not expected_ids:
        return 1.0  # nothing expected to retrieve
    top = set(retrieved_ids[:k])
    hits = top.intersection(set(expected_ids))
    return len(hits) / len(expected_ids)


def reciprocal_rank(retrieved_ids, expected_ids):
    for rank, doc_id in enumerate(retrieved_ids, start=1):
        if doc_id in expected_ids:
            return 1 / rank
    return 0.0


@pytest.mark.parametrize("case", [c for c in GOLDEN_SET if c["expected_chunk_ids"]])
def test_recall_at_5(case):
    results = retriever.search(case["question"])
    retrieved_ids = [r.chunk_id for r in results]
    score = recall_at_k(retrieved_ids, case["expected_chunk_ids"], k=5)
    assert score >= 0.5, f"Recall@5 too low for: {case['question']!r} (got {score})"


def test_mean_reciprocal_rank():
    scores = []
    for case in GOLDEN_SET:
        if not case["expected_chunk_ids"]:
            continue
        results = retriever.search(case["question"])
        retrieved_ids = [r.chunk_id for r in results]
        scores.append(reciprocal_rank(retrieved_ids, case["expected_chunk_ids"]))
    mrr = sum(scores) / len(scores)
    assert mrr >= 0.6, f"MRR dropped to {mrr}, check embedding model or chunking"

Run this suite against a mirror of your production collection, not a synthetic toy dataset. Retrieval quality depends heavily on real-world chunk boundaries, metadata, and index configuration, so a small hand-built collection gives you false confidence.

Two things worth testing here beyond recall and MRR:

• Negative retrieval: for out-of-scope questions, assert that either nothing relevant comes back, or that a low similarity score triggers your "no answer" fallback before the query ever reaches the generator.
• Chunking regressions: snapshot the chunk count and average chunk length per document. A change to your chunking function (splitter size, overlap, separator logic) should trip a test, not silently ship.

def test_chunking_snapshot():
    from pipeline.chunker import chunk_document
    with open("fixtures/billing-policy.md") as f:
        text = f.read()
    chunks = chunk_document(text, max_tokens=512, overlap=64)
    assert len(chunks) == 7, "Chunk count changed, verify this is intentional"
    assert all(len(c.tokens) <= 512 for c in chunks)

Testing the generator in isolation

Once retrieval is trustworthy, test the generator with fixed, known-good context, not context pulled live from the retriever. This isolates generation bugs from retrieval bugs. If you feed the retriever's live output into generation tests, a failing test could mean either component broke, and you'll waste time debugging the wrong one.

The core generation properties to check are faithfulness (does the answer only use facts present in the context) and answer relevancy (does the answer address the question). Both require semantic judgment, so this is where an LLM-as-judge pattern earns its cost.

# test_generator.py
import pytest
from pipeline.generator import generate_answer
from judge import faithfulness_score, relevancy_score

FIXED_CONTEXT = {
    "What is the refund window for annual plans?": [
        "Annual plans can be refunded within 30 days of purchase, prorated after that.",
        "Monthly plans are refundable within 7 days.",
    ]
}


@pytest.mark.parametrize("question,context", FIXED_CONTEXT.items())
def test_faithfulness(question, context):
    answer = generate_answer(question=question, context=context)
    score = faithfulness_score(answer=answer, context=context)
    assert score >= 0.8, f"Answer not grounded in context: {answer!r} (score {score})"


@pytest.mark.parametrize("question,context", FIXED_CONTEXT.items())
def test_answer_relevancy(question, context):
    answer = generate_answer(question=question, context=context)
    score = relevancy_score(question=question, answer=answer)
    assert score >= 0.7, f"Answer drifted off-topic: {answer!r} (score {score})"

A minimal LLM-as-judge implementation just asks a second model to score the pair, with a structured output so you get a number back instead of prose you have to parse by hand.

# judge.py
import json
from anthropic import Anthropic

client = Anthropic()

FAITHFULNESS_PROMPT = """You are grading whether an answer is fully supported by the given context.
Context:
{context}

Answer:
{answer}

Score from 0.0 to 1.0 how well every claim in the answer is backed by the context.
Return only JSON: {{"score": <float>, "reason": "<short reason>"}}"""


def faithfulness_score(answer: str, context: list[str]) -> float:
    prompt = FAITHFULNESS_PROMPT.format(context="\n".join(context), answer=answer)
    response = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=200,
        messages=[{"role": "user", "content": prompt}],
    )
    result = json.loads(response.content[0].text)
    return result["score"]

Use a cheap, fast model as the judge, not your most expensive one. Judge calls run on every CI build, so cost and latency matter more than they do for the production generation model. Calibrate the judge once by hand-scoring twenty examples yourself and checking the judge agrees within a reasonable margin, then trust it going forward.

End-to-end testing: the full pipeline

Retrieval and generation tests catch component-level regressions, but they can both pass while the end-to-end pipeline still fails, because of glue-code bugs: wrong top-K passed to the prompt template, context truncated by a token limit, citations mapped to the wrong chunk ID. End-to-end tests run the golden dataset through the real pipeline, retriever and generator together, and score the final answer.

# test_e2e.py
import pytest
from golden_dataset import GOLDEN_SET
from pipeline.pipeline import RAGPipeline
from judge import faithfulness_score, answer_correctness_score

pipeline = RAGPipeline(collection="docs-prod-mirror")


@pytest.mark.parametrize("case", GOLDEN_SET)
def test_end_to_end_answer_quality(case):
    result = pipeline.run(case["question"])

    if case["category"] == "out-of-scope":
        assert result.answer.lower().startswith(
            ("i don't have", "i don't know", "i'm not able to find")
        ), f"Should have declined to answer: {result.answer!r}"
        return

    correctness = answer_correctness_score(
        question=case["question"],
        answer=result.answer,
        expected_answer=case["expected_answer"],
    )
    faithfulness = faithfulness_score(answer=result.answer, context=result.retrieved_texts)

    assert correctness >= 0.7, f"Low correctness ({correctness}) for: {case['question']!r}"
    assert faithfulness >= 0.8, f"Not grounded ({faithfulness}) for: {case['question']!r}"


def test_e2e_latency_budget():
    import time
    case = GOLDEN_SET[0]
    start = time.perf_counter()
    pipeline.run(case["question"])
    elapsed = time.perf_counter() - start
    assert elapsed < 5.0, f"Pipeline took {elapsed:.2f}s, exceeds budget"

Track per-category pass rates, not just an aggregate pass/fail. multi-hop questions almost always score lower than factual ones, and that's expected. What you want to catch is a category's score dropping between builds, not a single absolute number.

def test_category_score_report(capsys):
    from collections import defaultdict
    scores = defaultdict(list)
    for case in GOLDEN_SET:
        result = pipeline.run(case["question"])
        score = answer_correctness_score(
            question=case["question"],
            answer=result.answer,
            expected_answer=case["expected_answer"],
        )
        scores[case["category"]].append(score)

    with capsys.disabled():
        for category, values in scores.items():
            avg = sum(values) / len(values)
            print(f"{category}: {avg:.2f} avg over {len(values)} cases")

Wiring RAG tests into CI

RAG tests are slower and more expensive than unit tests, so split them into tiers and run them at different points in your pipeline.

• Pre-commit / PR checks: run the fast, deterministic retriever tests (recall, chunking snapshots) on every push. These need no LLM calls and finish in seconds.
• PR gate for pipeline changes: when a PR touches pipeline/, prompts/, or embedding config, run the full generator and end-to-end suites against the golden dataset. Budget for the judge-model calls; a 100-row golden set with two judge calls per row is 200 LLM calls, which is fine for Haiku-tier models but adds up fast on larger models.
• Nightly: run against a larger, held-out dataset (queries logged from production, redacted) so you catch drift that a curated golden set misses.

# .github/workflows/rag-tests.yml
name: rag-tests
on:
  pull_request:
    paths:
      - "pipeline/**"
      - "prompts/**"
jobs:
  retrieval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: pytest test_retriever.py -v

  generation-and-e2e:
    runs-on: ubuntu-latest
    needs: retrieval
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: pytest test_generator.py test_e2e.py -v
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Fail the build on retrieval regressions unconditionally, since those are deterministic and cheap to fix. For generation and end-to-end scores, consider a soft threshold with a required human review rather than a hard fail, since LLM output has natural variance across runs and a single bad seed shouldn't block a merge. Re-run flaky-looking failures once before treating them as real regressions.

Common failure modes to test for

A few RAG-specific bugs show up often enough that they deserve their own explicit test cases rather than waiting for the golden dataset to catch them by accident:

• Lost-in-the-middle: when the context window has many chunks, models tend to underweight information buried in the middle. Test with a case where the correct fact sits in chunk 3 of 8, not just chunk 1.
• Stale index: if documents are updated but embeddings aren't regenerated, retrieval returns outdated chunks with high confidence. Add a test that re-embeds a changed fixture document and asserts the new chunk ID appears in results, replacing the old one.
• Citation mismatch: if your pipeline shows source citations to users, test that the citation shown actually matches the chunk the claim came from, not just that a citation exists.
• Context window overflow silently truncating: assert on the actual token count sent to the generator, not just the number of chunks. A chunking change that produces larger chunks can silently drop the last few chunks off the end of the prompt.
• Empty retrieval handling: assert the pipeline returns a graceful fallback message rather than passing an empty context list straight to the generator, which often causes confident hallucination.

FAQ

What's the difference between testing a RAG pipeline and testing a plain LLM chatbot? A plain chatbot only has one failure surface: the generator. A RAG pipeline has two, retrieval and generation, and they need separate test suites with separate metrics. You can have perfect retrieval and a bad generation prompt, or a great generator fed the wrong chunks, and only isolated component tests tell you which one broke.

Do I need an LLM-as-judge, or can I use exact-match assertions? Exact-match works for retrieval (chunk IDs, counts, latency) but not for generation, since the same correct answer can be phrased many valid ways. Use a cheap, fast model as a judge for faithfulness and relevancy scoring, and reserve exact-match or regex assertions for structural checks like citation format or refusal phrasing.

How big should my golden dataset be? Start with 50 to 100 hand-picked question-answer-source triples covering your main query categories, then grow it every time a production query produces a wrong or embarrassing answer. Size matters less than coverage of edge cases: multi-hop questions, out-of-scope questions, and ambiguous phrasing tend to catch more bugs than another ten straightforward factual questions.

Should retrieval tests run against a live vector database or a mock? Run them against a mirrored copy of your real production collection. Mocking the vector store gives you fast, deterministic tests, but it can't catch chunking regressions, embedding model drift, or index configuration issues, which are the most common real-world causes of retrieval failures.

How do I keep judge-model costs under control in CI? Use the smallest model that reliably agrees with your own hand-scoring on a calibration set, cap the golden dataset size for per-PR runs, and reserve larger held-out datasets for a nightly job instead of every pull request. Batch judge calls where your SDK supports it to cut per-request overhead.

What metrics matter most for retrieval versus generation? For retrieval: recall@K and mean reciprocal rank (MRR), since you mainly care whether the right chunk showed up and how high it ranked. For generation: faithfulness (is the answer grounded in the retrieved context) and answer correctness or relevancy (does it actually answer the question), since a fluent but ungrounded answer is a hallucination even if it reads well.