How to Evaluate a RAG Pipeline: Faithfulness, Recall and Precision Explained

Why "it looks right" is not an evaluation strategy

You built a RAG pipeline. You typed a few questions into it, the answers looked plausible, and you shipped it. Then a user asked something slightly out of distribution, the retriever pulled the wrong chunk, and the model confidently answered using information that was not in that chunk at all. Nobody noticed until a customer did.

This is the default failure mode for teams building retrieval-augmented generation systems. RAG has two independent subsystems — a retriever and a generator — and each one fails differently. The retriever can miss the right document, or it can pull in ten chunks when only one was relevant. The generator can ignore perfectly good context and hallucinate anyway, or it can answer a question that the context never actually addressed. If you only look at the final answer, you cannot tell which subsystem broke, and you definitely cannot tell if last week's prompt change quietly made things worse.

Evaluating RAG properly means measuring retrieval and generation separately, building a test set you trust, and running that test set automatically so regressions get caught before a deploy — not after a support ticket. This article walks through exactly how to do that: the metrics that matter, how to build and grow a golden dataset, how to generate synthetic eval data at scale, how to specifically catch hallucination, and how to wire the whole thing into CI so a regression blocks the merge instead of blocking your inbox.

The two halves of a RAG system need two different scorecards

Before touching any metric, internalize this split: retrieval-side metrics tell you whether the pipeline found the right information. Generation-side metrics tell you whether the model used that information correctly. A RAG system can score perfectly on one axis and fail completely on the other.

Consider two failure cases:

• The retriever pulls back three irrelevant chunks and misses the one paragraph that actually answers the question. The generator, doing its best with bad inputs, produces a vague or wrong answer. This is a retrieval failure.
• The retriever does its job and returns the exact right chunk. The generator ignores it and answers from parametric memory, stating something that sounds correct but was never in the retrieved context. This is a generation failure — specifically, hallucination.

If you only track "was the final answer correct," both cases look the same: wrong answer. You'll spend a week tuning your prompt to fix a retrieval bug, or re-indexing your vector store to fix a prompting bug. Splitting the scorecard is what makes debugging tractable.

Retrieval-side metrics: context precision and context recall

Retrieval metrics operate on the set of chunks your retriever returned for a given query, compared against the chunks that should have been returned. To measure them you need, for each test question, a labeled set of "ground truth" relevant chunks or documents.

Context recall answers: of all the information needed to answer the question correctly, how much did the retriever actually surface? If the answer requires facts from two paragraphs and your retriever only returned one of them, recall is low even if everything it returned was accurate. Low context recall is diagnosed by looking at whether the ground-truth answer can be fully reconstructed from the retrieved chunks alone. In practice, this is often computed by breaking the reference answer into individual claims and checking whether each claim is attributable to something in the retrieved context.

Context precision answers: of the chunks the retriever returned, how many were actually relevant and useful, and were the relevant ones ranked near the top? A retriever that returns ten chunks where only one is relevant has low precision, even if that one relevant chunk technically means recall is fine. Precision matters because irrelevant chunks are not free — they eat context window budget, dilute attention, and increase the odds that the generator latches onto the wrong passage. Ranking matters too: if the relevant chunk is buried at position eight out of ten, many generators effectively never see it, or see it after they've already "decided" on an answer from earlier chunks.

A useful mental model: recall is about not missing things, precision is about not including junk. You typically tune retrieval depth (how many chunks, top_k) and reranking (a second-pass model that reorders retrieved chunks by relevance) to trade off between them. Pulling more chunks tends to raise recall and lower precision. A good reranker raises precision without hurting recall, which is why reranking is one of the highest-leverage additions to a mediocre RAG pipeline.

Both metrics require ground truth — a mapping from query to the chunk(s) that should be retrieved. This is exactly what your golden test set needs to contain.

Generation-side metrics: faithfulness and answer relevancy

Once you know what the retriever handed the generator, the next question is what the generator did with it.

Faithfulness (also called groundedness) measures whether every claim in the generated answer is supported by the retrieved context. This is the single most important metric for catching hallucination in a RAG system, because it isolates the generator's behavior from whether the retrieval was any good in the first place. A common way to compute faithfulness: decompose the generated answer into individual factual statements, then check each statement against the retrieved context — is it directly supported, contradicted, or simply absent? The faithfulness score is roughly the fraction of statements that are supported. An answer can be faithful and still wrong, if the retrieved context itself was wrong or insufficient — which is exactly why you need both retrieval and generation metrics together, not either alone.

Answer relevancy measures whether the generated answer actually addresses the question that was asked, independent of whether it's grounded in the context. A model can produce a perfectly faithful answer — every sentence backed by a retrieved chunk — while dodging the actual question, padding with tangential facts, or answering a related-but-different question. One common technique: ask an LLM to generate several candidate questions that the given answer would be a good response to, then compare those generated questions against the original question using embedding similarity. If the generated answer is on-topic and specific, the reverse-engineered questions will closely match the original.

Put together, you get four numbers per test case: context recall, context precision, faithfulness, answer relevancy. A healthy pipeline is strong on all four. A pipeline that's faithful but has low answer relevancy is giving grounded non-answers. A pipeline with high context recall but low faithfulness is hallucinating despite having the right information in front of it — often the most damaging failure mode because it's the hardest one for a user to catch.

Building a golden test set that you actually trust

None of the above metrics mean anything without a golden test set — a curated collection of (query, expected chunks, reference answer) triples that represents real usage. This is the part teams skip, and it's the part that determines whether your evals catch real regressions or just produce numbers nobody checks.

Start with real user queries, not invented ones. If you have any production traffic, logs, or a support inbox, mine it. Real queries are messier than the ones engineers write for themselves — they're underspecified, they use different terminology than your docs, they sometimes ask two questions at once. That messiness is exactly what you want to test against, because it's what your pipeline will actually face.

For each query in your golden set, capture:

• The question, verbatim, including its natural phrasing and ambiguity.
• The specific source chunk(s) or document sections that contain the answer — this is your retrieval ground truth.
• A reference answer written or reviewed by a subject-matter expert, not by the same model you're evaluating.
• Metadata tags: difficulty, category, whether it's a single-hop question (answerable from one chunk) or multi-hop (requires synthesizing several chunks).

Multi-hop questions deserve special attention. Most RAG failures in production are not simple lookup failures — they're cases where the answer requires combining facts from two or three different places, and the retriever either misses one of them or the generator fails to synthesize across chunks. If your golden set is 100% single-hop questions, your eval scores will look great and your production incidents will keep happening.

Aim for at least 50-100 examples to start, covering your major query categories, with deliberate inclusion of edge cases: questions with no good answer in the corpus (to test whether the model correctly says "I don't know" instead of hallucinating), ambiguous questions, and questions where the corpus contains outdated or conflicting information. Keep the golden set under version control alongside your code. It should evolve — every production bug that gets reported should, after the fix, become a new entry in this set, so you never regress on the same failure twice.

Synthetic data generation to scale your eval set

Hand-curating 50-100 examples gets you a foundation, but it does not give you the volume needed to catch narrower regressions, and it's slow to refresh every time your knowledge base changes. This is where synthetic test-set generation earns its keep.

The core technique: feed chunks of your actual document corpus to an LLM and ask it to generate question-answer pairs that are answerable from that specific chunk. Because you control which chunk generated the question, you get retrieval ground truth for free — you already know which chunk is "correct" for that question, since it was the source. A well-designed synthetic generation pipeline typically produces several question types deliberately, rather than uniform simple questions:

• Simple factual questions answerable directly from one chunk.
• Reasoning questions that require connecting two facts within the same chunk or across adjacent chunks.
• Multi-context questions that are only fully answerable by pulling from multiple, possibly non-adjacent, chunks — these specifically stress-test context recall.
• Conditional or comparative questions ("what changed between version X and version Y") that require the model to notice differences rather than just retrieve facts.
• Distractor questions where the corpus contains a plausible-but-wrong answer nearby, testing whether the generator gets pulled toward the wrong chunk.

A practical workflow: sample a representative slice of your corpus (don't just use the first N chunks — stratify across document types, sections, and recency), generate 3-5 candidate question types per chunk, then run a lightweight filtering pass — either a human reviewer or a second LLM call — to discard questions that are ambiguous, unanswerable, or trivially gameable. Synthetic data is a volume multiplier for your hand-curated set, not a replacement for it. Treat model-reviewed synthetic examples as a lower-confidence tier in your test set and weight or separate them accordingly when you report scores — a regression in the human-curated tier should carry more weight than one in the synthetic tier.

Refresh your synthetic set whenever the underlying corpus changes meaningfully — new product docs, a policy update, a new content vertical — since stale synthetic questions against an outdated corpus snapshot will silently stop testing what actually matters.

Catching hallucination specifically, not just generally

"Hallucination" gets used loosely, but for a RAG system it has a precise, testable definition: a claim in the generated answer that is not supported by the retrieved context. That's exactly what the faithfulness metric targets, but faithfulness alone gives you a score, not a diagnosis. To actually catch and categorize hallucinations, break the check down:

• Claim decomposition. Split the generated answer into atomic factual statements. This is important because faithfulness at the whole-answer level hides partial hallucination — an answer with four true statements and one fabricated one still "sounds" mostly right, but that one fabricated statement might be the one the user acts on.
• Per-claim attribution. For each atomic claim, check whether it is entailed by the retrieved context, contradicted by it, or simply not mentioned. "Not mentioned" is the most common and most dangerous case — the model isn't necessarily lying, it's filling a gap with parametric knowledge that may or may not be true, and it does so fluently enough that reviewers miss it.
• Numeric and named-entity checks. Hallucinated numbers, dates, prices, and proper nouns are disproportionately harmful because they look precise and trustworthy. A dedicated check that extracts numbers and named entities from the answer and verifies each one appears in the retrieved context catches a large share of high-impact hallucinations cheaply, before you even need a full LLM-based faithfulness pass.
• Contradiction detection against the corpus, not just the retrieved chunk. Occasionally a model hallucinates something that is technically true and even present elsewhere in your corpus — but not in the context it was actually given. That's still a process failure worth flagging, because it means the model is not reliably grounding itself in what it was handed, and would produce different behavior with a differently-ordered context window.
• Refusal correctness. Explicitly test cases where the corpus has no answer. A pipeline that hallucinates a confident answer when it should say "I don't know" is often worse than one with mediocre recall, because users trust confident wrong answers more than they trust admitted uncertainty. Score refusal-appropriate test cases separately — don't let them get averaged away inside your overall faithfulness number.

The practical upshot: run faithfulness at the claim level, not the answer level, and keep a running category breakdown of hallucination type (unsupported claim, contradicted claim, wrong number/entity, missed refusal). A single aggregate faithfulness score of "0.87" tells you almost nothing actionable. A breakdown that says "numeric hallucinations went from 2% to 9% after the last prompt change" tells you exactly what to fix.

An illustrative eval loop

The following is conceptual pseudocode meant to show the shape of an eval loop, not a specific library's API.

# Illustrative pseudocode — not tied to a specific library

def run_eval(golden_set, rag_pipeline, judge_model):
    results = []

    for example in golden_set:
        query = example["query"]
        expected_chunks = example["expected_chunks"]
        reference_answer = example["reference_answer"]

        # Run the actual pipeline under test
        retrieved_chunks = rag_pipeline.retrieve(query)
        generated_answer = rag_pipeline.generate(query, retrieved_chunks)

        # Retrieval-side metrics
        context_recall = score_context_recall(
            reference_answer, retrieved_chunks, judge_model
        )
        context_precision = score_context_precision(
            query, retrieved_chunks, expected_chunks, judge_model
        )

        # Generation-side metrics
        claims = decompose_into_claims(generated_answer, judge_model)
        faithfulness = score_faithfulness(claims, retrieved_chunks, judge_model)
        answer_relevancy = score_answer_relevancy(query, generated_answer, judge_model)

        results.append({
            "query": query,
            "context_recall": context_recall,
            "context_precision": context_precision,
            "faithfulness": faithfulness,
            "answer_relevancy": answer_relevancy,
        })

    return aggregate_and_flag_regressions(results)

score_faithfulness, score_context_recall, and similar functions are typically implemented as calls to a capable judge model with a carefully constrained prompt — asking it to classify each claim rather than produce a free-form critique, since classification is far more consistent across runs than open-ended judgment. The exact prompt design, calibration against human labels, and handling of judge disagreement is a deep topic on its own — one we cover in full in the "LLM-as-a-Judge" course.

Wiring evals into CI so regressions block the deploy

An eval suite that only runs when someone remembers to run it manually will, eventually, not get run. The only durable fix is to make it part of your deploy pipeline, exactly like unit tests.

Run the eval suite on every pull request that touches retrieval or prompt code. This includes changes to your chunking strategy, embedding model, top_k, reranker config, system prompt, or the generation model itself. Trigger the eval job the same way you'd trigger a test suite — on PR open and on push to the PR branch.

Store a baseline and diff against it, not just an absolute threshold. Absolute thresholds ("faithfulness must be above 0.9") are useful as a floor, but the more actionable check is relative: did this change move any metric down by more than a small tolerance compared to the last known-good baseline on main? Store the baseline scores as a versioned artifact alongside the golden set itself, so the comparison is always against a specific, reproducible snapshot rather than a moving target.

Fail the build on regression, not just report it. A dashboard that shows scores trending down over three weeks is a dashboard nobody looks at until the incident. Set a CI gate: if aggregate faithfulness, context recall, or context precision drops beyond your tolerance, the check fails and the merge is blocked, the same way a failing unit test blocks a merge. Treat hallucination-category regressions (numeric/entity hallucination rate, refusal correctness) as their own gate, separate from the aggregate score, since aggregates can hide a spike in exactly the failure type you care most about.

Separate fast and slow eval tiers. Running your full synthetic-plus-golden set with an LLM judge on every commit can get slow and expensive if your set is large. A common pattern: run a smaller, fast subset (your highest-value hand-curated examples plus known regression cases) on every PR, and run the full set nightly or on merge to main, with alerting if the full run regresses even though the fast subset passed.

Log every eval run's raw outputs, not just scores. When a score drops, you need to look at which specific examples failed and read the actual retrieved chunks and generated answers, not just see a number go from 0.91 to 0.84. Store per-example results as build artifacts so a failing CI check comes with a direct link to the failing cases, ready for a human to inspect in under a minute.

Version the judge model deliberately. If your evals use an LLM as a judge, pin its version in CI configuration. Judge models get updated by providers, and a judge update can shift your scores independent of any change to your actual pipeline — which looks exactly like a regression until you realize the judge changed, not your code. When you do intentionally upgrade the judge, re-baseline explicitly rather than letting the shift get silently absorbed into your regression tolerance.

Common mistakes that quietly invalidate an eval setup

A few patterns show up repeatedly in teams that think they have RAG evaluation covered but don't:

• Using the same model as both generator and judge without any calibration against human labels. A judge model can share blind spots with the generator, scoring confidently wrong answers as faithful because both models "believe" the same plausible-sounding fabrication. Periodically spot-check judge scores against human review, especially after any model version change.
• Testing only on questions the corpus can answer. If every test case has a good answer available, you never find out whether your pipeline hallucinates gracefully or confidently when it shouldn't. Refusal cases are not optional.
• Averaging away multi-hop failures. If multi-hop questions are a small fraction of your golden set, their failures get diluted in the aggregate score even though they're often your highest-value, hardest queries in production. Report scores broken out by question category, not just as one blended number.
• Never refreshing the golden set. A test set frozen in time evaluates yesterday's corpus and yesterday's user behavior. Tie golden-set updates to your content update cadence.

Where to go from here

Evaluating a RAG pipeline is not a one-time checklist — it's an operating discipline. Separate retrieval metrics from generation metrics so you know which subsystem to fix. Build a golden set from real queries, expand it with synthetic generation, and never let it go stale. Break faithfulness down to the claim level so you can actually name and track hallucination categories instead of watching one fuzzy aggregate number. And put the whole thing in CI with a real regression gate, so the next prompt tweak or embedding model swap either proves itself safe or gets blocked before it reaches a user.

The hardest part of all of this is usually the judge: designing prompts that produce consistent, well-calibrated scores instead of noisy vibes-based grading. That's exactly what we go deep on, prompt by prompt and failure mode by failure mode, in the "LLM-as-a-Judge" course at teachyou.ai.