DeepEval

DeepEval for RAG: Faithfulness and Contextual Metrics in Practice

Pramod Dutta · Jun 12, 2026 · 14 min read

Why your RAG pipeline needs metrics, not vibes

You shipped a retrieval-augmented generation system. It answers questions from your knowledge base, cites source chunks, and demos beautifully. Then a user asks something slightly off the beaten path, and the model confidently states something that isn't in any retrieved document. Nobody notices until a support ticket comes in three weeks later.

This is the core problem with RAG systems: they fail silently. A chatbot that hallucinates doesn't throw an exception — it just produces fluent, wrong text. Unit tests that check for HTTP 200 responses or JSON schema validity are blind to this entire failure class. You need metrics that actually look at what the retriever fetched and what the generator said, then measure whether the two are consistent.

That's exactly the gap DeepEval fills. It's an open-source LLM evaluation framework built specifically for this kind of testing, and it ships a set of RAG-native metrics — FaithfulnessMetric, ContextualPrecisionMetric, ContextualRecallMetric, and ContextualRelevancyMetric — that map directly onto the two halves of a RAG pipeline: retrieval and generation. In this article we'll walk through each metric, show working code, and build a pattern you can drop into a CI pipeline so RAG regressions get caught before your users find them.

The two halves of a RAG pipeline, and why you evaluate them separately

Every RAG system has two distinct failure surfaces:

The retriever — is it fetching the right chunks from your vector store? If your embedding model or chunking strategy is bad, the generator never had a chance, no matter how good the LLM is.
The generator — given the retrieved chunks, does the LLM produce an answer that's actually grounded in them, and does it answer the user's question?

If you only measure end-to-end answer quality, you can't tell which half broke. A wrong answer could mean the retriever pulled irrelevant chunks, or it could mean the retriever did its job perfectly and the LLM hallucinated anyway. DeepEval's RAG metrics split cleanly along this boundary:

Retrieval quality: ContextualPrecisionMetric, ContextualRecallMetric, ContextualRelevancyMetric
Generation quality: FaithfulnessMetric (and typically paired with AnswerRelevancyMetric for relevance to the question)

Running all of these together on the same test case gives you a full breakdown: exactly where in the pipeline things are going wrong.

Setting up DeepEval

Installation is a single pip command, and DeepEval uses LLMTestCase as the unit of evaluation — a simple container for the four pieces of data a RAG evaluation needs.

pip install deepeval

Every test case needs up to four fields, though not every metric needs all four:

from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="What is your return policy for shoes that don't fit?",
    actual_output="We offer a 30-day full refund at no extra cost.",
    expected_output="You are eligible for a 30 day full refund at no extra cost.",
    retrieval_context=[
        "All customers are eligible for a 30 day full refund at no extra cost.",
        "Refunds are processed within 5-7 business days of receiving the returned item."
    ]
)

Note the distinction between expected_output and retrieval_context. expected_output is your ground-truth answer — the one a human reviewer or a golden dataset says is correct. retrieval_context is what your retriever actually pulled back from the vector database for this query. Some metrics only need retrieval_context (they're checking internal consistency), while others also need expected_output (they're checking against ground truth). Mixing these up is the single most common mistake when setting up DeepEval for RAG — get the field mapping wrong and your scores will look plausible but measure the wrong thing.

DeepEval also requires an LLM to act as the judge that scores each metric — by default it uses an OpenAI model, but you can swap in Anthropic's Claude, a local model, or any class implementing DeepEvalBaseLLM. This "LLM-as-judge" pattern is central to how DeepEval works: instead of exact-match string comparison, it uses an LLM to extract claims, verify them against context, and produce a defensible numeric score with a written reason.

It's worth pausing on why LLM-as-judge is the right approach for RAG evaluation in the first place. Traditional NLP metrics like BLEU or ROUGE compare generated text against a reference using word overlap, which breaks down immediately for RAG: two answers can use completely different wording and both be correct, or share most of their words and still contradict each other on the one fact that matters. Faithfulness and contextual grounding are semantic properties, not lexical ones — you need something that understands meaning, not just token overlap. That's why DeepEval builds every RAG metric on top of an LLM judge doing structured reasoning (claim extraction, verification against source text, and step-by-step scoring) rather than a fixed formula. The tradeoff is cost and latency per evaluation call, which is exactly why async_mode and batch evaluation matter once you move past a handful of manual test cases.

FaithfulnessMetric: catching hallucinations at the source

Faithfulness answers one question: does the generated answer contradict, or go beyond, what's actually in the retrieved context? This is the metric you want if hallucination is your top concern — and in most production RAG systems, it is.

Under the hood, FaithfulnessMetric works in two stages. First it extracts individual factual claims from actual_output. Then, for each claim, it checks whether the claim is supported by, contradicted by, or absent from retrieval_context. The final score is the proportion of claims that are faithful to the retrieved context.

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric

test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output="We offer a 30-day full refund at no extra cost, and you can also exchange for a different size for free.",
    retrieval_context=[
        "All customers are eligible for a 30 day full refund at no extra cost."
    ]
)

faithfulness = FaithfulnessMetric(
    threshold=0.7,
    model="gpt-4.1",
    include_reason=True
)

faithfulness.measure(test_case)
print(faithfulness.score)
print(faithfulness.reason)
print(faithfulness.is_successful())

In this example, the model claims free size exchanges are available — a claim that isn't backed by the retrieved chunk. FaithfulnessMetric would flag this claim as unsupported, dragging the score below a healthy threshold even though the refund claim itself is accurate. This is exactly the kind of partial hallucination that's easy to miss in manual review but obvious once you force claim-by-claim verification.

A few constructor options worth knowing:

threshold — the minimum score (0 to 1) required to pass; defaults to 0.5, but 0.7-0.8 is a more realistic bar for production-grade faithfulness.
strict_mode — forces binary pass/fail scoring instead of a continuous score, useful when you want a hard gate in CI rather than a gradient.
include_reason — when True, the judge LLM returns a natural-language explanation alongside the score, which is invaluable when you're debugging why a test failed.
async_mode — runs claim extraction and verification concurrently, which matters once you're evaluating hundreds of test cases in a batch.
truths_extraction_limit — caps how many factual claims get extracted per output, useful for very long generations where you want to bound evaluation cost.

ContextualPrecisionMetric: is your reranker doing its job?

Once you've confirmed the generator is faithful to what it retrieved, the next question is whether the retriever gave it good material to work with. ContextualPrecisionMetric specifically evaluates ranking quality — it checks whether the *relevant* chunks in your retrieval_context appear before the irrelevant ones.

This matters more than it sounds. Most RAG pipelines truncate context to a top-K window before feeding it to the LLM. If your reranker or retriever puts a relevant chunk at position 8 and irrelevant chunks fill positions 1-7, your LLM might never see the one chunk it needed — even though it was technically "retrieved."

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import ContextualPrecisionMetric

test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output="We offer a 30-day full refund at no extra cost.",
    expected_output="You are eligible for a 30 day full refund at no extra cost.",
    retrieval_context=[
        "Our warehouse is located in three regions across the country.",
        "All customers are eligible for a 30 day full refund at no extra cost.",
        "Shoes are made from synthetic and genuine leather materials."
    ]
)

contextual_precision = ContextualPrecisionMetric(
    threshold=0.7,
    model="gpt-4.1",
    include_reason=True
)

contextual_precision.measure(test_case)
print(contextual_precision.score)
print(contextual_precision.reason)

Here the genuinely relevant chunk sits in the middle of three retrieved passages, surrounded by noise. ContextualPrecisionMetric will use expected_output as its reference for "what should have been retrieved," then judge whether the relevant chunk was ranked appropriately relative to the noise. Note that this metric requires expected_output — it needs ground truth to know which chunks were actually relevant, not just plausible.

If this score is consistently low across your test suite, the fix usually isn't in the LLM at all — it's in your reranking step, your embedding model choice, or your chunk size.

ContextualRecallMetric: did you retrieve everything you needed?

Precision tells you if the good chunks were ranked well. Recall tells you a different thing entirely: did the retriever fetch all of the information needed to construct the expected answer, or did it miss pieces?

ContextualRecallMetric breaks expected_output down into individual statements and checks how many of them can be attributed back to something in retrieval_context. A low recall score means your embedding model or vector index is failing to surface documents that contain necessary information — even if what it did retrieve was perfectly relevant.

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import ContextualRecallMetric

test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output="We offer a 30-day full refund.",
    expected_output="You are eligible for a 30 day full refund at no extra cost, and returns must be initiated within 30 days of delivery.",
    retrieval_context=[
        "All customers are eligible for a 30 day full refund at no extra cost."
    ]
)

contextual_recall = ContextualRecallMetric(
    threshold=0.7,
    model="gpt-4.1",
    include_reason=True
)

contextual_recall.measure(test_case)
print(contextual_recall.score)
print(contextual_recall.reason)

Notice that expected_output mentions a 30-day initiation window that never shows up in retrieval_context at all. That's a genuine retrieval gap — no amount of prompt engineering on the generation side would have fixed this, because the information was never fetched in the first place. ContextualRecallMetric will catch this and score it lower than a case where every claim in the expected answer traces back to a retrieved chunk.

This is precisely why splitting retrieval and generation metrics matters: if you only measured FaithfulnessMetric here, the actual output ("We offer a 30-day full refund") is completely faithful to what was retrieved — it would score well. But the user still didn't get the complete answer, and only a recall-focused metric surfaces that.

ContextualRelevancyMetric: is your retriever pulling too much noise?

The third retrieval metric asks a more blunt question: of everything in `retrieval_context`, how much of it is actually relevant to the input query? This is a proxy for chunk size and top-K tuning — if you're retrieving 10 chunks and only 2 are relevant, ContextualRelevancyMetric will surface that as a low score, independent of whether the LLM was faithful or the answer was complete.

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import ContextualRelevancyMetric

test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output="We offer a 30-day full refund at no extra cost.",
    retrieval_context=[
        "All customers are eligible for a 30 day full refund at no extra cost.",
        "Our company was founded in 2015 and has since expanded to 12 countries.",
        "The CEO's favorite hiking trail is in the Rockies."
    ]
)

contextual_relevancy = ContextualRelevancyMetric(
    threshold=0.7,
    model="gpt-4.1",
    include_reason=True
)

contextual_relevancy.measure(test_case)
print(contextual_relevancy.score)
print(contextual_relevancy.reason)

Two of the three retrieved chunks here are irrelevant noise. In a small example like this it's obvious to a human, but at scale — hundreds of queries against a large vector store — this is exactly the kind of systemic issue that's invisible until you measure it directly. Chronic low relevancy scores usually point to chunking that's too coarse (mixing unrelated topics into one chunk) or a top-K setting that's too generous.

Running metrics together with evaluate() and pytest

Testing one metric on one test case is a debugging exercise. Testing RAG quality in a real project means running a battery of metrics across a whole dataset of queries, and doing it automatically whenever code changes. DeepEval's evaluate() function handles the batch case directly:

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    FaithfulnessMetric,
    ContextualPrecisionMetric,
    ContextualRecallMetric,
    ContextualRelevancyMetric,
)

test_cases = [
    LLMTestCase(
        input="What if these shoes don't fit?",
        actual_output="We offer a 30-day full refund at no extra cost.",
        expected_output="You are eligible for a 30 day full refund at no extra cost.",
        retrieval_context=["All customers are eligible for a 30 day full refund at no extra cost."]
    ),
    # add more test cases pulled from real production queries or a golden dataset
]

metrics = [
    FaithfulnessMetric(threshold=0.7),
    ContextualPrecisionMetric(threshold=0.7),
    ContextualRecallMetric(threshold=0.7),
    ContextualRelevancyMetric(threshold=0.7),
]

evaluate(test_cases=test_cases, metrics=metrics)

This single call scores every test case against all four metrics and prints a readable report with pass/fail status and reasons for each. For CI/CD, DeepEval also integrates with pytest via assert_test(), so a RAG regression becomes an actual failing test in your pipeline rather than something a human has to notice in a demo:

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, ContextualRecallMetric

def test_refund_policy_rag_response():
    test_case = LLMTestCase(
        input="What if these shoes don't fit?",
        actual_output="We offer a 30-day full refund at no extra cost.",
        expected_output="You are eligible for a 30 day full refund at no extra cost.",
        retrieval_context=["All customers are eligible for a 30 day full refund at no extra cost."]
    )
    assert_test(test_case, [
        FaithfulnessMetric(threshold=0.7),
        ContextualRecallMetric(threshold=0.7)
    ])

Run it with deepeval test run test_rag.py and it behaves like any other test suite — pass, fail, and a CLI exit code your CI system can act on. Wire this into a GitHub Actions workflow that runs on every pull request touching your retrieval or prompt code, and RAG quality regressions get caught the same way a broken unit test would be, instead of surfacing three weeks later in a support ticket.

Building a golden dataset instead of one-off test cases

The examples above use hand-written test cases, which is fine for learning the API but doesn't scale. In practice, you want an EvaluationDataset built from real or representative production queries, each with a curated expected_output and a snapshot of what your retriever actually returned. This is often called a "golden dataset" — a fixed, versioned set of query/answer/context triples that you re-run every time you change your embedding model, chunking strategy, or prompt template.

A practical way to build one: log real user queries and your RAG pipeline's retrieved chunks and generated answers in production, sample a representative subset weekly, have a human (or a stronger LLM as a first pass) verify or correct the expected_output, and add it to your dataset. Over a few weeks you'll have enough golden test cases that a regression in any single component — retriever or generator — shows up as a measurable drop in one of the four metrics rather than a vague sense that "answers feel worse lately."

Interpreting scores: what's actually a good threshold

A common mistake is treating the default threshold=0.5 as a meaningful bar. It isn't — it's a starting point. For faithfulness in particular, most production teams push this to 0.8 or higher, because even a single unsupported factual claim in a customer-facing answer is a real problem, not a rounding error. For contextual recall and precision, acceptable thresholds depend heavily on how forgiving your use case is: an internal documentation assistant can tolerate lower recall than a system answering questions about medical dosages or financial terms.

Rather than picking thresholds arbitrarily, run all four metrics against your golden dataset first without gating anything, look at the distribution of scores and read the reason output on the lowest-scoring cases, and only then set thresholds that reflect where your current pipeline actually sits — then tighten them over time as you improve retrieval and prompting. This turns evaluation from a pass/fail gate into a diagnostic tool that tells you exactly which part of your RAG stack needs attention next.

Where to go from here

DeepEval's contextual metrics give you something most RAG teams are missing: a way to separate "the retriever found the wrong stuff" from "the model made stuff up," instead of lumping both into a vague sense that answers are "off." Start with FaithfulnessMetric if hallucination is your biggest risk, add ContextualRecall and ContextualPrecision once you have expected_output data to evaluate against, and layer ContextualRelevancyMetric in when you're tuning chunk size and top-K.

If you want a structured, hands-on walkthrough — covering custom evaluation templates, integrating DeepEval with LangChain and LlamaIndex retrievers, building golden datasets from production logs, and wiring the whole thing into CI — check out the DeepEval Tutorial course on teachyou.ai, where we build a full evaluation harness for a real RAG application from scratch.