LLM EvaluationRAG evaluationhallucination detectionDeepEvalRAGAS

The Faithfulness Metric in LLM Evaluation

Pramod Dutta · Jun 30, 2026 · 10 min read

The faithfulness metric measures whether every claim in an LLM's generated answer can be traced back to the source context it was given, rather than invented, exaggerated, or misremembered. In a retrieval-augmented generation (RAG) pipeline, this is the single most important guardrail you can put in place, because a model can produce a fluent, confident, well-formatted answer that is completely unsupported by the documents it retrieved. Faithfulness catches exactly that failure mode: hallucination dressed up as a correct answer.

This article walks through what faithfulness actually measures, how it differs from answer relevancy and correctness, how the score gets computed under the hood, and how to wire it into a CI pipeline using DeepEval and RAGAS with runnable code.

What the faithfulness metric actually measures

Faithfulness is a context-grounding check, not a fact-checking check against the real world. It asks one narrow question: given the retrieved context, does the generated output contradict it or introduce claims that aren't present in it?

This distinction matters a lot in practice. If your retriever pulls back an outdated document that says a product costs $49, and the model faithfully repeats "$49," the faithfulness score will be high even though the real-world price changed to $59 last month. Faithfulness doesn't know that. It only checks generation-against-context consistency. Real-world correctness is a separate concern, usually handled by a correctness metric compared against a labeled ground truth, or by keeping your knowledge base fresh.

Three related metrics get confused with faithfulness constantly, so it's worth separating them clearly:

Faithfulness: does the answer contradict or invent facts relative to the retrieved context?
Answer relevancy: does the answer actually address the user's question, regardless of grounding?
Contextual precision/recall: did the retriever fetch the right chunks in the first place?

A RAG system can score well on relevancy and terribly on faithfulness (a confident hallucination that answers the question) or the reverse (a faithful but evasive answer that dodges the question using only what's in context). You need both signals, and they're measured independently.

How faithfulness scoring works under the hood

Most modern faithfulness implementations, including DeepEval's FaithfulnessMetric and RAGAS's Faithfulness, follow the same two-stage pipeline:

Claim extraction: an LLM (the "judge" model) decomposes the generated answer into a list of atomic factual claims. Each claim should be a single, independently verifiable statement.
Claim verification: each claim is checked against the retrieval context using natural language inference (NLI). The claim is labeled as supported, contradicted, or not-inferable-from-context.

The final score is typically:

faithfulness_score = (number of claims supported by context) / (total number of claims)

Some implementations only penalize outright contradictions and treat unverifiable-but-not-contradicted claims more leniently; others (like RAGAS's default) count anything not directly entailed by the context as unfaithful. Read the library's scoring rubric before you trust the number, because a 0.85 in one framework is not necessarily comparable to a 0.85 in another.

Here's a concrete example. Say the context contains:

Context: "The Eiffel Tower was completed in 1889 and stands 330 meters tall,
including antennas. It was designed by Gustave Eiffel's engineering company."

And the model generates:

Answer: "The Eiffel Tower was finished in 1889, is about 330 meters tall,
and was the tallest building in the world for over 40 years."

Claim extraction produces three claims:

The Eiffel Tower was finished in 1889. (supported)
It is about 330 meters tall. (supported)
It was the tallest building in the world for over 40 years. (not present in context, unverifiable)

Depending on the framework's strictness, claim 3 either drags the score down to 0.67 (2/3) or gets flagged separately as an unsupported addition. This is exactly the kind of subtle over-claiming that slips past a human skim-read but shows up reliably once you decompose the answer into atomic claims.

Computing faithfulness with DeepEval

DeepEval is the most common open-source choice for wiring faithfulness into a pytest-style test suite. Install it and set an API key for the judge model first.

pip install deepeval
export OPENAI_API_KEY=your_key_here

Then run a single faithfulness check:

from deepeval import evaluate
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="How tall is the Eiffel Tower and when was it built?",
    actual_output=(
        "The Eiffel Tower was finished in 1889, is about 330 meters tall, "
        "and was the tallest building in the world for over 40 years."
    ),
    retrieval_context=[
        "The Eiffel Tower was completed in 1889 and stands 330 meters tall, "
        "including antennas. It was designed by Gustave Eiffel's engineering company."
    ],
)

metric = FaithfulnessMetric(threshold=0.7, model="gpt-4o", include_reason=True)
metric.measure(test_case)

print(metric.score)
print(metric.reason)
print(metric.is_successful())

metric.reason returns a human-readable explanation naming which claims were unsupported, which is far more actionable than a bare float when you're debugging a regression. Wrap this in a pytest test to gate merges:

import pytest
from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

def test_rag_answer_is_faithful():
    test_case = LLMTestCase(
        input="How tall is the Eiffel Tower and when was it built?",
        actual_output=get_rag_pipeline_answer("How tall is the Eiffel Tower and when was it built?"),
        retrieval_context=get_retrieved_chunks("How tall is the Eiffel Tower and when was it built?"),
    )
    metric = FaithfulnessMetric(threshold=0.7)
    assert_test(test_case, [metric])

Run it with deepeval test run test_faithfulness.py and it slots straight into CI like any other test file.

Computing faithfulness with RAGAS

RAGAS is the other widely used option, and it's built specifically for RAG pipeline evaluation over a whole dataset rather than one test case at a time.

pip install ragas datasets

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness

data = {
    "question": ["How tall is the Eiffel Tower and when was it built?"],
    "answer": [
        "The Eiffel Tower was finished in 1889, is about 330 meters tall, "
        "and was the tallest building in the world for over 40 years."
    ],
    "contexts": [[
        "The Eiffel Tower was completed in 1889 and stands 330 meters tall, "
        "including antennas. It was designed by Gustave Eiffel's engineering company."
    ]],
}

dataset = Dataset.from_dict(data)
results = evaluate(dataset, metrics=[faithfulness])
print(results.to_pandas()[["question", "faithfulness"]])

RAGAS shines when you already have an evaluation set of a few hundred question-answer-context triples and want an aggregate faithfulness score across the whole set, plus a pandas dataframe you can slice by category, retriever version, or prompt template. DeepEval shines when you want per-case reasoning and a pytest-native gate. Many teams use both: RAGAS for batch regression reports, DeepEval for the CI gate on individual pull requests.

Choosing a judge model and threshold

The claim-extraction and claim-verification steps are themselves LLM calls, so your faithfulness score inherits the judge model's own reasoning quality and biases. A few practical rules:

Use a strong judge model. Faithfulness scoring with a weak or heavily quantized judge produces noisy claim decomposition. Reserve your best available model for the judge role, even if the pipeline under test uses a cheaper model.
Don't use the same model as both generator and judge without care. A model tends to rate its own outputs more favorably than an independent judge would. If budget allows, use a different model family for judging.
Set the threshold based on your risk tolerance, not a default. A customer-support RAG bot answering billing questions needs a much higher bar (0.9+) than an internal brainstorming assistant (0.6 might be fine). Pull the threshold from an incident, not a tutorial.
Sample and spot-check the reasoning traces. Read metric.reason on a handful of failing cases every week. Judge models make extraction mistakes too, especially with claims that mix numbers, dates, and qualifiers in one sentence.

Common pitfalls that skew faithfulness scores

Context that's too short or too fragmented. If your retriever returns tiny, disconnected chunks, correct claims can get marked unfaithful simply because the supporting detail was split across two chunks that never got concatenated for the judge. Check your chunking strategy before you blame the generator.

Claims that require reasonable inference. "The tower was designed to be temporary" being inferred from "built for the 1889 World's Fair" is a judgment call some judges will mark as unsupported even though a careful human reader would accept it. Decide upfront whether your use case wants strict entailment or reasonable-inference tolerance, and pick a metric implementation that matches.

Numeric and unit claims. LLMs are notoriously sloppy about carrying units, rounding, and currency across a generation. Faithfulness catches straightforward contradictions ("330 meters" vs "230 meters") but can miss unit-swap errors if the claim decomposition doesn't isolate the number cleanly. For numeric-heavy domains (finance, medical dosing, engineering specs), add a dedicated regex or rule-based check alongside the LLM-judged faithfulness score rather than relying on it alone.

Treating faithfulness as a substitute for correctness. As covered above, faithfulness only checks against retrieved context, not ground truth. If your knowledge base is stale or wrong, a perfectly faithful answer can still be a wrong answer. Pair faithfulness with a correctness or answer-accuracy metric evaluated against curated ground truth, not just the retriever's own output.

Building a faithfulness regression suite

A minimal but effective setup looks like this:

Collect 50-200 real user queries from logs or support tickets, spanning easy, ambiguous, and adversarial cases (queries designed to tempt the model into filling gaps with invented detail).
Run each through your production RAG pipeline to capture input, actual_output, and retrieval_context.
Score every case with FaithfulnessMetric and store scores alongside a timestamp and pipeline version.
Set a CI gate: fail the build if the average faithfulness score across the suite drops below your threshold, or if any single case drops more than a fixed delta from its last known score.
Re-run the suite on every change to the prompt template, retriever, chunking strategy, or embedding model, since faithfulness is sensitive to all four.

import json
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

def run_faithfulness_suite(cases_path: str, threshold: float = 0.7):
    with open(cases_path) as f:
        cases = json.load(f)

    metric = FaithfulnessMetric(threshold=threshold, include_reason=True)
    results = []

    for case in cases:
        test_case = LLMTestCase(
            input=case["input"],
            actual_output=case["actual_output"],
            retrieval_context=case["retrieval_context"],
        )
        metric.measure(test_case)
        results.append({
            "input": case["input"],
            "score": metric.score,
            "passed": metric.is_successful(),
            "reason": metric.reason,
        })

    avg_score = sum(r["score"] for r in results) / len(results)
    failed = [r for r in results if not r["passed"]]

    print(f"Average faithfulness: {avg_score:.3f}")
    print(f"Failed cases: {len(failed)}/{len(results)}")
    for r in failed:
        print(f"- {r['input'][:80]}: {r['score']:.2f} -- {r['reason']}")

    return avg_score, failed

This gives you a version-over-version trend line, not just a single snapshot, which is what actually catches slow drift when someone tweaks a system prompt and nobody notices the retriever context got quietly truncated.

FAQ

What is a good faithfulness score? There's no universal number. Treat 0.9+ as a reasonable bar for high-stakes domains like billing, medical, or legal answers, and 0.7-0.8 as workable for lower-stakes assistants. The right threshold comes from your own incident history and risk tolerance, not a fixed industry benchmark.

Is faithfulness the same as hallucination detection? They overlap but aren't identical. Faithfulness specifically measures grounding against retrieved context. A broader hallucination metric might also flag claims that are false relative to general world knowledge, even without any retrieval step involved, which faithfulness alone won't catch.

Does faithfulness work without a RAG pipeline? It needs some form of reference context to check against. Without retrieval context, you can still run it against a fixed reference document, a system prompt containing facts, or a knowledge base excerpt, but you can't run it on a pure open-domain chat with no grounding source.

Can I compute faithfulness without an LLM judge? Rule-based or embedding-similarity approaches exist but tend to miss subtle contradictions and can't reliably decompose compound sentences into atomic claims. LLM-as-judge remains the practical standard for production faithfulness scoring, with the tradeoff being judge cost, latency, and judge-model bias that you need to account for.

How often should I re-run faithfulness evaluation? At minimum on every change to the prompt, retriever, chunking, or embedding model, since faithfulness is sensitive to all of them. Many teams also run a lightweight faithfulness sample continuously in production on a subset of live traffic to catch silent drift between full regression runs.

Why did my faithfulness score drop after I improved answer quality? This usually means the model started adding helpful-sounding elaboration that isn't in the retrieved context, like extra background facts or comparisons. That elaboration reads well to a human reviewer but counts against faithfulness because it isn't grounded. Tighten the system prompt to explicitly instruct the model to answer only from the provided context, and add an instruction to omit details it can't verify from the sources given.