RAG Hallucination Testing

RAG systems fail in two distinct places: retrieval can supply the wrong evidence, and generation can make claims that the evidence does not support. Effective rag hallucination testing measures both stages separately, then blocks releases when groundedness, answer correctness, or citation quality falls below an explicit threshold. This guide builds that workflow with Python, pytest, JSONL fixtures, deterministic retrieval checks, and model-based graders that remain auditable.

What rag hallucination testing must prove

A useful test does more than ask whether an answer sounds reasonable. It proves that each important claim is supported by retrieved context, that the answer addresses the question, and that retrieval exposed the necessary source material.

Treat these as separate properties:

• Retrieval relevance: returned chunks concern the question.
• Retrieval coverage: returned chunks contain enough facts to answer.
• Groundedness: answer claims follow from those chunks.
• Answer correctness: answer matches a trusted reference or required facts.
• Citation correctness: cited chunk identifiers actually support nearby claims.
• Abstention behavior: the system refuses when evidence is missing or conflicting.

An answer may be grounded but wrong because the knowledge base is outdated. It may be correct but ungrounded because the model used memorized knowledge. It may cite a relevant document while inventing a number absent from that document. One aggregate score hides these differences, so keep individual measurements and failure messages.

Start with release risks. For an internal policy assistant, unsupported deadlines and invented approvals may be critical. For developer documentation, incorrect commands and nonexistent configuration fields deserve special cases. Turn those risks into examples before selecting metrics or tools.

Define the failure contract

Write a short contract that makes hallucination observable. A practical contract can state:

1. Every externally verifiable claim must be entailed by retrieved evidence.
2. Numbers, dates, names, commands, and quoted policy language require direct support.
3. The answer must not use facts from model memory unless the product explicitly permits them.
4. When evidence is insufficient, the assistant must abstain and explain what is missing.
5. Citations must identify the source chunks used for each material claim.

Define the evidence boundary precisely. If your prompt includes chat history, tool output, metadata, and retrieved chunks, decide which inputs count as valid evidence. Otherwise a grader may flag a claim supported by tool output, or accept a claim leaked through the test reference answer.

Use severity labels. A fabricated legal requirement is not equivalent to an unnecessary adjective. For example:

• critical: unsafe instruction, fabricated obligation, wrong financial value.
• major: unsupported product behavior, date, limit, or command.
• minor: harmless elaboration that does not change the action.

CI should reject any critical failure even when the average score is high. Averages are useful for trends, not for forgiving dangerous cases.

Build an evaluation dataset

Store cases in JSONL so engineers can review diffs and run one case locally. Each case should capture the query, expected facts, forbidden claims, relevant document identifiers, whether abstention is expected, and optional metadata.

{"id":"leave-carryover","question":"How many leave days carry into next year?","required_facts":["Up to 5 unused days may carry over"],"forbidden_facts":["Unlimited days carry over"],"relevant_doc_ids":["leave-policy-v3"],"expect_abstention":false,"severity":"major"}
{"id":"unknown-country","question":"What is our parental leave policy in Brazil?","required_facts":[],"forbidden_facts":[],"relevant_doc_ids":[],"expect_abstention":true,"severity":"major"}

Include four families of cases:

• Normal answerable questions, including paraphrases and short queries.
• Unanswerable questions for which the corpus has no evidence.
• Adversarial questions with false premises, conflicting sources, or requests to ignore documents.
• Boundary questions involving similar entities, units, versions, dates, or permissions.

Create cases from production incidents, support tickets, documentation changes, and subject matter expert review. Synthetic generation can expand coverage, but a human should verify required facts against the source. Never let the same model freely generate both test questions and unquestioned gold answers.

Pin a corpus snapshot or record document hashes. Without that, a failed case may reflect a content update rather than a code regression. Metadata should include applicable region, product version, effective date, and access scope when those affect truth.

Keep a smaller smoke suite for every pull request and a broader regression suite for scheduled runs. Preserve every confirmed production hallucination as a permanent test.

Instrument the RAG pipeline

Tests need more than the final string. Return a structured trace containing retrieved chunks, scores, identifiers, prompt version, model identifier, answer, citations, latency, and request settings. Do not log secrets or unrestricted document text in shared CI artifacts.

from dataclasses import dataclass

@dataclass
class Chunk:
    id: str
    doc_id: str
    text: str
    score: float

@dataclass
class RagTrace:
    question: str
    chunks: list[Chunk]
    answer: str
    citations: list[str]
    prompt_version: str
    model: str

def answer_question(question: str) -> RagTrace:
    chunks = retriever.search(question, top_k=5)
    prompt = build_prompt(question, chunks)
    result = generator.generate(prompt, temperature=0)
    return RagTrace(
        question=question,
        chunks=chunks,
        answer=result.text,
        citations=result.citations,
        prompt_version="grounded-v4",
        model=result.model,
    )

Temperature zero reduces variation but does not guarantee identical outputs across infrastructure or model updates. Record the resolved model version when the provider exposes one. Cache evaluation responses where policy allows, and rerun a sample to estimate variance.

Make chunk IDs stable across runs. An ID derived from document identity, section path, and content hash is more useful than a database row number. Stable IDs allow exact citation checks and explain retrieval changes.

Test retrieval before generation

If the required document never enters the context, generation cannot reliably succeed. Measure retrieval independently using gold document IDs or chunk IDs.

def recall_at_k(retrieved_doc_ids: list[str], relevant_doc_ids: list[str], k: int) -> float:
    relevant = set(relevant_doc_ids)
    if not relevant:
        return 1.0
    found = set(retrieved_doc_ids[:k]) & relevant
    return len(found) / len(relevant)

def reciprocal_rank(retrieved_doc_ids: list[str], relevant_doc_ids: list[str]) -> float:
    relevant = set(relevant_doc_ids)
    for rank, doc_id in enumerate(retrieved_doc_ids, start=1):
        if doc_id in relevant:
            return 1.0 / rank
    return 0.0

Document-level recall can be too generous when a long document contains many unrelated sections. Add chunk-level required evidence strings or semantic labels. For high-risk facts, assert that the exact policy clause or normalized value appears in the context.

def context_contains_fact(chunks: list[Chunk], phrase: str) -> bool:
    haystack = "\n".join(c.text for c in chunks).casefold()
    return phrase.casefold() in haystack

def test_leave_policy_retrieval():
    trace = answer_question("How many leave days carry over?")
    assert "leave-policy-v3" in {c.doc_id for c in trace.chunks}
    assert context_contains_fact(trace.chunks, "up to 5 unused days")

Exact matching works for contractual strings, values, commands, and names. Use human-labeled chunk IDs for paraphrased concepts. Embedding similarity alone is not proof of factual coverage.

Log rank, retrieval score, filters, rewritten query, and reranker output. Common retrieval failures include missing metadata filters, stale indexes, aggressive chunking, acronym mismatch, and rerankers preferring fluent but irrelevant passages.

Run claim-level rag hallucination testing

Whole-answer grading often misses one fabricated detail inside an otherwise correct response. Split the answer into atomic claims, then test each claim against only the retrieved evidence. Atomic means one independently verifiable proposition, such as Employees may carry over five days.

A grader should return structured JSON rather than prose:

{
  "claims": [
    {
      "text": "Employees may carry over five unused days.",
      "verdict": "supported",
      "evidence_chunk_ids": ["leave-policy-v3:carryover:8d21"],
      "reason": "The chunk states the same limit."
    }
  ],
  "unsupported_claim_count": 0
}

Use verdicts such as supported, contradicted, and not_in_evidence. Contradiction is more severe than absence. Require the grader to cite chunk IDs and validate that every returned ID exists in the trace.

The grading prompt must not include the reference answer during groundedness evaluation. Otherwise the grader may treat reference facts as evidence. Pass only the question, answer, and retrieved chunks. Use the reference answer separately for correctness.

def groundedness_prompt(question: str, answer: str, chunks: list[Chunk]) -> str:
    evidence = "\n".join(f"<{c.id}> {c.text}" for c in chunks)
    return f"""Evaluate claims using only EVIDENCE.
Return JSON with claims and unsupported_claim_count.
Verdicts: supported, contradicted, not_in_evidence.
QUESTION: {question}
ANSWER: {answer}
EVIDENCE:
{evidence}
"""

Treat model graders as tests with their own error rate. Calibrate them against a human-labeled set containing clear support, subtle contradictions, partial support, numerical changes, and irrelevant citations. Review disagreement cases regularly. For critical domains, deterministic assertions and expert review should complement model judgments.

Add deterministic assertions

Model-based grading provides breadth, while deterministic checks catch precise failures cheaply. Extract and compare high-risk tokens:

• Numbers and units, including percentages, currencies, durations, and limits.
• Dates and version identifiers.
• Shell commands, API paths, configuration keys, and enum values.
• Named entities, policy names, regions, and roles.
• Citation identifiers and source URLs stored as plain data.

import re

NUMBER = re.compile(r"\b\d+(?:\.\d+)?%?\b")

def unsupported_numbers(answer: str, chunks: list[Chunk]) -> set[str]:
    evidence = " ".join(c.text for c in chunks)
    answer_numbers = set(NUMBER.findall(answer))
    evidence_numbers = set(NUMBER.findall(evidence))
    return answer_numbers - evidence_numbers

def assert_valid_citations(trace: RagTrace) -> None:
    available = {c.id for c in trace.chunks}
    unknown = set(trace.citations) - available
    assert not unknown, f"Unknown citation IDs: {sorted(unknown)}"

This numeric check intentionally favors recall over precision. It may flag list numbering, converted units, or derived arithmetic. Either prohibit derivations, add an allowlist per case, or require the answer to show the calculation from supported operands.

Also scan for refusal behavior. A refusal should not include a guessed answer after phrases such as I do not have enough information. Assert that expected abstention cases contain an approved refusal signal and no forbidden facts.

Build a runnable pytest harness

Keep provider-specific calls behind interfaces. The test runner should accept an existing application endpoint or local pipeline, making evaluation independent of a single SDK.

import json
from pathlib import Path
import pytest

def load_cases(path: str):
    return [json.loads(line) for line in Path(path).read_text().splitlines() if line]

CASES = load_cases("evals/rag_cases.jsonl")

@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_rag_case(case):
    trace = answer_question(case["question"])
    retrieved = [c.doc_id for c in trace.chunks]

    if case["relevant_doc_ids"]:
        assert recall_at_k(retrieved, case["relevant_doc_ids"], 5) == 1.0

    for forbidden in case["forbidden_facts"]:
        assert forbidden.casefold() not in trace.answer.casefold()

    assert_valid_citations(trace)
    assert not unsupported_numbers(trace.answer, trace.chunks)

    grade = grade_groundedness(trace)
    assert grade["unsupported_claim_count"] == 0

    if case["expect_abstention"]:
        assert is_abstention(trace.answer)

Run one case during debugging and the complete suite in CI:

python -m venv .venv
source .venv/bin/activate
python -m pip install pytest
pytest -q evals/test_rag.py -k leave-carryover
pytest -q evals/test_rag.py --junitxml=artifacts/rag-evals.xml

In a real repository, pin dependencies with the project's package manager and lockfile. Keep credentials in the CI secret store. Put rate limits, timeouts, retry limits, and maximum evaluation cost controls around external grader calls.

Set release gates without misleading averages

Establish thresholds from a reviewed baseline, then tighten them as the suite matures. Do not copy thresholds from another product because risk, corpus, labels, and graders differ.

A robust gate can require:

• Zero critical unsupported or contradicted claims.
• Every high-risk case passes deterministic assertions.
• Retrieval coverage does not regress beyond an agreed tolerance.
• Groundedness and correctness meet product-specific minimums.
• No unexplained increase in abstention for answerable questions.

Compare candidate results with the current production baseline on the same cases, corpus snapshot, and grader configuration. A paired comparison is more actionable than an isolated score. Save per-case output so a reviewer sees the question, retrieved IDs, answer, failed property, and evidence.

Flaky outputs need a declared policy. Rerun only model-dependent failures a limited number of times, report every attempt, and fail cases whose pass rate is unstable. Never rerun silently until green. If a test is nondeterministic, track distributions in scheduled evaluations while keeping critical deterministic checks blocking.

Segment reports by query type, language, document age, access scope, and severity. Overall improvement can conceal a regression for a small but important segment.

Diagnose failures systematically

Use the failing property to route investigation:

1. If required evidence was not retrieved, inspect indexing, chunking, filters, query rewriting, and reranking.
2. If evidence was retrieved but omitted from the prompt, inspect token budgeting, ordering, deduplication, and serialization.
3. If evidence was present but the answer contradicted it, strengthen instructions, improve evidence formatting, reduce distracting context, or change the generator.
4. If claims are supported but the final answer is wrong, check the source itself, reference labels, and temporal applicability.
5. If citations are wrong, require structured citation output and validate identifiers before rendering.
6. If the system answers an unanswerable question, improve evidence sufficiency checks and abstention examples.

Changing the model is only one possible fix. Many hallucinations originate in stale content, ambiguous source documents, or retrieving too many near-duplicate chunks. Store traces from both passing and failing versions, then diff retrieved ranks and prompt contents before adjusting generation.

Add a pre-generation sufficiency step for high-risk applications. It can verify that required entity, jurisdiction, effective date, and key fact are present. If not, return a controlled abstention without invoking free-form generation.

Test adversarial and changing conditions

Documents can contain prompt injection, conflicting instructions, and obsolete text. Add corpus fixtures that say ignore previous instructions, then verify that the system treats those words as data, not authority. Separate trusted system instructions from untrusted retrieved content with clear delimiters and role boundaries.

Test conflicts explicitly. When two policies disagree, the assistant should use reliable metadata such as status and effective date, or disclose the conflict. Do not expect the model to infer document authority from writing style.

Other valuable mutations include:

• Replace a supported number with a nearby number.
• Swap similar product or region names.
• Remove the one chunk that proves the answer.
• Add a highly similar but obsolete document.
• Truncate a chunk at the decisive sentence.
• Ask a leading question containing a false claim.
• Request an answer without citations.

Run these mutations against the same base case. The system should update, abstain, or flag conflict according to the evidence, not repeat the original answer from memory.

Evaluate access control before relevance. A perfectly grounded answer can still leak information when retrieval includes documents the caller cannot access. Tests should use identities with different permissions and assert that forbidden document IDs never enter the trace.

Operate the suite in production

Offline tests protect releases, but production traffic reveals vocabulary and combinations the fixture set missed. Log privacy-safe signals, sample traces under policy, and provide users a way to report unsupported answers. Convert confirmed incidents into minimized regression cases.

Monitor retrieval empty rates, abstention rates, citation validation failures, and supported-claim rates by deployment version. Watch changes after corpus ingestion, embedding updates, chunking changes, prompt revisions, and model migrations. Each of those can alter behavior without application code changes.

Create a version manifest for each evaluation run:

{
  "application_commit": "8f31c2a",
  "corpus_snapshot": "policies-2026-07-01",
  "retriever_config": "hybrid-v6",
  "prompt_version": "grounded-v4",
  "generator": "configured-model-id",
  "grader": "configured-grader-id",
  "dataset": "rag-regression-v12"
}

This makes a result reproducible without claiming that hosted models never change. Retain manifests, case outcomes, and redacted traces according to your security policy.

Schedule expert review of a sample of passes as well as failures. False acceptance is more dangerous than a noisy false alarm because it makes the dashboard look healthy. Periodically relabel cases when policies or product behavior change, but retain historical dataset versions for auditability.

FAQ

What is the best metric for RAG hallucinations?

There is no single best metric. Use claim-level groundedness for unsupported content, retrieval coverage for missing evidence, correctness for agreement with trusted facts, citation validation for attribution, and explicit abstention tests for unanswerable questions. Preserve separate results rather than collapsing everything into one score.

Can I test hallucinations without an LLM grader?

Yes, for many high-value properties. You can assert retrieved document IDs, required phrases, forbidden facts, numbers, commands, citations, and refusal behavior. A grader helps with paraphrases and claim entailment, but deterministic checks are easier to reproduce and should cover critical values whenever possible.

Should the reference answer be shown to the groundedness grader?

No. Groundedness asks whether the generated answer follows from retrieved evidence. Showing the reference answer can leak facts that were never retrieved. Use a separate correctness evaluation for comparison with the reference.

How many test cases do I need?

Begin with the highest-risk intents and every known incident, then expand based on coverage and production traffic. Case quality and segmentation matter more than an arbitrary count. Each case should have verified evidence, an expected behavior, and a reason it protects the product.

How do I reduce flaky evaluation results?

Use stable corpus snapshots, record resolved configurations, minimize sampling, request structured grader output, validate schemas, cache when appropriate, and calibrate graders against human labels. Run paired comparisons and expose repeated outcomes rather than silently retrying failures.

Does a citation prove that an answer is grounded?

No. A citation proves only that the system emitted an identifier. Validate that the identifier belongs to retrieved context and that the cited chunk entails the nearby claim. Numerical and procedural claims deserve especially strict checks.

When should a RAG system abstain?

It should abstain when evidence is absent, insufficient, conflicting without a clear authority rule, outside the user's access scope, or not applicable to the requested entity, region, or date. Test abstention as a product behavior, including what the assistant says and what it does not guess.