Ragas Noise Sensitivity Metric: Testing Retrieval Robustness

Why Your RAG Pipeline Passes Tests But Still Hallucinates

You have run your RAG evaluation suite. Faithfulness looks good. Answer relevancy looks good. Context precision looks good. Then a user reports that the chatbot confidently mixed up two product SKUs, or cited a refund policy from a completely unrelated document. You go back to your eval dashboard and everything is still green. What happened?

This is the blind spot that most RAG evaluation setups have: they measure whether the system behaves well when retrieval is clean, but they rarely measure what happens when retrieval is messy. In production, retrieval is almost never perfectly clean. Every retriever, no matter how well tuned, will occasionally pull in a chunk that is topically adjacent but factually irrelevant to the question being asked. The real question is not "does my system work when context is perfect" — it is "does my system degrade gracefully when context is imperfect."

That is exactly the gap that Ragas Noise Sensitivity was built to close. It is one of the more underused metrics in the Ragas library, largely because it requires a bit more setup than Faithfulness or Answer Relevancy, but it tells you something those metrics cannot: whether your LLM generator gets confused and starts hallucinating the moment noisy or irrelevant context sneaks into the retrieved set. In this article we will unpack what Noise Sensitivity actually measures, how it is computed under the hood, how to run it with real code, and how to use the results to make your retrieval pipeline more robust.

What Noise Sensitivity Actually Measures

Noise Sensitivity answers a narrow, well-defined question: when the retrieved context contains a mix of relevant and irrelevant chunks, does the generated answer include claims that are only supported by the irrelevant chunks?

In other words, it is not asking "is retrieval good?" (that is Context Precision and Context Recall). It is not asking "is the answer relevant to the question?" (that is Answer Relevancy). It is asking a much more specific and much more dangerous question: does noise in the context leak into the final answer as false or unsupported claims?

Think of it this way. A retriever fetches five chunks for a user question. Two of them are directly relevant. Three are noise — they mention similar keywords but do not actually answer the question. A well-behaved generator should ignore the noise and answer using only the two relevant chunks, or say "I don't have enough information" if the relevant chunks are insufficient. A noise-sensitive generator, on the other hand, will pick up fragments from the irrelevant chunks and weave them into the answer as if they were facts, producing a response that sounds fluent and confident but is subtly or grossly wrong.

Ragas computes this by breaking the generated answer into individual claims, then checking each claim against two sets of context: the relevant chunks and the irrelevant chunks. A claim that can only be attributed to a chunk from the irrelevant set is flagged as a case of noise leaking into the output. The final score is the proportion of claims in the answer that are attributable to noise rather than to the relevant, ground-truth-supporting context.

The score ranges from 0 to 1. A score of 0 means the generator never leaked noise into its answer — every claim traces back to relevant context. A score closer to 1 means the generator is highly susceptible to being derailed by irrelevant retrieved chunks. Unlike most Ragas metrics where higher is better, for Noise Sensitivity, lower is better.

How It Differs From Faithfulness

This is the question everyone asks first, because on the surface Noise Sensitivity sounds like a variant of Faithfulness. They are related but they measure different failure modes, and conflating them will lead you to draw the wrong conclusions from your eval results.

Faithfulness checks whether every claim in the generated answer is supported by *any* of the retrieved context, relevant or not. If your retriever hands over ten chunks and the answer is fully grounded in one of them, Faithfulness will score it as faithful — even if that one chunk happened to be irrelevant noise that coincidentally supports a claim. Faithfulness cares about grounding in the context window as a whole, not about whether the model correctly distinguished signal from noise.

Noise Sensitivity specifically partitions the context into relevant and irrelevant subsets (using the reference answer or ground truth to determine relevance) and checks whether claims are grounded in the *relevant* subset specifically. A model can be perfectly "faithful" in the traditional sense — every claim is technically traceable to some retrieved chunk — while still scoring poorly on Noise Sensitivity because the claims it is making come from the wrong chunks.

Here is a concrete way to see the difference. Suppose a user asks "What is the cancellation policy for the Pro plan?" The retriever returns:

• Chunk A (relevant): "Pro plan subscriptions can be cancelled anytime with a 30-day notice period."
• Chunk B (irrelevant, about a different plan): "Enterprise plan subscriptions require a 90-day notice period and a signed offboarding form."

If the generated answer says "Pro plan subscriptions require a 90-day notice period," that claim is faithful in the strict sense — it is grounded in Chunk B, which was part of the retrieved context. But it is wrong for the actual question, and it is exactly the kind of error Noise Sensitivity is designed to catch, because Chunk B is irrelevant to the specific question about the Pro plan.

This is why teams that only track Faithfulness are often surprised when users report factually wrong answers despite high Faithfulness scores. Faithfulness protects against pure hallucination (claims invented with no textual basis at all). Noise Sensitivity protects against a more insidious failure: correct-looking hallucination that is actually just misattribution across chunks.

The Math Behind the Metric

Ragas computes Noise Sensitivity in a few steps, and it is worth understanding them because it clarifies exactly what inputs you need to supply.

Step 1 — Decompose the answer into claims. The generated response is broken down into atomic, checkable statements using an LLM. For example, "The Pro plan costs $49/month and can be cancelled with 30 days notice" becomes two claims: "The Pro plan costs $49/month" and "The Pro plan can be cancelled with 30 days notice."

Step 2 — Partition the retrieved context. Using the reference (ground truth) answer, each retrieved chunk is classified as relevant or irrelevant to answering the question correctly. This partitioning is what makes Noise Sensitivity require a reference field in your evaluation dataset — without a ground truth to compare against, Ragas cannot tell which chunks were actually useful versus which ones merely looked topically similar.

Step 3 — Attribute each claim. For every claim extracted in Step 1, an LLM judge checks whether that claim is entailed by the relevant chunks, the irrelevant chunks, both, or neither.

Step 4 — Compute the score. The Noise Sensitivity score is the number of claims attributable to irrelevant context, divided by the total number of claims in the answer.

Formally, you can express it as:

Noise Sensitivity = (claims_supported_by_irrelevant_context) / (total_claims_in_answer)

Ragas actually exposes two flavors of this: noise_sensitivity_relevant (the default, which measures leakage into the answer from irrelevant chunks when relevant chunks were also present) and a variant that considers the answer against irrelevant chunks alone. In practice, most teams use the default relevant-mode version, since it reflects the realistic production scenario of a mixed retrieval set.

Setting Up the Evaluation Dataset

Because Noise Sensitivity needs to distinguish relevant from irrelevant context, your evaluation samples need four fields: the user question, the retrieved contexts, the generated answer, and a reference (ground truth) answer. This is one reason teams skip this metric early on — it requires reference answers, which means you need either a labeled test set or a strong LLM to generate synthetic ground truths.

Here is a minimal setup using the Ragas Python SDK:

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import NoiseSensitivity
from langchain_openai import ChatOpenAI
from ragas.llms import LangchainLLMWrapper

# Sample data mimicking a support-bot RAG pipeline
data = {
    "question": [
        "What is the cancellation policy for the Pro plan?"
    ],
    "answer": [
        "The Pro plan can be cancelled anytime, but requires a 90-day "
        "notice period and a signed offboarding form."
    ],
    "contexts": [
        [
            "Pro plan subscriptions can be cancelled anytime with a "
            "30-day notice period.",
            "Enterprise plan subscriptions require a 90-day notice "
            "period and a signed offboarding form.",
        ]
    ],
    "reference": [
        "Pro plan subscriptions can be cancelled anytime with a "
        "30-day notice period."
    ],
}

dataset = Dataset.from_dict(data)

evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))

result = evaluate(
    dataset=dataset,
    metrics=[NoiseSensitivity(llm=evaluator_llm)],
)

print(result)

Running this, you would expect a high Noise Sensitivity score, because the generated answer's "90-day notice period" claim is only supported by the irrelevant Enterprise-plan chunk, not by the relevant Pro-plan chunk. That is precisely the failure mode we described earlier, and the metric catches it even though a naive Faithfulness check would have let it slide.

Building a Realistic Batch Evaluation

In a real project you will not be hand-writing single examples — you will be running Noise Sensitivity across a batch of test cases pulled from your RAG pipeline's actual retrieval and generation logs. Here is a more complete example that shows how you might wire this into an evaluation script that also deliberately injects noise to stress-test the pipeline.

import random
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import NoiseSensitivity, Faithfulness, ContextPrecision
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI


def build_noisy_contexts(relevant_chunks, distractor_pool, num_distractors=2):
    """
    Simulates imperfect retrieval by mixing in distractor chunks
    that are topically close but not actually relevant.
    """
    distractors = random.sample(distractor_pool, k=num_distractors)
    mixed = relevant_chunks + distractors
    random.shuffle(mixed)
    return mixed


def run_generation(question, contexts, rag_chain):
    """
    Placeholder for your actual RAG generation call.
    Swap this out for your real chain, agent, or API call.
    """
    return rag_chain.invoke({"question": question, "contexts": contexts})


def build_eval_dataset(test_cases, distractor_pool, rag_chain):
    questions, answers, contexts_list, references = [], [], [], []

    for case in test_cases:
        noisy_contexts = build_noisy_contexts(
            case["relevant_chunks"], distractor_pool
        )
        generated_answer = run_generation(
            case["question"], noisy_contexts, rag_chain
        )

        questions.append(case["question"])
        answers.append(generated_answer)
        contexts_list.append(noisy_contexts)
        references.append(case["reference_answer"])

    return Dataset.from_dict({
        "question": questions,
        "answer": answers,
        "contexts": contexts_list,
        "reference": references,
    })


evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))

metrics = [
    NoiseSensitivity(llm=evaluator_llm),
    Faithfulness(llm=evaluator_llm),
    ContextPrecision(llm=evaluator_llm),
]

# dataset = build_eval_dataset(test_cases, distractor_pool, rag_chain)
# results = evaluate(dataset=dataset, metrics=metrics)
# results_df = results.to_pandas()
# print(results_df[["question", "noise_sensitivity_relevant", "faithfulness"]])

Notice the deliberate structure here: build_noisy_contexts injects distractor chunks into every test case on purpose. This is the core idea of stress-testing retrieval robustness — you are not just evaluating your pipeline on its best day, you are evaluating it under conditions that mimic a retriever having a bad day. If your Noise Sensitivity scores stay low even with two or three distractors mixed in, that is strong evidence your generator's prompt and grounding instructions are doing their job. If scores spike, you have found a concrete, reproducible weakness before your users do.

Reading and Interpreting the Scores

A single Noise Sensitivity number in isolation does not tell you much. What matters is how you interpret it in context.

• A score near 0 across your test set means your generator reliably ignores irrelevant chunks and only draws claims from context that actually supports the correct answer. This is what you want, and it is a strong signal that your prompt engineering around "only use the provided context that is relevant to the question" is effective.
• A moderate score (roughly 0.1 to 0.3) suggests occasional leakage. This is common in production systems and is not necessarily alarming, but it is worth triaging which specific questions triggered it. Often you will find a pattern — for example, questions where the ground truth requires distinguishing between two similar entities (two product tiers, two policy versions, two dates) are disproportionately represented in the failures.
• A high score (above 0.4 or so) is a red flag. It means a meaningful fraction of your answer content is being pulled from chunks that should not be informing the response at all. This usually points to one of three root causes: the generation prompt does not instruct the model to disregard irrelevant context, the retrieved chunks are not clearly enough differentiated for the model to tell them apart, or the model itself (especially smaller or weaker models) struggles with the multi-document reasoning needed to keep sources separate.

It also helps to cross-reference Noise Sensitivity against Context Precision. If Context Precision is low (meaning your retriever pulls in a lot of irrelevant chunks) and Noise Sensitivity is also high, the priority fix is retrieval — better chunking, better embeddings, or reranking. If Context Precision is reasonably high but Noise Sensitivity is still elevated, the problem is squarely in generation — your prompt or model is not robust to the small amount of noise that inevitably gets through even a good retriever.

Practical Ways to Reduce Noise Sensitivity

Once you have identified that your pipeline is noise-sensitive, there are concrete levers to pull, roughly in order of effort.

1. Tighten the generation prompt. Explicitly instruct the model to only use context that directly answers the question and to ignore chunks that are topically related but do not address the specific query. Adding a line like "If a piece of context does not directly answer the question, disregard it even if it seems related" measurably reduces leakage in many pipelines.
2. Add source attribution requirements. Asking the model to cite which chunk each claim came from (even if you strip the citations before showing the user) forces it to reason more carefully about grounding, which tends to reduce noise leakage as a side effect.
3. Improve reranking before generation. If a reranker pushes truly irrelevant chunks further down or filters them out entirely before they reach the generator, there is simply less noise for the model to get confused by. This attacks the problem at the retrieval layer instead of relying on the generator to always get it right.
4. Reduce the number of chunks passed to the generator. Sometimes teams over-retrieve "just in case," passing 8 or 10 chunks per query. Every additional chunk is another opportunity for noise. Tightening top_k and relying more on a good reranker to select fewer, higher-quality chunks often cuts Noise Sensitivity scores directly.
5. Try a stronger or better-instructed generator model. In head-to-head tests across model families, weaker models are consistently more prone to conflating similar-sounding chunks. If you have exhausted prompt and retrieval fixes, this is a lever worth pulling before assuming the whole architecture is broken.

Common Pitfalls When Adopting This Metric

A few mistakes come up repeatedly when teams start using Noise Sensitivity.

• Skipping the reference field. Because Noise Sensitivity needs to know what "relevant" context actually looks like, it depends on a reference answer. Teams that only have question, contexts, and answer (no ground truth) cannot run this metric meaningfully. If you do not have labeled references yet, invest in building even a small golden set of 30 to 50 question-answer pairs before trying to rely on this metric at scale.
• Testing only on "easy" retrieval. If your evaluation dataset only contains cases where the retriever happens to fetch exactly the right chunks, Noise Sensitivity will look great — because there is no noise to be sensitive to. The whole value of this metric comes from deliberately including noisy, imperfect retrieval scenarios in your test set, as shown in the batch example above.
• Treating it as a replacement for Faithfulness or Context Precision. Noise Sensitivity is a complement, not a substitute. Run it alongside Faithfulness, Answer Relevancy, Context Precision, and Context Recall to get a full picture. A pipeline can fail on any one of these axes independently of the others.
• Ignoring the LLM-judge cost. Like most Ragas metrics, Noise Sensitivity relies on an LLM to decompose claims and classify context relevance, which means every evaluation run costs tokens and time. For large test sets, consider using a cheaper model as the judge (many teams use a smaller model like gpt-4o-mini) and reserve your most expensive model for spot-checking disagreements.

Wrapping Up

Noise Sensitivity fills a real gap in RAG evaluation. Faithfulness tells you whether the model invents facts out of thin air. Context Precision and Recall tell you whether your retriever fetched the right documents. But neither of those tells you what happens in the messy middle ground where retrieval is *mostly* right but not perfectly right — which, in production, is most of the time. Noise Sensitivity is the metric that specifically asks whether your generator can tell the difference between "this chunk answers the question" and "this chunk merely sounds like it might."

If you have not run this metric on your pipeline yet, the fastest way to start is exactly what we did in the batch example above: take your existing golden test set, deliberately mix in a couple of distractor chunks per question, run generation, and score the output with NoiseSensitivity. You will likely be surprised at how much leakage shows up even in pipelines that score well on every other metric.

If you want a structured, hands-on walkthrough of this metric and the rest of the Ragas evaluation suite — including how to build golden datasets, wire up custom LLM judges, and interpret results across real pipelines — that is exactly what we cover in the Ragas Tutorial course on teachyou.ai. It walks through each metric with working code, from Faithfulness and Answer Relevancy through to the more advanced ones like Noise Sensitivity, so you can build evaluation pipelines that actually catch the failures your users would otherwise find first.