Ragas

Interpreting Low Ragas Scores: A Diagnostic Checklist

Ira Menon · Jun 2, 2026 · 14 min read

The 0.41 Faithfulness Score That Ruins Your Afternoon

You ran your evaluation suite, watched the progress bar crawl through a hundred test cases, and finally the numbers landed. Faithfulness: 0.41. Context precision: 0.38. Answer relevancy looks fine at 0.87, but that single number cannot save the rest. Now what?

This is the moment most teams get stuck. Ragas gives you a score, not a diagnosis. It tells you *that* something is wrong, not *what*. A low faithfulness score could mean your retriever is pulling irrelevant chunks, your LLM is hallucinating on top of good context, your chunking strategy is fragmenting facts across boundaries, or your ground truth dataset was written carelessly. Each of those has a completely different fix, and guessing wrong wastes days.

This article is a diagnostic checklist. Not a metrics glossary — you can read the Ragas docs for that — but a structured way to walk backward from a bad number to a root cause. We will go metric by metric, show you how to inspect the underlying traces, and give you code you can run today against your own evaluation results.

Start By Separating Retrieval Metrics From Generation Metrics

The single biggest mistake in Ragas debugging is treating all metrics as one blob. Ragas metrics split cleanly into two families, and mixing them up sends you down the wrong path every time.

Retrieval metrics (context precision, context recall, context relevancy) tell you whether your retriever fetched the right chunks. These depend entirely on your vector store, embedding model, chunking, and retrieval algorithm — the LLM generating the answer is irrelevant here.

Generation metrics (faithfulness, answer relevancy, answer correctness) tell you whether the LLM did something reasonable *given the context it received*. If the context was garbage, faithfulness will look bad even though your generation prompt is perfect, because there was nothing faithful to be to.

The diagnostic rule: always check context precision and context recall first. If those are low, fix retrieval before you touch your prompt. If those are high but faithfulness or answer relevancy are low, the problem is downstream in generation.

import pandas as pd

def triage(results_df: pd.DataFrame) -> str:
    """Quick triage: is this a retrieval problem or a generation problem?"""
    avg_precision = results_df["context_precision"].mean()
    avg_recall = results_df["context_recall"].mean()
    avg_faithfulness = results_df["faithfulness"].mean()

    if avg_precision < 0.6 or avg_recall < 0.6:
        return "RETRIEVAL PROBLEM: fix chunking/embeddings/retriever before anything else"
    elif avg_faithfulness < 0.6:
        return "GENERATION PROBLEM: retrieval is fine, check prompt and model hallucination"
    else:
        return "Both stages look healthy — check dataset quality or metric configuration"

print(triage(results_df))

Run this before you do anything else. It takes ten seconds and saves you from spending an afternoon rewriting prompts when the real problem is that your retriever never fetched the answer in the first place.

Diagnosing Low Context Precision

Context precision measures whether the relevant chunks are ranked near the top of what your retriever returned. A low score here means one of three things: your retriever is surfacing noise above signal, your chunk size is wrong, or your query itself is a bad match for how your documents are phrased.

The first thing to check is whether precision is uniformly bad or bad only for certain question types. Pull the worst-scoring rows and read them by hand.

worst_precision = results_df.nsmallest(15, "context_precision")[
    ["question", "contexts", "context_precision"]
]

for _, row in worst_precision.iterrows():
    print(f"Q: {row['question']}")
    print(f"Score: {row['context_precision']:.2f}")
    print(f"Top context: {row['contexts'][0][:200]}...")
    print("-" * 60)

Look for patterns. Common culprits:

Chunk size mismatch. If your chunks are 2000 tokens and the answer is one sentence buried in the middle, the embedding for that chunk represents the whole chunk's topic, not the specific fact. Precision suffers because semantically similar-but-irrelevant chunks rank just as high.
Query-document vocabulary gap. Users ask "how do I cancel my plan" but your docs say "subscription termination procedure." Dense embeddings handle this better than keyword search, but not perfectly — consider hybrid retrieval.
Missing metadata filtering. If you retrieve across all documents when the question is clearly scoped to one product or one version, you are diluting precision by letting irrelevant-but-topically-similar chunks compete.
Wrong top-k. Retrieving 10 chunks when only 2 are relevant tanks precision even if recall is perfect. Try lowering k and rerunning the eval.

A practical fix worth trying first: add a reranker between retrieval and generation. Rerankers (cross-encoders) are slower but far better at precision because they score query-chunk pairs jointly instead of relying on pre-computed embedding similarity.

Diagnosing Low Context Recall

Context recall asks a different question: of everything needed to answer correctly, how much did you actually retrieve? Low recall means the answer's ingredients are scattered outside your retrieved set entirely — this is often the worse of the two retrieval failures because no amount of clever prompting can recover information the model never saw.

Low recall usually traces back to one of these:

Chunking split a fact across a boundary. The answer requires sentence A and sentence B, but they landed in different chunks, and only one was retrieved.
The ground truth requires synthesis across multiple documents. If your test set has questions that require combining facts from three different pages, and you only retrieve top-3 chunks from a single embedding pass, you will systematically underretrieve.
Embedding model is too generic. A general-purpose embedding model may not capture domain-specific similarity well, especially in fields like legal, medical, or highly technical documentation with specialized vocabulary.
top-k is simply too low. This is the boring but common answer — bump k from 3 to 8 and see if recall jumps significantly. If it does, you were retrieval-starved.

def recall_gap_analysis(row, k_values=(3, 5, 8, 12)):
    """Re-run retrieval at increasing k to see where recall saturates."""
    from your_retriever import retriever  # your existing retriever instance

    results = {}
    for k in k_values:
        retrieved = retriever.get_relevant_documents(row["question"], k=k)
        retrieved_text = " ".join([d.page_content for d in retrieved])
        hit = row["ground_truth"].lower() in retrieved_text.lower()
        results[k] = hit
    return results

If recall keeps climbing as k increases and never plateaus, your chunking is fragmenting information too aggressively — the fix is bigger chunks or overlapping windows, not just a bigger k. If recall plateaus early and stays low, the information may not be embeddable well at all, and you need better source documents or metadata-based retrieval instead of pure similarity search.

Diagnosing Low Faithfulness

Faithfulness is the metric people panic about most, and for good reason — it is Ragas's proxy for hallucination. It measures whether every claim in the generated answer can be traced back to something actually present in the retrieved context.

Before assuming your model is hallucinating wildly, confirm context precision and recall are healthy first (see above). If they are not, low faithfulness is actually a retrieval symptom wearing a generation costume. Assuming context is good, here is what actually causes low faithfulness:

The model is adding reasonable-sounding elaboration. LLMs love to be helpful. Asked "what is the refund window," a model might answer "the refund window is 30 days, and we recommend keeping your receipt for verification" — that second clause is true-sounding advice invented from nowhere, and it tanks faithfulness even though the core answer was correct.
The model is filling gaps with prior/parametric knowledge. If the context is incomplete, a well-trained model will often patch the hole with what it already knows from pretraining rather than saying "I don't know." This looks like a good answer and scores terribly on faithfulness.
Prompt doesn't instruct grounding strictly enough. If your system prompt says "answer the question using the context" but doesn't say "only use the context, and say you don't know if the context is insufficient," you are inviting exactly this behavior.
Numeric or list-heavy answers get subtly wrong. Faithfulness checks are sensitive to small factual drift — a percentage rounded differently, an extra item added to a list. These read as unfaithful even when a human would call the answer "close enough."

The fix that moves faithfulness the most, in practice, is tightening the system prompt:

STRICT_GROUNDING_PROMPT = """You must answer using ONLY the information in the
context below. Do not use any outside knowledge, even if you are confident it
is correct. If the context does not contain enough information to answer the
question, respond exactly with: "I don't have enough information to answer this."

Do not add extra advice, caveats, or elaboration that is not directly stated
in the context.

Context:
{context}

Question:
{question}

Answer:"""

Re-run your eval after this change alone, before touching anything else. In most pipelines this single prompt fix recovers a meaningful chunk of faithfulness score because it removes the model's incentive to be "helpfully wrong."

Diagnosing Low Answer Relevancy

Answer relevancy is often misunderstood. It does not measure correctness — it measures whether the answer actually addresses the question asked, by checking how well synthetic questions generated from the answer match the original question. A technically correct but rambling or tangential answer scores low here even if every fact in it is true.

Common causes:

The model answers a broader question than was asked. User asks "does the free tier include API access," and the model answers with a full paragraph about all tier differences. True, thorough, but not relevant to the specific question.
The model hedges excessively. Answers padded with disclaimers, caveats, and "it depends" framing without ever committing to a direct answer will score lower because the semantic core of the answer drifts from the question's core.
Retrieved context nudges the model off-topic. If the top chunk is tangentially related, the model may anchor on it and answer the *chunk's* implicit question rather than the *user's* actual question.

Read actual transcripts, not just the score:

low_relevancy = results_df.nsmallest(10, "answer_relevancy")

for _, row in low_relevancy.iterrows():
    print(f"Question: {row['question']}")
    print(f"Answer: {row['answer'][:300]}")
    print(f"Relevancy: {row['answer_relevancy']:.2f}\n")

If you see a pattern of over-answering, tighten your prompt to demand directness: "Answer the specific question asked. Do not provide additional information unless it is necessary to answer the question." If you see hedging, it is often a symptom of the model being unsure because of a weak context — which loops back to a retrieval issue in disguise.

Check Your Test Dataset Before You Blame Your Pipeline

This is the step teams skip, and it is often where the real bug lives. Ragas scores are only as good as the ground truth and question set you evaluate against. A shockingly common cause of "low scores" is a badly constructed evaluation dataset, not a badly built RAG pipeline.

Things to audit in your dataset:

Are ground truth answers actually derivable from your document corpus? If your test set was synthetically generated by an LLM from a different document version, or written by someone who had access to information not in your retrieval corpus, your pipeline is being graded on an impossible task.
Are questions ambiguous or underspecified? "What's the pricing?" without specifying which product or plan will produce genuinely different-but-valid answers depending on which chunk gets retrieved, and Ragas will penalize the mismatch even though the pipeline behaved reasonably.
Is the ground truth stale? Documentation changes. If your ground truth was written three product releases ago, your current pipeline may be *correctly* citing new information that doesn't match old ground truth.
Are there duplicate or near-duplicate questions with contradictory ground truths? This happens more than people expect when datasets are merged from multiple sources or generated in batches without deduplication.

def audit_dataset_grounding(dataset, corpus_texts):
    """Flag ground truths that don't appear to be supported by any corpus document."""
    unsupported = []
    for item in dataset:
        gt = item["ground_truth"].lower()
        gt_keywords = set(gt.split()) - COMMON_STOPWORDS
        found = any(
            len(gt_keywords & set(doc.lower().split())) / max(len(gt_keywords), 1) > 0.3
            for doc in corpus_texts
        )
        if not found:
            unsupported.append(item["question"])
    return unsupported

This is a rough heuristic, not a rigorous check, but it is enough to surface candidates for manual review. If ten percent of your low-scoring questions turn out to have unsupported or stale ground truth, you have found a dataset problem, not a pipeline problem — and no amount of prompt engineering will fix that.

Watch for Metric Configuration Mistakes

Sometimes the "low score" isn't a real problem at all — it's a misconfigured evaluator. This is worth ruling out early because it is fast to check and embarrassing to miss.

Wrong LLM as judge. Ragas metrics use an LLM under the hood to make judgments (e.g., faithfulness decomposes the answer into claims and checks each against context). If you configured a weak or cheap model as the judge, you'll get noisier, harsher scores than if you use a stronger judge model. Check which model your evaluator_llm is actually pointing to.
Embedding model mismatch for semantic metrics. Context relevancy and answer similarity metrics depend on an embedding model. If it's misconfigured or defaulting to something inappropriate for your language/domain, scores will be unreliable across the board, not just occasionally.
Async/batch failures silently zeroing out rows. In some Ragas versions, if the judge LLM call fails or times out on a row (rate limits are the usual suspect), that row can silently score as 0 or NaN, dragging your average down without any real quality problem. Always check for NaNs and near-zero outliers before trusting the aggregate mean.

import numpy as np

def check_for_evaluation_failures(results_df, metric_cols):
    """Surface rows where the metric likely failed to compute rather than
    reflecting real quality issues."""
    suspicious = results_df[
        results_df[metric_cols].apply(lambda col: (col == 0) | col.isna()).any(axis=1)
    ]
    print(f"{len(suspicious)} of {len(results_df)} rows have zero/NaN metrics")
    return suspicious

If a meaningful fraction of your "low scores" are actually failed evaluations rather than failed pipeline responses, your real average is better than it looks, and you should fix the evaluation run (retry with backoff, use a more reliable judge model) before drawing conclusions about your RAG system.

Build a Repeatable Diagnostic Loop, Not a One-Off Investigation

The teams that get good at this stop treating a low Ragas score as a fire drill and instead build a small, repeatable diagnostic script that runs every time scores dip. The checklist above collapses into a short routine:

Split metrics into retrieval vs. generation and check which family is failing.
If retrieval is failing, inspect worst-scoring rows for chunking, top-k, and query-vocabulary mismatches.
If generation is failing, inspect faithfulness failures for parametric-knowledge leakage and check answer relevancy failures for over-answering or hedging.
Audit a sample of the ground truth dataset for staleness or ambiguity.
Rule out evaluator misconfiguration and silent failures before trusting the raw numbers.

def diagnostic_report(results_df):
    print("=== Ragas Diagnostic Report ===\n")
    print(triage(results_df))
    print(f"\nWorst 5 context_precision rows:\n{results_df.nsmallest(5, 'context_precision')[['question']]}")
    print(f"\nWorst 5 faithfulness rows:\n{results_df.nsmallest(5, 'faithfulness')[['question']]}")
    print(f"\nRows with possible evaluator failure: {len(check_for_evaluation_failures(results_df, ['faithfulness', 'context_precision', 'context_recall']))}")

diagnostic_report(results_df)

Keep this script in your repo next to your eval harness. Every time you change your retriever, your prompt, your chunking strategy, or your embedding model, run it again. Over time you'll build intuition for what "normal" looks like for your specific pipeline, and a genuine regression will jump out immediately instead of triggering a confused afternoon of guesswork.

Turning Diagnosis Into Durable Pipeline Improvements

A low Ragas score is not a verdict, it's a starting point for investigation. The number alone tells you almost nothing about the fix; the trace behind the number tells you almost everything. Once you get in the habit of separating retrieval from generation, reading actual failing examples instead of staring at aggregates, and auditing your dataset with the same skepticism you apply to your model, Ragas stops feeling like a black box and starts feeling like an actual debugging tool.

The mistake to avoid is reflexively tuning the wrong layer — rewriting your generation prompt for a week when the real issue was that your retriever never surfaced the right chunk, or re-chunking your entire corpus when the real issue was three mislabeled ground truth answers in your eval set. Diagnose first, then fix, and always re-run the full metric suite after each change so you know whether you actually moved the needle or just shuffled the failure mode somewhere else.

If you want to go deeper on building this diagnostic muscle — including hands-on labs on tracing faithfulness failures, constructing evaluation datasets that don't lie to you, and wiring Ragas into a CI pipeline so regressions get caught automatically — check out the Ragas Tutorial course on teachyou.ai. It walks through exactly this kind of failure-mode-first debugging with real RAG pipelines, not toy examples.