Ragas Metrics Explained: Faithfulness, Context Precision and Recall

You shipped a RAG pipeline, the demo looked great, and then it hit production traffic and started confidently making things up half the time. Sound familiar? The problem almost never announces itself as "retrieval bug" or "generation bug" — it just shows up as "the answer feels off." You need numbers that separate those two failure modes, and that's exactly the gap Ragas metrics fill. This is a working reference for the four metrics you'll reach for most: Faithfulness, Context Precision, Context Recall, and Answer Relevancy. No hand-waving about "evaluation is important" — just what each metric actually measures, how it's computed under the hood, a worked example of a good score versus a bad one, and what a low number tells you to go fix.

Why RAG needs metric-level diagnosis, not a single score

A RAG system has two moving parts: the retriever and the generator. When the final answer is wrong, that failure could originate in either stage, or both. A single end-to-end "is this answer correct" judgment collapses that distinction and leaves you guessing.

Say a user asks about your refund policy and the bot says something incorrect. Did the retriever fetch the wrong policy document? Did it fetch the right document but miss the specific clause? Did it fetch everything needed, but the LLM ignored the context and hallucinated from its training data? Each of these has a completely different fix — reindexing, better chunking, a reranker, or a prompt change — and you can't tell which one you need from a single pass/fail judgment.

This is the core idea behind Ragas: instead of one blended score, you get separate, orthogonal signals for retrieval quality and generation quality. Faithfulness and Answer Relevancy score the generator. Context Precision and Context Recall score the retriever. Once you decompose the pipeline this way, debugging stops being guesswork and starts being triage.

All four metrics in the Ragas framework are typically computed using an LLM as a judge, applied to a structured question / context / answer (and sometimes ground-truth) tuple. That LLM-as-judge design is deliberate — these are semantic judgments, not string-matching problems, and no amount of regex is going to tell you whether a claim is "supported by" a paragraph of prose. Let's go metric by metric.

Faithfulness

Faithfulness measures whether every claim in the generated answer is actually supported by the retrieved context. It is the metric most directly aimed at catching hallucination — not "is the answer true in the real world," but "is the answer traceable to what was retrieved." That distinction matters: an answer can be factually correct and still score low on faithfulness if it wasn't grounded in the provided context (meaning the model likely pulled it from parametric memory rather than the retrieved documents), and an answer can be faithful while still being wrong, if the retrieved context itself was wrong.

Conceptually, here's how it's computed. The generated answer is first decomposed into a set of atomic claims — simple, single-fact statements. This decomposition step itself is usually done by an LLM, prompted to break a sentence like "Ragas was created by the Exploding Gradients team and supports both open-source and proprietary evaluator models" into two separate claims: (1) Ragas was created by the Exploding Gradients team, and (2) Ragas supports both open-source and proprietary evaluator models. Each claim is then checked against the retrieved context, again by an LLM judge, and marked as supported or unsupported. The Faithfulness score is the ratio of supported claims to total claims:

faithfulness_score = (number of claims supported by context) / (total number of claims in answer)

The claim-decomposition step is what makes this metric more reliable than just asking "is this answer grounded, yes or no." A long answer might be 90% grounded and 10% fabricated, and if you evaluate it holistically an LLM judge will often just round up to "yes, grounded" because most of the answer checks out. Breaking the answer into atomic units forces the judge to catch the one inserted claim that isn't backed by anything.

Worked example.

Question: "What is the maximum file size for uploads on the free plan?"

Retrieved context: "Free plan users can upload files up to 25MB. Paid plans raise this limit to 500MB and include priority processing."

Good answer (illustrative — high faithfulness): "On the free plan, the maximum upload size is 25MB." Both claims here (there's really just one: 25MB limit on free plan) map directly onto a sentence in the context. Illustrative score: something like 1.0.

Bad answer (illustrative — low faithfulness): "On the free plan, the maximum upload size is 25MB, and uploads are automatically scanned for malware before processing." The first claim is supported. The second claim — automatic malware scanning — appears nowhere in the retrieved context. It might be true of the product, it might be something the model has seen in similar SaaS documentation during pretraining, but it's not grounded in what was retrieved. Illustrative score: roughly 0.5, since one of two claims is unsupported.

What a low Faithfulness score tells you to fix. This is a generation-side problem, not a retrieval-side one — assuming the context genuinely contained what was needed. The usual culprits: your prompt isn't instructing the model firmly enough to stick to the provided context ("answer only using the information below" language is often too weak on its own); your temperature is too high for a QA task where you want conservative, grounded completions; the model is filling gaps with plausible-sounding parametric knowledge when the context is incomplete instead of saying "I don't know"; or you're not truncating/prioritizing context well and the model is drowning in irrelevant text and defaulting to its own priors. Fixes to try: tighten the system prompt with explicit instructions to refuse or hedge when context is insufficient, lower temperature, add a "cite your source sentence" requirement that forces traceability, or run a smaller instruction-tuned model that's known to follow grounding instructions more strictly.

Code snippet — computing Faithfulness with Ragas:

from ragas import evaluate
from ragas.metrics import faithfulness
from datasets import Dataset

data = {
    "question": ["What is the maximum file size for uploads on the free plan?"],
    "answer": ["On the free plan, the maximum upload size is 25MB, and uploads are automatically scanned for malware."],
    "contexts": [[
        "Free plan users can upload files up to 25MB. "
        "Paid plans raise this limit to 500MB and include priority processing."
    ]],
}

dataset = Dataset.from_dict(data)
result = evaluate(dataset, metrics=[faithfulness])
print(result)

Context Precision

Context Precision asks a narrower question than Faithfulness: of the chunks your retriever actually pulled back, how many were relevant, and were the relevant ones ranked near the top? This is a pure retrieval-quality metric — it doesn't care what the generator did with the context, only whether the retriever handed over a clean, well-ordered set of chunks or a pile of noise with the useful bit buried at position seven.

Conceptually, here's how it's computed. For each retrieved chunk, an LLM judge determines whether that chunk is relevant to answering the question (usually by comparing it against the ground-truth answer or the question itself). Then, instead of a flat relevant/irrelevant ratio, Context Precision computes a precision@k-style metric that weights relevance by rank position — a relevant chunk at position 1 contributes more to the score than a relevant chunk at position 5, because ranking quality matters for how usable the context is downstream, and often for how much of it survives context-window truncation. Concretely, this is typically implemented as an average of precision@k across the ranks where relevant items appear:

context_precision = average over k of ( precision@k * relevance_indicator(k) )

where precision@k is the proportion of relevant chunks among the top-k retrieved, and relevance_indicator(k) is 1 if the chunk at rank k is relevant and 0 otherwise. In practice, this rewards retrievers that put the good stuff first, not just retrievers that eventually include the good stuff somewhere in the list.

Worked example.

Question: "What's the cancellation policy for annual subscriptions?"

Ground truth: "Annual subscriptions can be cancelled anytime, but refunds are only issued within the first 14 days."

Good retrieval (illustrative — high Context Precision): Chunk 1 is "Annual subscription cancellations are accepted at any time; refunds apply only within a 14-day window from purchase." Chunk 2 is "Monthly subscriptions can be cancelled with no refund after the billing date has passed." Here the single most relevant chunk is ranked first, and the model barely needs chunk 2. Illustrative score: high, close to 1.0.

Bad retrieval (illustrative — low Context Precision): Chunk 1 is "Our support team is available 9am–6pm on weekdays." Chunk 2 is "We offer three subscription tiers: Basic, Pro, and Enterprise." Chunk 3 (finally) is "Annual subscription cancellations are accepted at any time; refunds apply only within a 14-day window." The relevant information exists in the retrieved set, but it's buried at the bottom, and two of three chunks are noise. Illustrative score: low, maybe 0.3, because the useful chunk is ranked last among three.

What a low Context Precision score tells you to fix. This is squarely a retrieval and ranking problem. Common causes: your embedding model isn't capturing semantic similarity well for this domain (generic embeddings often underperform on jargon-heavy or highly technical corpora); your chunking strategy is too coarse, so relevant and irrelevant content get glued into the same chunk and the whole chunk gets a middling relevance score; you're retrieving too many chunks (top_k too high) and diluting precision by design; or you don't have a reranking step, so your first-pass retriever's rough similarity ranking is being trusted as final order when it shouldn't be. Fixes: add a reranker (a cross-encoder is usually the highest-leverage single change here), tighten chunk boundaries around semantic units instead of fixed token counts, experiment with a domain-tuned or larger embedding model, and reduce top_k if you're over-retrieving just to be safe.

Context Recall

Context Recall is the mirror image of Context Precision. Precision asks "is what we retrieved relevant." Recall asks "did we retrieve everything relevant that exists." A retriever can have perfect precision — every chunk it returns is spot-on — while still missing half of what was actually needed to fully answer the question. That's a recall failure, and it looks completely different in the data than a precision failure, which is exactly why you need both.

Conceptually, here's how it's computed. Context Recall requires a ground-truth answer (this is the one metric here that typically can't run without one). The ground-truth answer is broken down into its constituent claims or sentences, similar to the claim decomposition in Faithfulness. Then, for each claim in the ground truth, an LLM judge checks whether that claim can be attributed to (is supported by) the retrieved context. The score is the proportion of ground-truth claims that are covered:

context_recall = (number of ground-truth claims supported by retrieved context) / (total number of ground-truth claims)

If the ground truth says "X, Y, and Z" and your retrieved context only supports X and Y, your recall is roughly 0.67 — one third of what was needed to fully answer the question never made it into the retrieved set, no matter how good the generator is.

Worked example.

Question: "What are the requirements to qualify for the enterprise discount?"

Ground truth: "To qualify for the enterprise discount, an account must have at least 50 seats, a minimum 12-month commitment, and be on the Pro plan or higher."

Good retrieval (illustrative — high Context Recall): Retrieved chunks together state the 50-seat minimum, the 12-month commitment requirement, and the Pro-plan-or-higher requirement, even if spread across two different chunks. All three ground-truth claims are covered. Illustrative score: 1.0.

Bad retrieval (illustrative — low Context Recall): Retrieved chunks mention the 50-seat minimum and the 12-month commitment, but nothing in the retrieved set mentions the Pro-plan requirement — maybe that detail lives in a different document that never got indexed, or it exists but didn't rank high enough to be pulled into top_k. Two of three ground-truth claims are covered. Illustrative score: roughly 0.67.

What a low Context Recall score tells you to fix. This is a coverage problem, and it's usually more structural than a precision problem. Check: is the source document that contains the missing information even in your index (a surprisingly common miss — content that lives in a PDF, a Notion page, or a support macro that never got ingested)? Is your chunking splitting a single logical requirement across a chunk boundary so that no single chunk fully expresses it, causing the relevance judge to score it as unsupported? Is top_k too low, so the relevant-but-lower-ranked chunk is getting cut off before it reaches the generator? Is your retrieval strategy purely dense/semantic when the missing fact would be easier to catch with a keyword or hybrid search pass? Fixes: audit your ingestion pipeline for coverage gaps, increase top_k (this trades off against precision, so tune both together), switch to hybrid retrieval (dense + BM25) for the cases where semantic search alone underperforms, and consider query expansion or query rewriting so a single user question generates multiple retrieval queries that each target a different sub-requirement.

Answer Relevancy

Answer Relevancy is the one metric here that isn't really about correctness at all — it measures whether the generated answer actually addresses the question that was asked, independent of whether the content is true or well-grounded. You can have an answer that is perfectly faithful to the context and still scores low on relevancy, if it's evasive, padded with irrelevant tangents, or answers a nearby-but-different question instead of the one the user actually asked.

Conceptually, here's how it's computed. This metric works backwards from the answer. An LLM is prompted to generate several plausible questions that the given answer would be a good response to. Each of these generated questions is then embedded, along with the original question, and the cosine similarity between the original question's embedding and each generated question's embedding is computed. The average of these similarities is the Answer Relevancy score:

answer_relevancy = average_cosine_similarity(embedding(original_question), embedding(generated_question_i))  for i in generated questions

The intuition: if the answer is genuinely relevant and on-topic, an LLM reverse-engineering "what question would this answer be responding to" should land close to the original question every time. If the answer wandered off-topic, hedged excessively, or answered something adjacent, the reverse-engineered questions will drift semantically from what was actually asked.

Worked example.

Question: "How do I reset my two-factor authentication if I lost my phone?"

Good answer (illustrative — high Answer Relevancy): "If you've lost the device with your 2FA app, go to Account Settings > Security > Two-Factor Authentication and select 'Lost my device.' You'll need to verify your identity via your backup email, after which 2FA will be disabled so you can re-enroll a new device." A model reverse-engineering questions from this answer would land squarely on "how do I recover 2FA access without my phone" — high similarity to the original question.

Bad answer (illustrative — low Answer Relevancy): "Two-factor authentication adds an extra layer of security to your account by requiring a second verification step beyond your password. It's a good idea to keep 2FA enabled at all times to protect against unauthorized access." This is accurate and might even be faithful to some retrieved context about what 2FA is, but it never actually answers "how do I reset it if I lost my phone." Reverse-engineered questions from this answer would cluster around "what is 2FA" or "why should I use 2FA" — noticeably distant from the original question. Illustrative score: low.

Note the important asymmetry here: the bad answer above could still score reasonably on Faithfulness (every sentence in it might well be supported by some retrieved chunk about 2FA in general) while scoring poorly on Answer Relevancy. That's exactly why you need both metrics rather than assuming a faithful answer is automatically a good answer.

What a low Answer Relevancy score tells you to fix. This is usually a prompt-engineering or query-understanding problem rather than a retrieval problem. Common causes: the generator prompt doesn't emphasize directly addressing the user's specific question (models left to their own devices often drift toward giving general background instead of the specific procedural answer requested); the retrieved context itself is topically adjacent but doesn't contain the precise answer, so the model pads around the gap with related-but-non-answering content; or the question is ambiguous and the model picked a reasonable-but-wrong interpretation. Fixes: tighten the generation prompt to explicitly instruct "answer the specific question asked, do not provide general background unless it's necessary to answer," add few-shot examples showing the desired directness, and if the issue correlates with certain question types, consider a query-classification step that adjusts the prompt template based on whether the user is asking "what is X" versus "how do I do X."

Putting the four together

Here's the practical workflow: don't look at these metrics one at a time in isolation — look at the pattern across all four for a given failing example, because the combination tells you where in the pipeline to look first.

• Low Context Precision + low Context Recall — your retrieval is broadly broken. Start with indexing and chunking before touching anything else.
• High Context Precision + low Context Recall — the retriever finds clean, relevant chunks but misses other necessary information. Look at top_k and coverage gaps in your index.
• High Context Recall + low Context Precision — you're retrieving everything you need, buried among a lot of noise. A reranker is probably your fastest win.
• High Context Precision and Recall + low Faithfulness — retrieval is fine, the generator is the problem. Fix your grounding prompt.
• High Faithfulness + low Answer Relevancy — the model is accurately reporting what it found, but what it found (or how it's framing the response) doesn't address the actual question. Look at query understanding and prompt directness.

Running these four metrics as a regression suite against a held-out evaluation set — the same way you'd run unit tests against code — is what turns RAG evaluation from vibes-based spot-checking into something you can track release over release. A chunking change, an embedding model swap, or a new system prompt should each be validated against this suite before it ships, not just eyeballed on three example queries.

If you want to go deeper into how retrieval and generation fit together architecturally, and how these evaluation techniques slot into a production pipeline from ingestion to reranking to generation, that's exactly the ground we cover, hands-on, in Introduction to RAG — the course walks through building and evaluating a real retrieval-augmented pipeline end to end, including where metrics like these catch the failures that a demo never shows you.