RAG Evaluation: Precision and Recall

RAG precision and recall are the two numbers that tell you whether your retrieval-augmented generation pipeline is actually retrieving the right chunks, not just chunks that look plausible. Precision measures how much of what you retrieved is relevant; recall measures how much of what's relevant you actually retrieved. Get either one wrong and the generation step downstream inherits the problem, no matter how good your prompt is. This article walks through the math, shows you how to compute both metrics by hand, and then hooks them into an automated evaluation loop using open-source tooling.

Most teams building a RAG system jump straight to "does the answer look right" and skip the retrieval layer entirely. That's backwards. If your retriever pulls the wrong five chunks, the language model is being asked to hallucinate a correct answer from bad evidence. Precision and recall let you catch that failure mode before it ever reaches the generation step.

What Precision and Recall Mean in a RAG Pipeline

Borrowed straight from classic information retrieval, the two metrics answer different questions:

• Precision: Of the chunks the retriever returned, how many were actually relevant to the query?
• Recall: Of all the chunks that were relevant somewhere in the corpus, how many did the retriever actually return?

The formulas:

Precision = (relevant chunks retrieved) / (total chunks retrieved)
Recall    = (relevant chunks retrieved) / (total relevant chunks in the corpus)

A retriever that returns one chunk, and that chunk happens to be perfect, scores precision 1.0 but probably has terrible recall if the answer actually needed evidence spread across three documents. A retriever that returns fifty chunks to be safe will likely have high recall but precision that craters, because the language model now has to sift through forty-five irrelevant passages to find the five that matter, and every added irrelevant passage is a chance for the model to get distracted or contradict itself.

In a RAG system these two metrics are in tension, and the trade-off point depends on your top_k setting, your chunking strategy, and how your reranker (if you have one) is tuned. There is no single "good" precision or recall value; it depends entirely on your corpus and your tolerance for irrelevant context in the prompt window.

Why Standard IR Metrics Don't Fully Capture RAG Quality

Classic precision and recall assume a fixed set of "relevant documents" per query, decided ahead of time by a human labeler. That works fine for a search engine benchmark. RAG breaks this assumption in three ways.

First, relevance is not binary. A chunk can be partially relevant, containing one useful sentence buried in three paragraphs of noise, and traditional precision treats it as either a hit or a miss.

Second, order matters more in RAG than in search results a human scrolls through. If the correct chunk is retrieved but placed eighth out of ten in the context window, the "lost in the middle" effect means the model may still ignore it. Precision and recall alone say nothing about position.

Third, RAG has a second failure surface downstream: even with perfect retrieval, the generator can still misread, ignore, or contradict the retrieved evidence. That's why frameworks like Ragas split the evaluation into retrieval metrics (context precision, context recall) and generation metrics (faithfulness, answer relevancy) rather than reporting one blended score. We'll cover both later in this article, but the retrieval half is where you should start, because a broken generator built on solid retrieval is a much easier bug to find than the reverse.

Setting Up a Test Set for RAG Evaluation

You cannot compute precision or recall without ground truth. Before writing any evaluation code, build a small labeled test set:

# eval_dataset.py
eval_set = [
    {
        "question": "What is the maximum context window for the model used in our chatbot?",
        "ground_truth_chunk_ids": ["doc12_chunk3", "doc12_chunk4"],
    },
    {
        "question": "How does the refund policy handle partial shipments?",
        "ground_truth_chunk_ids": ["doc45_chunk1"],
    },
    {
        "question": "What authentication method does the public API require?",
        "ground_truth_chunk_ids": ["doc7_chunk2", "doc9_chunk5"],
    },
]

Twenty to fifty questions is enough to start catching regressions. Pull the questions from real support tickets, real user queries in your logs, or have a subject-matter expert write them against your actual document set. Each question needs a ground_truth_chunk_ids list: the chunk IDs a human has confirmed are the evidence needed to answer correctly. This is the part teams skip and it's the part that makes everything downstream trustworthy. Without labeled ground truth, "precision" is just a number with no anchor to reality.

Computing Retrieval Precision and Recall in Python

Once you have labeled data, computing the metrics is a straightforward set comparison. Here's a minimal implementation you can drop into any RAG pipeline that exposes a retrieve(query, top_k) function:

def precision_at_k(retrieved_ids, relevant_ids):
    if not retrieved_ids:
        return 0.0
    hits = len(set(retrieved_ids) & set(relevant_ids))
    return hits / len(retrieved_ids)

def recall_at_k(retrieved_ids, relevant_ids):
    if not relevant_ids:
        return 0.0
    hits = len(set(retrieved_ids) & set(relevant_ids))
    return hits / len(relevant_ids)

def evaluate_retriever(retriever, eval_set, top_k=5):
    precisions, recalls = [], []
    for item in eval_set:
        retrieved = retriever.retrieve(item["question"], top_k=top_k)
        retrieved_ids = [chunk.id for chunk in retrieved]
        relevant_ids = item["ground_truth_chunk_ids"]

        precisions.append(precision_at_k(retrieved_ids, relevant_ids))
        recalls.append(recall_at_k(retrieved_ids, relevant_ids))

    avg_precision = sum(precisions) / len(precisions)
    avg_recall = sum(recalls) / len(recalls)
    return {"precision": avg_precision, "recall": avg_recall}

# Usage
results = evaluate_retriever(my_retriever, eval_set, top_k=5)
print(f"Precision@5: {results['precision']:.3f}")
print(f"Recall@5: {results['recall']:.3f}")

Run this after every change to your chunking size, embedding model, or reranker, and track the two numbers over time in a spreadsheet or a simple CSV log. The moment either number drops after a change, you know exactly what caused the regression, because you changed one variable.

If you also want the F1 score, which balances the two into a single number for quick comparisons across experiments:

def f1_score(precision, recall):
    if precision + recall == 0:
        return 0.0
    return 2 * (precision * recall) / (precision + recall)

f1 = f1_score(results["precision"], results["recall"])
print(f"F1@5: {f1:.3f}")

F1 is convenient for a leaderboard-style comparison of five different retriever configurations, but don't rely on it exclusively. A pipeline answering medical or legal questions should weight recall heavily, missing evidence is worse than including one extra irrelevant chunk, so watch the individual numbers, not just the blended score.

Precision@k, Recall@k, and Why k Matters

Every precision and recall number in RAG is implicitly a function of k, the number of chunks you retrieve. Report top_k alongside every metric, and test multiple values of k before locking in a production setting:

for k in [1, 3, 5, 10, 20]:
    results = evaluate_retriever(my_retriever, eval_set, top_k=k)
    print(f"k={k:2d}  precision={results['precision']:.3f}  recall={results['recall']:.3f}")

You'll typically see a curve like this: precision starts high and falls as k grows, because you're adding more chunks and most of the additions are noise. Recall rises with k and then plateaus once you've captured all the relevant chunks in the corpus. The useful k is usually where the recall curve flattens, retrieving further past that point only hurts precision and adds tokens to your prompt without adding new evidence.

If your recall never reaches something close to 1.0 even at k=20, the problem isn't your reranker or your top_k setting, it's that the correct chunks aren't embedding close to the query in the first place. That points to a chunking or embedding model problem, not a retrieval-count problem, and no amount of tuning k will fix it.

Using Ragas to Automate RAG Precision and Recall

Hand-rolled precision/recall works for the retrieval layer when you have exact chunk-ID ground truth. In practice, many teams don't want to label chunk IDs by hand for every question, and instead use an LLM-as-judge to estimate relevance against a reference answer. The Ragas library (installed via pip install ragas) implements this pattern and is worth adopting once your hand-rolled version proves the concept.

from ragas import evaluate
from ragas.metrics import context_precision, context_recall
from datasets import Dataset

data = {
    "question": [item["question"] for item in eval_set],
    "contexts": [retrieve_contexts(item["question"]) for item in eval_set],
    "ground_truth": [item["reference_answer"] for item in eval_set],
}

dataset = Dataset.from_dict(data)

results = evaluate(
    dataset,
    metrics=[context_precision, context_recall],
)

print(results)

Ragas uses an LLM call under the hood to judge whether each retrieved context sentence is relevant to answering the question, then aggregates that into a precision-like score, and separately checks whether the reference answer's claims can be attributed back to the retrieved contexts for the recall-like score. This is more expensive per evaluation run (it costs API calls) than the exact-match version above, but it scales to hundreds of questions without manual chunk labeling, and it handles the partial-relevance problem that binary set comparison misses.

Run the exact-match version during development on your labeled set of twenty to fifty questions, where you trust the ground truth completely. Run the Ragas-style LLM-judged version on a larger, less curated set of a few hundred real user questions as a periodic health check, since it tolerates missing exact labels.

Context Precision vs Context Recall (Ragas Definitions)

Because these terms get used loosely across blog posts and tooling, pin down the exact definitions Ragas uses so your dashboards mean the same thing to everyone on the team.

Context precision asks: among the chunks retrieved, are the relevant ones ranked near the top? It's not just "is this chunk relevant," it's "is this chunk relevant *and* was it placed early in the retrieved list." A retriever that buries the one useful chunk at position nine out of ten scores worse on context precision than one that puts it first, even if both retrieved the same set of chunks. This captures the "lost in the middle" concern directly.

Context recall asks: can every claim in the reference answer be traced back to something in the retrieved context? If the reference answer states three facts and only two of them show up anywhere in the retrieved chunks, context recall for that question is roughly 0.67. This is a stricter, claim-level version of the exact chunk-ID recall from earlier.

Both scores range from 0 to 1, and neither one tells the whole story alone:

• High precision, low recall: your retriever is conservative and clean, but missing evidence. Increase top_k, or check whether your chunk size is too large and burying answers inside irrelevant surrounding text.
• Low precision, high recall: you're retrieving everything remotely related. Add a reranker, or reduce top_k and rely on better embeddings to surface the right chunk earlier.
• Low precision, low recall: the embedding model or chunking strategy is fundamentally mismatched to your corpus. Revisit chunk boundaries (semantic chunking instead of fixed-size splitting is a common fix) before touching top_k at all.

Generation-Level Metrics: Faithfulness and Answer Relevancy

Precision and recall stop at the retrieval boundary. Once the retrieved chunks reach the language model, two more failure modes can appear that retrieval metrics can't catch:

from ragas.metrics import faithfulness, answer_relevancy

results = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy],
)

Faithfulness measures whether every claim in the generated answer is actually supported by the retrieved context, catching hallucination even when retrieval was perfect. A model can be handed the exact right chunk and still invent a detail that isn't in it.

Answer relevancy measures whether the generated answer actually addresses the question asked, independent of whether it's grounded in the context. A model can produce a perfectly faithful answer to a slightly different question than the one asked, common when the retrieved context is topically adjacent but not quite on point.

Track all four numbers together, context precision, context recall, faithfulness, and answer relevancy, in the same evaluation run. A regression in any one of them tells you which layer of the pipeline to debug: retrieval config for the first two, prompt or generation model for the last two.

Building a Continuous Evaluation Loop

A one-time evaluation run tells you where you stand today. The value compounds when you wire it into CI so every change to chunking, embeddings, or prompts gets scored automatically before it ships:

# ci_eval.py
import json
import sys

THRESHOLDS = {"precision": 0.75, "recall": 0.80}

def run_gate():
    results = evaluate_retriever(my_retriever, load_eval_set(), top_k=5)
    print(json.dumps(results, indent=2))

    failed = [
        metric for metric, min_value in THRESHOLDS.items()
        if results[metric] < min_value
    ]
    if failed:
        print(f"FAILED thresholds: {failed}")
        sys.exit(1)
    print("All thresholds passed.")

if __name__ == "__main__":
    run_gate()

Wire ci_eval.py into a GitHub Actions step (or whatever your CI runner is) that runs on every pull request touching the retrieval or ingestion code. Set the thresholds slightly below your current baseline so the gate catches regressions without blocking incremental improvements. Store each run's numbers, timestamped, in a simple log or a lightweight experiment tracker, so you can plot precision and recall over time and see whether a chunking change six weeks ago quietly degraded recall that nobody noticed until now.

Refresh the eval set periodically too. A test set that never changes will eventually stop reflecting the questions users actually ask, and a retriever that's overfit to a stale eval set can look great on paper while performing poorly in production.

Common Pitfalls When Measuring RAG Precision and Recall

Using the wrong unit of comparison. Comparing retrieved document titles instead of chunk IDs will inflate both metrics artificially, since one long document can contain both relevant and irrelevant chunks, and matching at the document level hides that distinction.

Ignoring reranker position. If you retrieve 20 chunks and rerank down to 5, always compute precision and recall on the post-rerank set that actually reaches the prompt, not the pre-rerank candidate pool. The candidate pool number tells you about your embedding model; the post-rerank number tells you what the language model actually sees.

Treating recall as always more important than precision. For high-stakes domains (medical, legal, compliance) missing evidence is dangerous and recall should dominate. For a customer support chatbot with a tight context window and a cost-sensitive model, precision often matters more because every irrelevant chunk adds token cost and dilution risk with no compensating benefit.

Evaluating only on easy questions. If your eval set is all single-fact lookups, you'll never catch failures on multi-hop questions that require combining evidence from several chunks. Deliberately include a few multi-hop questions in your labeled set, since recall failures concentrate there.

Skipping the labeled ground truth step entirely and jumping straight to LLM-judge metrics. LLM judges are convenient but imperfect, and without at least a small hand-labeled set to sanity-check them against, you can't tell if the judge is systematically biased for or against your particular retriever.

FAQ

What's a good precision and recall score for a RAG system? There's no universal number. A well-tuned retriever on a clean, narrow corpus can hit precision and recall above 0.85 at top_k=5. A broad, noisy corpus with ambiguous questions might top out around 0.60-0.70 even after tuning. Track your own baseline and optimize relative to it rather than chasing an external benchmark.

Should I optimize for precision or recall first? Fix recall first. If the correct chunk is never retrieved, no amount of reranking or prompt engineering downstream can produce a correct answer. Once recall is acceptable, tighten precision with a reranker or a smaller top_k to keep the prompt clean.

Do I need Ragas, or is the manual precision/recall calculation enough? The manual calculation is enough for small, tightly labeled test sets where you trust the ground truth chunk IDs completely. Reach for Ragas or a similar LLM-judged framework when you want to evaluate against hundreds of real user questions where hand-labeling every relevant chunk isn't practical.

How often should I re-run RAG evaluation? On every pull request that touches chunking, embeddings, retrieval config, or the reranker, gated in CI. Separately, run a full evaluation against a refreshed question set on a monthly or quarterly cadence to catch drift as your corpus and user questions evolve.

Can precision and recall both be 1.0 at the same time? Yes, if the retriever returns exactly the relevant chunks and nothing else. This is achievable on narrow, well-structured corpora with clean chunk boundaries. It gets harder as the corpus grows and as questions require combining evidence from multiple, non-adjacent chunks.

Does chunk size affect precision and recall differently? Yes. Larger chunks tend to raise recall (more likely to contain the answer somewhere) but lower precision (more irrelevant text bundled in with the answer). Smaller chunks do the reverse: higher precision per chunk, but a higher chance the answer gets split across a boundary and neither half fully qualifies as relevant. Semantic chunking, which splits on topic boundaries rather than fixed token counts, usually improves both metrics simultaneously compared to naive fixed-size splitting.