Evaluating RAG with Ragas: A Tutorial

If you have shipped a retrieval-augmented generation pipeline, you already know the uncomfortable truth: it "looks fine" in a demo and then quietly hallucinates in production. This ragas tutorial walks through Ragas, an open-source library built specifically to score RAG systems on the axes that actually break in the wild: whether the answer is grounded in the retrieved context, whether the retrieved context was even relevant, and whether the final answer actually addresses the question. By the end you will have a working evaluation script, a small test dataset, and a harness you can drop into CI so a bad retriever or a regressed prompt fails a build instead of shipping.

Why RAG needs its own evaluation approach

Standard LLM evaluation (BLEU, ROUGE, exact match) was built for tasks with a single correct string. RAG systems fail differently. A RAG pipeline has two moving parts that can each break independently:

• The retriever can pull irrelevant chunks, miss the one chunk that actually answers the question, or return redundant duplicates that crowd out useful context.
• The generator can ignore the retrieved context entirely and answer from parametric memory (a hallucination that reads as confident and correct), or it can partially use the context and pad the rest with invented details.

Because these two failure modes are independent, you need metrics that separate them. Ragas does this by scoring retrieval quality and generation quality as distinct dimensions instead of collapsing everything into one "did it get the answer right" number. That separation is the whole point of this ragas tutorial: once you can see which half of the pipeline is failing, you know whether to fix your chunking strategy or your prompt.

Setting up the environment

Ragas works with LangChain, LlamaIndex, or a raw OpenAI-compatible client under the hood. Install it alongside the dependencies you already use for your RAG pipeline.

pip install ragas langchain-openai datasets pandas

Set your model provider's API key as an environment variable before running anything:

export OPENAI_API_KEY="your-key-here"

Ragas uses an LLM as a judge for most of its metrics, and an embedding model for the similarity-based ones. You can point both at any provider LangChain supports (OpenAI, Azure OpenAI, Anthropic via a LangChain wrapper, or a local model server). Here is the minimal setup:

from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper

judge_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini", temperature=0))
judge_embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings(model="text-embedding-3-small"))

Keep the judge model separate from the model powering your actual RAG pipeline. Using the same model to both generate answers and grade them introduces a self-preference bias where the judge rates its own outputs more favorably.

Building an evaluation dataset

Ragas expects a small, structured dataset for each evaluation run. Every row needs four fields:

• question: the user query
• answer: what your RAG pipeline actually generated
• contexts: the list of chunks your retriever returned for that question
• ground_truth: a reference answer written by a human (only required for some metrics)

Start with 15-30 real questions pulled from support tickets, product docs FAQs, or your own domain knowledge. Synthetic questions generated by an LLM are fine for a first pass, but real user questions expose edge cases synthetic ones miss, like ambiguous phrasing or multi-part questions.

from datasets import Dataset

eval_data = {
    "question": [
        "What is the refund window for annual subscriptions?",
        "Can I export my course progress data?",
        "Does the platform support SSO for teams?",
    ],
    "answer": [
        "Annual subscriptions can be refunded within 14 days of purchase.",
        "Yes, you can export progress as a CSV from the account settings page.",
        "SSO is available on the Team and Enterprise plans via SAML.",
    ],
    "contexts": [
        ["Refund policy: Annual plans are refundable within 14 days of the original purchase date, no questions asked."],
        ["Account settings includes a Data Export section where learners can download course progress, quiz scores, and certificates as CSV files."],
        ["Team and Enterprise plans include SAML-based single sign-on. Individual and Pro plans do not include SSO."],
    ],
    "ground_truth": [
        "Annual subscriptions are refundable within 14 days of purchase.",
        "Users can export course progress data as a CSV file from account settings.",
        "SSO via SAML is available only on Team and Enterprise plans.",
    ],
}

dataset = Dataset.from_dict(eval_data)

In a real pipeline you would populate answer and contexts by actually running your RAG system against each question, not hand-writing them. Wire that up with a loop:

def run_pipeline_for_eval(questions, retriever, generator):
    answers, contexts_list = [], []
    for q in questions:
        retrieved_chunks = retriever.retrieve(q, top_k=4)
        context_texts = [chunk.text for chunk in retrieved_chunks]
        generated_answer = generator.generate(question=q, context=context_texts)
        answers.append(generated_answer)
        contexts_list.append(context_texts)
    return answers, contexts_list

Swap retriever.retrieve and generator.generate for whatever wraps your actual vector store and LLM call. The important part is that contexts captures exactly what the generator saw, not some idealized version of it.

The core Ragas metrics

Ragas ships a set of metrics that each isolate one failure mode. These are the ones worth running on every evaluation.

Faithfulness

Faithfulness checks whether every claim in the generated answer can be traced back to the retrieved context. Ragas breaks the answer into individual statements, then asks the judge LLM whether each statement is supported by the context. A low faithfulness score means your generator is hallucinating, even when the retrieved context was good.

Answer relevancy

Answer relevancy measures how directly the answer addresses the question, independent of whether it is factually correct. Ragas generates several synthetic questions from the answer and compares their embedding similarity to the original question. A rambling answer that technically contains correct facts but never actually answers what was asked will score low here.

Context precision

Context precision looks at the retrieved chunks and checks whether the relevant ones are ranked near the top. If your retriever returns one great chunk buried under four irrelevant ones, context precision drops even though the right information was technically retrieved. This metric catches ranking problems in your retriever, not just recall problems.

Context recall

Context recall compares the retrieved contexts against the ground truth answer and checks whether all the necessary information was actually retrieved. This is the metric that tells you your retriever missed something entirely, as opposed to retrieving it in the wrong order.

Context entities recall

For domains with lots of named entities (product names, API endpoints, error codes), this metric checks whether the specific entities in the ground truth also appear in the retrieved context. It is a sharper signal than generic context recall when precision on names and terms actually matters.

Answer correctness

Answer correctness combines a semantic similarity check with a factual overlap check against the ground truth. It is the closest thing to an overall accuracy score, but should not be used alone since it can mask whether a wrong answer came from bad retrieval or bad generation.

Running the evaluation

With the dataset and metrics defined, running an evaluation is a single function call:

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
    answer_correctness,
)

result = evaluate(
    dataset=dataset,
    metrics=[
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall,
        answer_correctness,
    ],
    llm=judge_llm,
    embeddings=judge_embeddings,
)

print(result)

This returns a score per metric averaged across the dataset, plus a to_pandas() method that gives you a per-row breakdown:

df = result.to_pandas()
print(df[["question", "faithfulness", "context_recall", "answer_correctness"]])

The per-row view matters more than the average. A pipeline can average 0.85 on faithfulness while having two questions that score 0.2, and those two questions are exactly where your users are getting wrong answers. Always inspect the low-scoring rows individually before trusting the aggregate.

Diagnosing failures with the metric combination

The reason to run multiple metrics together, not just answer correctness, is that the combination tells you where to look:

• High context recall, low faithfulness: the retriever did its job, but the generator is ignoring the context and hallucinating. Fix the prompt: add an explicit instruction to answer only from the provided context, and consider lowering temperature.
• Low context recall, high faithfulness: the generator is faithfully summarizing context that does not contain the answer. Fix the retriever: your chunking strategy, embedding model, or top-k value is missing the relevant document.
• High context precision, low context recall: your retriever ranks well but does not cast a wide enough net. Try increasing top-k or adding a hybrid keyword search alongside vector search.
• Low answer relevancy, high faithfulness: the answer is accurate but does not address the actual question, often because the prompt template buries the question or the generator is answering a related but different question. Rework the prompt so the question is restated clearly before the context.

Running these metrics as a table across your test set turns "the bot gave a bad answer" into a specific, fixable engineering task.

Wiring Ragas into CI

Once you trust the metrics, gate deployments on them. A minimal CI script fails the build if any metric drops below a threshold:

import sys

THRESHOLDS = {
    "faithfulness": 0.80,
    "context_recall": 0.75,
    "answer_relevancy": 0.75,
}

result = evaluate(
    dataset=dataset,
    metrics=[faithfulness, context_recall, answer_relevancy],
    llm=judge_llm,
    embeddings=judge_embeddings,
)

failures = []
for metric_name, threshold in THRESHOLDS.items():
    score = result[metric_name]
    if score < threshold:
        failures.append(f"{metric_name}: {score:.2f} below threshold {threshold}")

if failures:
    print("RAG evaluation failed:")
    for f in failures:
        print(f"  - {f}")
    sys.exit(1)

print("RAG evaluation passed.")

Drop this into a GitHub Actions step or any CI runner that has your dataset and API keys available. Run it on every pull request that touches the retriever, the prompt template, or the chunking logic. This is the single highest-leverage habit in this ragas tutorial: catching a retrieval regression before it hits users costs a few minutes of CI time, catching it after costs a support queue full of wrong answers.

python scripts/eval_rag.py --dataset eval/questions.json --output eval/results.csv

Store the output CSV as a build artifact so you can track metric trends across commits, not just pass or fail on the latest one. A faithfulness score that drifts from 0.90 to 0.82 over three weeks without crossing your threshold is still a signal worth catching early.

Expanding the test set over time

A 20-question dataset catches obvious regressions but will not catch everything. Grow it by logging real production queries (with user consent and PII stripped) and periodically pulling the ones where users gave negative feedback, rephrased their question, or abandoned the conversation. Those are the questions your current pipeline handles worst, which makes them the highest-value additions to your Ragas dataset.

Also stratify the dataset by question type: factual lookups, multi-hop questions that need two chunks combined, and out-of-scope questions where the correct answer is "this isn't covered." RAG pipelines that only get tested on easy factual lookups look great in evaluation and then embarrass themselves the first time a user asks something the knowledge base does not cover, because nothing forced the generator to learn (or be prompted) to say "I don't know."

FAQ

What is Ragas used for? Ragas is an evaluation library for retrieval-augmented generation pipelines. It scores retrieval quality and generation quality as separate metrics (faithfulness, context precision, context recall, answer relevancy, answer correctness) so you can pinpoint whether a bad answer came from a retrieval failure or a generation failure.

Do I need ground truth answers for every Ragas metric? No. Faithfulness and answer relevancy only need the question, the generated answer, and the retrieved contexts. Context recall and answer correctness need a ground_truth reference answer, so budget time to write those for at least the metrics you plan to gate CI on.

Can I use Ragas without LangChain? Yes. LangChain wrappers are the easiest path, but Ragas also accepts raw LLM and embedding clients through its own wrapper interfaces, so you can plug in any OpenAI-compatible endpoint or a locally hosted model.

How many questions do I need in the evaluation dataset? Fifteen to thirty well-chosen questions covering your main use cases is enough for a first CI gate. Treat it as a living dataset: add real production questions over time, especially ones tied to negative user feedback, rather than trying to get the count "right" up front.

Is the judge LLM's score reliable? It is reliable enough to catch regressions and rank pipeline variants against each other, which is the main use case. Treat absolute scores as directional rather than ground truth, use a judge model separate from your generation model to avoid self-preference bias, and always spot-check the lowest-scoring rows manually before trusting an aggregate number.

What is a good faithfulness threshold to gate on? There is no universal number since it depends on your domain and risk tolerance, but many teams start around 0.80-0.85 for faithfulness and tighten it as they collect more evaluation data and understand their pipeline's baseline variance.