RagasRAGEvaluation

Ragas Tutorial: Evaluating RAG Pipelines in 30 Minutes

Pramod Dutta · May 29, 2026 · 15 min read

You shipped a RAG pipeline last week. The demo looked great, the boss nodded, and now you're three weeks in with no idea whether last night's prompt tweak made answers better or quietly broke retrieval for an entire category of questions. This is the point where most teams start "vibe checking" outputs by eye, which works until it doesn't — usually right before a customer complains that the bot invented a refund policy. Ragas exists to close that gap. It gives you numeric, repeatable scores for the four things that actually break in production RAG systems: retrieval quality, groundedness, relevance, and factual recall. This tutorial gets you from pip install to a working evaluation report in about 30 minutes, using a small dataset you can build from your own app today.

What Ragas Actually Measures

Before writing code, it helps to know what you're buying into. A RAG pipeline has two failure surfaces: the retriever (did we fetch the right chunks?) and the generator (did the LLM use those chunks correctly?). Ragas scores both, using an LLM-as-judge approach combined with some statistical heuristics, so you don't need human annotators for every run.

The four core metrics you'll use constantly are:

Faithfulness — does the generated answer only contain claims that are supported by the retrieved contexts? This catches hallucination.
Answer relevancy — does the answer actually address the question, or does it wander off-topic while still being technically "faithful"?
Context precision — of the chunks you retrieved, how many were actually useful/relevant to answering the question? This flags a noisy retriever.
Context recall — did your retriever pull back all the information needed to construct the ground truth answer? This flags a retriever that's missing things entirely.

Each metric returns a score between 0 and 1. None of these numbers are magic — they're LLM judgments, so they carry the same variance and quirks as any other LLM call. Treat them as a strong directional signal for regression testing, not a courtroom-grade certificate of correctness.

It's worth understanding *how* Ragas computes these under the hood, because it changes how much you trust a given score. Faithfulness works by asking the judge LLM to break the generated answer into a list of discrete factual claims, then checking each claim against the retrieved contexts individually — the final score is the fraction of claims that are supported. Answer relevancy works almost in reverse: it asks the judge to generate several plausible questions that the given answer would satisfy, embeds those synthetic questions, and compares them against the embedding of the original question. A high score means the answer's content maps tightly back to what was actually asked. Context precision and context recall both operate at the level of individual retrieved chunks — precision asks "was this specific chunk useful for the answer," recall asks "does the ground truth's content appear somewhere across the retrieved chunks." Knowing this decomposition matters later, when you're debugging a surprising score and need to know whether the judge misjudged one claim out of ten, or whether the whole answer collapsed.

Installing Ragas and Setting Up Your Environment

Ragas is a Python library built on top of LangChain-style abstractions, and it needs an LLM and an embedding model to act as the judge. Start with a clean virtual environment.

python -m venv ragas-env
source ragas-env/bin/activate

pip install ragas datasets langchain-openai python-dotenv

You'll also need an API key for whichever model you want acting as the evaluator judge. Most teams start with an OpenAI model for the judge since Ragas's defaults are tuned around it, but you can swap in any chat model that supports function calling or structured output, including self-hosted ones, once you're comfortable with the basics.

export OPENAI_API_KEY="sk-..."

Do a quick sanity import to confirm the install worked before you build anything on top of it.

import ragas
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)

print(ragas.__version__)

If that import fails, it's almost always a dependency mismatch between ragas, datasets, and your LangChain packages. Pin versions in a requirements.txt early — Ragas moves fast and breaking changes between minor versions are common.

Building a Small Evaluation Dataset

Ragas doesn't need thousands of examples to be useful. Ten to twenty well-chosen question/answer pairs that represent your real user traffic will tell you more than a thousand random ones. Each row in your evaluation dataset needs four fields:

question — the actual user query
contexts — the list of text chunks your retriever returned for that query
answer — what your RAG pipeline actually generated
ground_truth — the correct answer, written by a human who knows the source material

The ground_truth field is the one teams skip, and it's the one that unlocks context recall. Without it, you can only measure faithfulness and relevancy — useful, but you're flying blind on whether your retriever is missing information entirely. Budget an hour to write 15-20 solid ground truth answers pulled directly from your knowledge base. It pays for itself the first time a regression shows up.

Here's what a realistic dataset construction step looks like, pulling contexts from your actual retriever rather than hand-waving them:

from datasets import Dataset

# Assume `my_rag_pipeline` exposes retrieve() and generate()
# from your actual application code.
questions = [
    "What is the refund window for annual subscriptions?",
    "Can I switch from monthly to annual billing mid-cycle?",
    "Does the API rate limit reset at midnight UTC or account creation time?",
]

ground_truths = [
    "Annual subscriptions can be refunded in full within 14 days of purchase.",
    "Yes, switching to annual billing mid-cycle prorates the remaining monthly balance.",
    "The API rate limit resets at midnight UTC, regardless of account creation time.",
]

records = []
for question, ground_truth in zip(questions, ground_truths):
    retrieved_chunks = my_rag_pipeline.retrieve(question, top_k=4)
    generated_answer = my_rag_pipeline.generate(question, retrieved_chunks)

    records.append({
        "question": question,
        "contexts": [chunk.text for chunk in retrieved_chunks],
        "answer": generated_answer,
        "ground_truth": ground_truth,
    })

eval_dataset = Dataset.from_list(records)
print(eval_dataset)

Notice that contexts is a list of strings, not a single blob. Ragas needs the chunk boundaries intact because context precision scores each retrieved chunk individually before aggregating. If you flatten your contexts into one string before this step, you lose the ability to diagnose which specific chunks were noise.

Running the Core Metrics

With the dataset built, running the evaluation is a single function call. This is also where most of Ragas's API surface lives — the evaluate() function takes your dataset and a list of metric objects, runs the judge LLM against every row for every metric, and returns a results object you can convert straight to a pandas DataFrame.

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

judge_llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
judge_embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

results = evaluate(
    dataset=eval_dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    llm=judge_llm,
    embeddings=judge_embeddings,
)

df = results.to_pandas()
print(df[["question", "faithfulness", "answer_relevancy", "context_precision", "context_recall"]])

Run this and you'll get a per-row breakdown alongside an aggregate score for each metric. Expect this to take a few minutes for even a small dataset — each metric involves at least one, and sometimes several, LLM calls per row, since Ragas often decomposes an answer into individual claims before checking each one against the retrieved contexts.

A practical note: set temperature=0 on your judge model. Evaluation metrics that fluctuate between runs because of judge randomness will make you distrust the whole exercise, and you want stable numbers you can diff across pipeline versions.

Reading the Scores Without Fooling Yourself

A single run of numbers is close to useless in isolation. A faithfulness score of 0.82 doesn't tell you if that's good — it tells you something once you compare it against last week's 0.91 after you changed your system prompt. Ragas scores are fundamentally comparative, not absolute. Build the habit of running the same eval suite before and after every meaningful pipeline change, and store the results (a CSV per commit works fine) so you have a timeline, not just a snapshot.

That said, some rough calibration helps when you're staring at numbers for the first time:

Faithfulness below 0.7 usually means your model is embellishing or inferring beyond what's in the context — worth investigating row by row.
Answer relevancy below 0.7 often means the answer is technically true but doesn't address what was asked, which is common when retrieval pulls adjacent-but-wrong topics.
Context precision below 0.6 means your retriever is pulling in a lot of noise alongside the useful chunk — the generator is doing extra work filtering signal from garbage.
Context recall below 0.7 means the ground truth information genuinely isn't in what you retrieved, which is a much more serious retrieval problem than precision issues.

Don't chase a perfect 1.0 across the board. Real production RAG systems that are performing well typically sit in the 0.75-0.9 range on most metrics, with occasional dips on genuinely hard multi-hop questions. If every score is a suspiciously flat 0.95+, check whether your ground truths are too similar to your retrieved contexts — that's a sign your eval set doesn't actually stress the system.

Diagnosing and Fixing Weak Metrics

This is where Ragas earns its keep — the four metrics map to distinct fixes, so a low score tells you where to spend your engineering time instead of guessing.

Low context recall, healthy faithfulness. Your generator is doing fine with what it's given, but the retriever isn't finding the right chunks in the first place. Start by checking your chunking strategy — chunks that are too large dilute the embedding signal, chunks that are too small lose surrounding context needed for a full answer. Also check top_k: bumping it from 3 to 6 is a cheap first experiment. If recall is still weak after that, consider hybrid search (BM25 plus dense retrieval) since pure vector search often misses exact-match terms like product names, error codes, or SKUs.

Low context precision, healthy recall. You're finding the right information, but it's buried in a pile of irrelevant chunks. This is a strong signal to add a reranking step — a cross-encoder reranker after your initial vector search retrieval pass will usually give you the single biggest precision jump for the least engineering effort. It's also worth checking whether your top_k is simply too high, retrieving chunks past the point of usefulness.

Low faithfulness, healthy context precision/recall. The retriever handed the generator good material, but the generator ignored it or embellished. This is a prompt problem, not a retrieval problem. Add explicit grounding instructions — something like "only answer using the provided context, and say 'I don't know' if the context doesn't contain the answer" — and consider lowering generation temperature. If faithfulness stays low even with a tight grounding prompt, check whether your context window is being truncated silently, which forces the model to fill gaps from its training data.

Low answer relevancy. This one is sneaky because faithfulness can be high at the same time — the model can accurately restate context that doesn't actually answer the question. Usually this traces back to the question being ambiguous, or your prompt not explicitly instructing the model to directly answer the user's question before adding supporting detail. Tightening the answer format in your prompt template ("Start with a direct one-sentence answer, then explain") fixes this more often than people expect.

The pattern to internalize: retrieval metrics point you at your retriever, generation metrics point you at your prompt. Don't reflexively reach for a bigger model when context recall is the problem — a bigger model reading the wrong chunks still gives you the wrong answer.

One more failure mode worth naming explicitly: all four metrics look fine, but users are still unhappy. This usually means your eval dataset doesn't actually represent the queries causing pain — teams often write eval questions that mirror their documentation's structure rather than the messy, underspecified way real users actually ask things. If your scores look great but support tickets disagree, the fix isn't a new metric, it's a better dataset. Go pull twenty real queries from your logs, including the ones that got a shrug or a follow-up "that's not what I meant" from the user, and add them to your eval set before you trust the numbers again.

Iterating: Turning Ragas Into a Regression Suite

A one-off eval run is a snapshot. The real value shows up when you wire Ragas into your development loop so every pipeline change gets scored automatically. A minimal version of this just means running your eval script on every pull request that touches retrieval or prompt code, and diffing the aggregate scores against a stored baseline.

import json
from pathlib import Path

BASELINE_PATH = Path("eval_baseline.json")

current_scores = {
    "faithfulness": float(df["faithfulness"].mean()),
    "answer_relevancy": float(df["answer_relevancy"].mean()),
    "context_precision": float(df["context_precision"].mean()),
    "context_recall": float(df["context_recall"].mean()),
}

if BASELINE_PATH.exists():
    baseline_scores = json.loads(BASELINE_PATH.read_text())
    print("Metric            Baseline   Current    Delta")
    for metric, current_value in current_scores.items():
        baseline_value = baseline_scores.get(metric, 0.0)
        delta = current_value - baseline_value
        flag = "REGRESSION" if delta < -0.03 else ""
        print(f"{metric:<17} {baseline_value:.3f}      {current_value:.3f}     {delta:+.3f}  {flag}")
else:
    print("No baseline found — writing current scores as the new baseline.")

BASELINE_PATH.write_text(json.dumps(current_scores, indent=2))

A drop of more than a few points on any single metric is worth blocking a merge over, the same way you'd block on a failing unit test. Set the threshold based on your dataset size — with only 15-20 examples, expect some run-to-run noise, so don't panic over a 0.01 wobble. As your eval set grows past 50-100 examples, tighten the threshold since the aggregate becomes more stable.

Keep your eval dataset alive as a living artifact. Every time a real user hits an edge case your pipeline handled badly, add that question and its correct ground truth answer to the dataset. Within a couple of months you'll have a evaluation suite that actually reflects your production traffic instead of a handful of happy-path examples someone wrote in an afternoon.

Common Pitfalls When Running Ragas in Production

A few things trip people up consistently enough to call out directly.

Judge model cost adds up fast. Faithfulness alone can issue several LLM calls per row since it decomposes answers into individual factual claims. Running a 200-row eval suite on every commit with a frontier model gets expensive quickly — use a cheaper judge model for day-to-day iteration and save the more expensive one for release gates.
Async execution matters at scale. Ragas supports running metric computation concurrently, which is the difference between a ten-minute eval run and an eighty-minute one once your dataset grows past a hundred rows. If your evaluation script is looping row by row synchronously, check whether you're leaving Ragas's built-in batching and async execution on the table — it's usually a couple of configuration lines, not a rewrite.
A single eval run is a sample, not a certainty. Because the judge LLM has some inherent variance even at temperature zero (sampling isn't perfectly deterministic across API calls, and prompt-internal randomness from the judge's own reasoning can shift borderline cases), don't treat one run as gospel. For a release-gating decision, run the suite twice and look at whether the delta between versions survives that noise, not just whether it's nonzero.
Contexts must match what production actually retrieves. It's tempting to hand-pick "good" contexts when building your eval set. Don't. Always pull contexts live from your actual retriever so the eval reflects reality, including its flaws.
Ground truth quality determines context recall quality. A vague or overly broad ground truth answer will make context recall look worse than it is, because the judge can't tell if the retrieved chunks "cover" something fuzzy. Write ground truths as tightly and specifically as you'd want the RAG system's own answer to be.
Metric scores don't compose linearly across question types. Factual lookup questions and multi-hop reasoning questions behave very differently under these metrics. If your dataset mixes both, segment your results by question type before drawing conclusions — an aggregate score can hide a category that's failing badly while another category is carrying the average.
Don't skip re-running the baseline after upgrading the judge model itself. Ragas scores are only comparable when the judge is held constant. Swapping your evaluator LLM invalidates your historical baseline — treat it like changing your test framework, not like changing your test data.

Where to Go From Here

Thirty minutes gets you a working eval loop, but Ragas has more in the toolbox once you're comfortable with these four metrics — synthetic test-set generation from your own documents, custom metrics for domain-specific correctness checks, and integration with tracing tools so you can click from a low score straight into the exact retrieval call that caused it. None of that matters if the foundation isn't solid, though. Get faithfulness, answer relevancy, context precision, and context recall running reliably on a real dataset first, wire it into your regression workflow, and you'll catch the silent regressions that used to only surface as angry support tickets.

If you want the deeper version of everything covered here — building the retriever from scratch, chunking strategy, hybrid search, reranking, and a full evaluation harness built alongside a real production-style RAG system — that's exactly what we cover step by step inside Introduction to RAG, the course Ira Menon and I built for engineers who want to ship RAG systems that hold up under real traffic, not just demo day.