Ragas

Using Ragas to Compare Two RAG Pipeline Versions

Ira Menon · Jun 3, 2026 · 21 min read

Why "it feels better" is not a evaluation strategy

You changed your chunking strategy from fixed-size to semantic chunking. Or you swapped your retriever from a plain vector search to a hybrid BM25-plus-embeddings setup. Or you bumped the context window from top-3 to top-5 chunks. Now someone asks: "Is the new version actually better?"

Most teams answer this question by eyeballing five or six sample outputs and saying "yeah, looks good to me." That works until it doesn't. RAG pipelines have a lot of moving parts — the chunker, the embedding model, the retriever, the reranker, the prompt template, the generator model — and changing one of them can quietly break something else while appearing to fix the thing you were targeting. Semantic chunking might improve answer relevance while silently hurting faithfulness because the new chunks span multiple topics and give the generator room to hallucinate connections that aren't in the source text.

The eyeball test also fails in a subtler way: it's biased toward whatever you were focused on when you made the change. If you spent a week rewriting the retriever, you'll naturally read the new outputs looking for retrieval improvements, and you'll skim right past a generation-side regression because that wasn't the thing on your mind. Confirmation bias is not a hypothetical risk here — it's the default outcome of manual review, and it gets worse the more invested you are in the change being an improvement.

Ragas exists to replace "looks good to me" with numbers you can actually compare across pipeline versions. It's an open-source evaluation framework purpose-built for RAG systems, and it ships a fixed set of metrics — faithfulness, answer relevancy, context precision, context recall, and a few others — that let you score two pipeline versions against the exact same test questions and get a side-by-side comparison instead of a vibe check.

The value isn't that Ragas is some magic oracle of RAG quality — it's an LLM judging text, and LLM judges have their own blind spots. The value is *consistency*. The same judge, the same rubric, the same question set, applied twice, gives you a controlled comparison where the only thing that changed is the pipeline. That's a much stronger claim than "I read ten outputs and version two felt more confident," and it's reproducible: a teammate can rerun your script six months later and get comparable numbers, which a gut feeling can never offer.

This article walks through exactly how to set up that comparison: building a fixed evaluation dataset, running it through two pipeline versions, scoring both with Ragas, reading the resulting deltas without fooling yourself, and wiring a lightweight version of this into CI so regressions get caught automatically instead of in production.

What "comparing two versions" actually means

Before touching code, get the mental model straight, because this is where most people get sloppy.

A RAG pipeline comparison requires three things held constant and one thing changed:

The same question set. If pipeline A gets evaluated on 20 questions and pipeline B gets evaluated on a different (even if overlapping) 20 questions, any score difference could be an artifact of question difficulty, not pipeline quality.
The same ground truth / reference answers, if you're using reference-based metrics like context recall or answer correctness.
The same scoring judge model. Ragas uses an LLM as a judge for most metrics. If version A was scored with GPT-4o and version B with GPT-4o-mini, your comparison is meaningless — you're comparing judges, not pipelines.
One deliberate change between A and B. Ideally you isolate a single variable: chunk size, retriever type, prompt template, or generator model. If you change three things simultaneously, Ragas will tell you the aggregate score moved, but you won't know which change caused it.

Once those three are locked, the only thing that should vary between the two runs is the pipeline itself, which is exactly what you want to measure.

It helps to think of this as an A/B test rather than a "before and after" snapshot. In an A/B test you'd never accept "we changed the checkout flow and also happened to run a pricing promotion that week" as a clean read on conversion — you'd insist on isolating variables. RAG pipeline evaluation deserves the same discipline, even though it's tempting to skip it because you're moving fast and the change "obviously" only touches retrieval.

There's also a versioning discipline worth adopting early: name your pipeline versions descriptively (v1_fixed_chunking_top3, v2_semantic_chunking_hybrid_retriever) rather than just v1 and v2. Six months from now, when you're comparing v4 against v1 to justify a bigger architectural bet, you want the label itself to remind you what actually changed, not just an incrementing counter.

Building the fixed evaluation dataset

Your evaluation dataset is the backbone of a fair comparison. It needs four columns for a typical Ragas run: the question, the answer generated by the pipeline, the contexts retrieved for that question, and (for reference-based metrics) a ground_truth or reference answer.

Here's a minimal, reusable dataset you can build once and reuse across every pipeline version you ever test:

# eval_dataset.py
"""
Fixed evaluation set for comparing RAG pipeline versions.
Keep this file under version control — it is your benchmark.
"""

evaluation_questions = [
    {
        "question": "What is the refund window for annual subscriptions?",
        "ground_truth": "Annual subscriptions can be refunded within 30 days of purchase, minus a 5% processing fee.",
    },
    {
        "question": "Does the platform support single sign-on for enterprise plans?",
        "ground_truth": "Yes, SSO via SAML 2.0 is available on Enterprise plans and must be configured by an account admin.",
    },
    {
        "question": "What happens to my data if I downgrade from Pro to Free?",
        "ground_truth": "Data is retained for 90 days after downgrade, after which projects beyond the Free tier limit are archived.",
    },
    {
        "question": "Can I export my course progress as a certificate?",
        "ground_truth": "Certificates are generated automatically upon completing all modules and passing the final assessment with 70% or higher.",
    },
    # Aim for 30-100 questions covering your real query distribution:
    # factual lookups, multi-hop questions, and a few "should refuse" cases.
]

A few practical notes on building this set:

Pull real questions from production logs if you have them. Synthetic questions written by the team tend to be easier than what actual users ask, which inflates every score uniformly and hides regressions.
Include edge cases on purpose — questions with no good answer in your knowledge base, ambiguous phrasing, and multi-hop questions that require combining two chunks. These are exactly where pipeline changes tend to show their differences.
30 to 100 questions is a reasonable range for a fast iteration loop. Fewer than that and metric noise dominates; more than that and your feedback loop gets slow enough that you stop running it.
Ragas also has a synthetic test-set generator (TestsetGenerator) that can bootstrap questions from your documents if you're starting from zero, but treat generated questions as a supplement to real ones, not a replacement.
Tag each question with a category — factual, multi-hop, out-of-scope, ambiguous — while you're writing it. You'll want these tags later when you segment your comparison, and it's far easier to tag questions as you write them than to go back and classify fifty questions after the fact.

One question worth asking before you write a single row: who owns this file? In practice, the evaluation dataset should be treated the same way you'd treat a database migration or an API contract — reviewed, versioned, and changed deliberately. If anyone can edit eval_dataset.py on a whim, someone eventually will, right before running the comparison that makes their pipeline change look good. Put it behind the same review process as production code, and resist the urge to "just tweak one question" mid-comparison.

It's also worth deciding up front how you'll handle questions the pipeline should refuse to answer — for example, questions about topics entirely outside your knowledge base. These are some of the most valuable rows in the dataset because a pipeline that hallucinates a confident-sounding answer to an out-of-scope question is often more dangerous than one that retrieves the wrong chunk for an in-scope question. Ragas's faithfulness metric will usually catch a fabricated answer here, since there's no supporting context for it, but only if you've actually included these cases in your evaluation set.

Running both pipeline versions against the same questions

With the question set fixed, run each pipeline version end to end and capture what it retrieved and what it answered. This is the step people skip by reusing old logs — don't. Contexts and answers must come from an actual run of that specific pipeline version, not a memory of what it used to output.

# run_pipeline.py
from eval_dataset import evaluation_questions

def run_pipeline_version(pipeline, questions):
    """
    Executes a RAG pipeline over a fixed question set and
    captures the retrieved contexts alongside the final answer.
    `pipeline` must expose .retrieve(question) and .generate(question, contexts)
    """
    records = []
    for item in questions:
        question = item["question"]
        contexts = pipeline.retrieve(question)          # list[str]
        answer = pipeline.generate(question, contexts)  # str

        records.append({
            "question": question,
            "contexts": contexts,
            "answer": answer,
            "ground_truth": item["ground_truth"],
        })
    return records


if __name__ == "__main__":
    from pipeline_v1 import PipelineV1
    from pipeline_v2 import PipelineV2

    v1_results = run_pipeline_version(PipelineV1(), evaluation_questions)
    v2_results = run_pipeline_version(PipelineV2(), evaluation_questions)

    import json
    with open("v1_results.json", "w") as f:
        json.dump(v1_results, f, indent=2)
    with open("v2_results.json", "w") as f:
        json.dump(v2_results, f, indent=2)

Note that PipelineV1 and PipelineV2 here represent your actual retrieval and generation logic — for instance, v1 might be fixed-size chunking with top-3 vector retrieval, while v2 is semantic chunking with a hybrid retriever and a reranker on top. The point of this script is just to freeze the outputs to disk so you can score them deterministically without re-running the (possibly expensive, possibly non-deterministic) pipeline every time you tweak your evaluation.

Freezing the outputs to JSON is a small step that pays for itself quickly. Ragas evaluation runs make LLM calls for every metric on every row, which means scoring even a 50-question dataset across four metrics can add up to 200 judge calls. If you re-run the pipeline itself every time you want to re-score, you're paying for both the pipeline's generation calls and the judge's evaluation calls on every iteration, and you're reintroducing non-determinism from the pipeline side into what should be a stable scoring step. Separating "produce outputs" from "score outputs" also makes it trivial to re-score old results if you decide to add a new metric later — you already have the frozen contexts and answers sitting in v1_results.json.

It's worth logging retrieval latency and token usage at this stage too, even though Ragas doesn't need them. You'll want them later when you're deciding whether a quality improvement is worth its operational cost, and it's far cheaper to capture them during this run than to re-run the whole pipeline again just to measure timing.

# run_pipeline.py (extended to capture latency and token cost)
import time

def run_pipeline_version(pipeline, questions):
    records = []
    for item in questions:
        question = item["question"]

        start = time.perf_counter()
        contexts = pipeline.retrieve(question)
        retrieval_ms = (time.perf_counter() - start) * 1000

        start = time.perf_counter()
        answer, usage = pipeline.generate_with_usage(question, contexts)
        generation_ms = (time.perf_counter() - start) * 1000

        records.append({
            "question": question,
            "contexts": contexts,
            "answer": answer,
            "ground_truth": item["ground_truth"],
            "retrieval_ms": retrieval_ms,
            "generation_ms": generation_ms,
            "prompt_tokens": usage.get("prompt_tokens"),
            "completion_tokens": usage.get("completion_tokens"),
        })
    return records

Scoring both runs with Ragas

Now for the actual evaluation. Ragas expects a Hugging Face Dataset object with the right column names, and you pick the metrics you care about.

# evaluate_versions.py
import json
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)

def load_and_score(path, label):
    with open(path) as f:
        raw = json.load(f)

    dataset = Dataset.from_dict({
        "question": [r["question"] for r in raw],
        "answer": [r["answer"] for r in raw],
        "contexts": [r["contexts"] for r in raw],
        "ground_truth": [r["ground_truth"] for r in raw],
    })

    result = evaluate(
        dataset,
        metrics=[
            faithfulness,
            answer_relevancy,
            context_precision,
            context_recall,
        ],
    )

    df = result.to_pandas()
    df["pipeline_version"] = label
    return df


if __name__ == "__main__":
    v1_scores = load_and_score("v1_results.json", "v1_fixed_chunking")
    v2_scores = load_and_score("v2_results.json", "v2_semantic_chunking")

    print("=== V1 averages ===")
    print(v1_scores[["faithfulness", "answer_relevancy",
                      "context_precision", "context_recall"]].mean())

    print("\n=== V2 averages ===")
    print(v2_scores[["faithfulness", "answer_relevancy",
                      "context_precision", "context_recall"]].mean())

Two things to be careful about here:

Pin the judge model explicitly. Ragas defaults to whatever LLM you've configured (commonly through an OpenAI-compatible client). Set it once, in one place, and use the identical configuration for both scoring runs. If you change the judge model between v1 and v2 scoring, stop and rerun both.
Run scoring back to back, ideally in the same script execution, so you're not scoring v1 today and v2 next week after the judge model provider silently updated their weights.

It's also worth understanding, at least at a high level, what Ragas is doing under the hood for a metric like faithfulness, because it changes how you interpret a borderline score. Faithfulness works by first decomposing the generated answer into individual factual claims, then checking each claim against the retrieved contexts to see if it's supported. The final score is the fraction of claims that check out. This means a long, multi-part answer with one unsupported claim tucked into an otherwise well-grounded response can still score noticeably lower than a short, fully-supported answer — which is exactly the kind of nuance a five-minute manual read would miss entirely.

If your dataset is large enough that evaluation cost or time becomes a concern, Ragas supports running metrics selectively rather than the full suite on every row. It's reasonable to run the full four-metric suite on your primary comparison and drop down to just faithfulness and context recall for quick sanity checks during active iteration, saving the full run for the version you're actually deciding whether to ship.

Reading the metrics without fooling yourself

Ragas gives you a handful of core metrics, and each one tells you something different about *where* in the pipeline a regression or improvement happened. This is the actual value of doing this instead of eyeballing outputs — a single quality score can't tell you whether your retriever or your generator is responsible for a change.

Context precision measures whether the retrieved chunks that are actually relevant are ranked near the top. If this drops between versions, your retriever or reranker got worse at ordering results, even if it's still finding the right chunks somewhere in the list.
Context recall measures whether the retrieved contexts contain everything needed to answer the question, compared against the ground truth. A drop here means your retriever is missing information it used to find — a strong signal that a chunking or indexing change broke something.
Faithfulness measures whether the generated answer is actually supported by the retrieved contexts, independent of whether the answer is "good." A drop in faithfulness with stable context recall is a red flag: your retriever is fine, but your generator or prompt is now hallucinating more than before.
Answer relevancy measures whether the answer actually addresses the question asked, regardless of factual grounding. This can catch regressions where the generator got more "correct" in a narrow sense but started giving verbose, tangential, or hedge-everything answers.

The pattern to watch for is a mismatch between context metrics and generation metrics. If context precision and recall both improved but faithfulness dropped, the problem isn't retrieval — it's how your prompt or generator model is using the (better) retrieved context. That's an actionable, specific finding you'd never get from staring at five sample outputs.

Here's a concrete way this plays out. Say you swap top-3 retrieval for top-6 retrieval, hoping to catch more relevant context for multi-hop questions. Context recall goes up, as expected — you're retrieving more, so you're more likely to have everything you need. But context precision drops, because half of those extra three chunks are now irrelevant noise sitting in the prompt. If your generator model isn't great at ignoring irrelevant context, faithfulness can drop too, because the model starts blending unrelated chunks into its answer. Ragas surfaces all three of these movements distinctly, which tells you the actual fix isn't "retrieve more" or "retrieve less" — it's "retrieve more, but add a reranker to push the truly relevant chunks to the top before they hit the prompt."

Another pattern worth naming explicitly: improved metrics with an unchanged or worse user experience. It's possible for faithfulness and relevancy to both tick up while the actual usefulness of answers declines, if the change made the generator more conservative — for example, more likely to say "the provided context does not contain this information" instead of attempting a nuanced answer. That's technically faithful and technically relevant to the question, but it's a worse product experience if half your users' questions now get non-answers. This is a good argument for keeping a small qualitative review step even after you've adopted quantitative evaluation — read a sample of the actual answers for any version you're about to ship, specifically looking for this failure mode.

Putting the comparison side by side

Once you have both dataframes, build a simple delta table so the comparison is unambiguous instead of buried in two separate print statements.

# compare_results.py
import pandas as pd

def build_comparison(v1_df, v2_df, metrics):
    v1_avg = v1_df[metrics].mean()
    v2_avg = v2_df[metrics].mean()

    comparison = pd.DataFrame({
        "v1_fixed_chunking": v1_avg,
        "v2_semantic_chunking": v2_avg,
    })
    comparison["delta"] = comparison["v2_semantic_chunking"] - comparison["v1_fixed_chunking"]
    comparison["pct_change"] = (comparison["delta"] / comparison["v1_fixed_chunking"] * 100).round(2)

    return comparison


if __name__ == "__main__":
    metrics = ["faithfulness", "answer_relevancy", "context_precision", "context_recall"]
    # v1_scores, v2_scores loaded from evaluate_versions.py output
    comparison = build_comparison(v1_scores, v2_scores, metrics)
    print(comparison)

A meaningfully better pipeline version should show improvement, or at least no regression, across all four metrics — not just the one you were specifically targeting. If your goal was "improve retrieval for multi-hop questions" and context recall went up but faithfulness went down by a comparable margin, you haven't shipped an improvement, you've shipped a trade-off, and that's worth flagging explicitly before merging.

It's also worth segmenting the comparison by question type rather than only looking at the aggregate. A pipeline change can improve the average while making a specific category — say, "questions with no answer in the knowledge base" — noticeably worse. Group your evaluation questions by tag (factual, multi-hop, out-of-scope, ambiguous) and run the same delta comparison per group. Aggregate numbers hide exactly the regressions you're trying to catch.

# compare_by_category.py
def build_comparison_by_category(v1_df, v2_df, metrics, category_map):
    """
    category_map: dict of question -> category tag, e.g.
    {"What is the refund window...": "factual", ...}
    """
    v1_df = v1_df.copy()
    v2_df = v2_df.copy()
    v1_df["category"] = v1_df["question"].map(category_map)
    v2_df["category"] = v2_df["question"].map(category_map)

    v1_grouped = v1_df.groupby("category")[metrics].mean()
    v2_grouped = v2_df.groupby("category")[metrics].mean()

    delta = v2_grouped - v1_grouped
    return delta.round(3)

Running this per-category breakdown is often the difference between shipping a change confidently and shipping it with a documented caveat. A result like "average faithfulness improved 4%, but faithfulness on out-of-scope questions dropped 15%" is something a team can make an informed decision about — maybe that's an acceptable trade for the overall gain, maybe it means the release needs a follow-up fix first. Either way, that's a decision, not a surprise discovered by a user two weeks later.

How many questions do you actually need

A fair question at this point is whether 30 questions is enough to trust a delta of, say, 0.03 in context recall. It's not a rigorous statistical test, but a quick sanity check goes a long way: look at the per-question score distribution, not just the mean, before declaring a winner.

# check_significance.py
import numpy as np

def summarize_distribution(df, metric):
    values = df[metric].dropna().to_numpy()
    return {
        "mean": round(values.mean(), 3),
        "std": round(values.std(), 3),
        "min": round(values.min(), 3),
        "max": round(values.max(), 3),
        "n": len(values),
    }

def paired_delta_summary(v1_df, v2_df, metric):
    # Assumes both dataframes are in the same question order
    deltas = v2_df[metric].to_numpy() - v1_df[metric].to_numpy()
    return {
        "mean_delta": round(deltas.mean(), 3),
        "pct_questions_improved": round((deltas > 0).mean() * 100, 1),
        "pct_questions_regressed": round((deltas < 0).mean() * 100, 1),
    }

If a metric's average improved but pct_questions_regressed shows that 40% of individual questions actually got worse, the aggregate is masking a real trade-off — a handful of big wins are outweighing a broad, shallow regression. That's a very different situation from a version that improved almost every question by a small amount, even though both could produce the same average delta. As a rule of thumb, treat any comparison built on fewer than 30 questions as directional rather than conclusive, and be especially cautious about small deltas (under roughly 0.03–0.05 on a 0-1 scale) on datasets under 50 questions — that's well within the range of noise from LLM-judge variance alone.

Common pitfalls that invalidate the comparison

A handful of mistakes will quietly poison an otherwise well-set-up comparison:

Non-deterministic generation. If your generator uses a high temperature, the same pipeline version can produce different faithfulness scores on different runs. Set temperature to 0 (or as close as your model allows) for evaluation runs specifically, even if production uses a higher temperature for variety.
Judge model drift. LLM-as-judge scores are not perfectly stable across provider-side model updates. If you're evaluating v1 today and v2 three weeks from now, don't assume the judge behaves identically. Re-score v1 alongside v2 if there's been any gap in time.
Silent dataset drift. If your knowledge base itself changed between when you tested v1 and when you tested v2 — new documents added, old ones removed — you're no longer comparing pipelines, you're comparing pipelines against different underlying data. Snapshot your knowledge base or run both versions against the same corpus checkpoint.
Cherry-picked question sets. It's tempting to evaluate only on the questions you know are hard for the retriever, since those are the interesting ones. But if you designed v2 to fix exactly those questions, of course it will win — build your set before you know what the fix will be, and keep it fixed afterward.
Ignoring cost and latency. A version that scores two points higher on faithfulness but triples your token spend per query, or adds two seconds of reranking latency, might not be a net win for your product even if the Ragas numbers say it's "better." Track cost and latency alongside the quality metrics in the same comparison table.

A lightweight CI check for regressions

Once you trust this workflow, it's worth wiring a simplified version into CI so a pipeline change can't merge if it silently tanks faithfulness or recall. This doesn't need to be the full comparison — a threshold gate against your last known-good scores is enough to catch obvious regressions automatically.

# ci_regression_gate.py
import sys
import json

# Minimum acceptable score for each metric, based on your last
# accepted pipeline version. Update these deliberately, not automatically.
THRESHOLDS = {
    "faithfulness": 0.85,
    "answer_relevancy": 0.80,
    "context_precision": 0.75,
    "context_recall": 0.75,
}

def check_regression(scores_path):
    with open(scores_path) as f:
        scores = json.load(f)  # dict of metric -> average score

    failures = []
    for metric, minimum in THRESHOLDS.items():
        actual = scores.get(metric)
        if actual is None:
            failures.append(f"{metric}: missing from results")
        elif actual < minimum:
            failures.append(f"{metric}: {actual:.3f} below threshold {minimum}")

    if failures:
        print("RAG evaluation gate FAILED:")
        for f in failures:
            print(f"  - {f}")
        sys.exit(1)

    print("RAG evaluation gate passed.")


if __name__ == "__main__":
    check_regression(sys.argv[1])

This kind of gate won't replace the full side-by-side comparison when you're deliberately iterating on the pipeline, but it's a cheap safety net that catches accidental regressions — a dependency bump that changes your embedding model's default behavior, a prompt template edit that quietly loosens grounding — before they reach production.

Wrapping up

Comparing two RAG pipeline versions is fundamentally a controlled experiment: fix the question set, fix the judge, fix the knowledge base snapshot, and change exactly one thing. Ragas gives you the metrics that turn "this feels better" into "context recall went from 0.71 to 0.83 and faithfulness held steady at 0.89," which is the kind of statement you can actually defend in a design review or a postmortem.

The workflow in this article — a versioned evaluation dataset, frozen pipeline outputs, Ragas scoring, a delta table, and a CI threshold gate — scales from a solo project to a team shipping RAG changes weekly. Start with even 20-30 real questions and four core metrics; you can always widen the question set and add metrics like answer correctness or harmfulness checks once the basic loop is running.

If you want to go deeper into building this evaluation muscle — writing custom Ragas metrics, generating synthetic test sets at scale, and integrating evaluation gates directly into your deployment pipeline — check out the Ragas Tutorial course on teachyou.ai, where we build this exact comparison workflow from scratch on a real RAG project.