Ragas vs DeepEval: Choosing Your RAG Evaluation Framework

The Monday morning the support bot started hallucinating

A three-person startup ships a RAG-based support bot on a Friday. It answers billing questions, pulls refund policy details from a knowledge base, and cites the right doc chunks in testing. By Monday, a customer asks about a pricing tier that was deprecated six months ago, and the bot confidently answers using stale retrieved context — because nobody built the retraining pipeline for the vector store yet. The founder pastes the exchange into Slack with one line: "how do we even measure if this happens again?"

This is the moment every team building on top of a retriever plus an LLM eventually hits. You have a working pipeline. You do not have a way to know, systematically, whether a change to your chunking strategy, embedding model, or prompt template made things better or worse. You need an eval framework, and you need to pick one this week, not after a three-month bake-off. The two names that come up immediately are Ragas and DeepEval. Both will get you unstuck. They are not the same tool wearing different logos, and the choice you make now will shape how much rework you do in six months. This article walks through that decision the way the support-bot team actually made it — starting narrow, hitting a wall, and reconsidering.

What Ragas actually is: a RAG-shaped ruler

Ragas (Retrieval Augmented Generation Assessment) was built for exactly one job: scoring how well a retrieval-augmented pipeline is doing, using metrics that map directly onto the RAG architecture itself. It does not try to be a general test runner. It assumes you have a question, a retrieved context, a generated answer, and often a ground-truth reference, and it gives you numbers for each stage of that pipeline.

The headline metrics are worth knowing by name because they map onto real failure modes:

• Faithfulness — does the generated answer only contain claims that can be traced back to the retrieved context, or is the model adding things it wasn't given
• Answer relevancy — does the answer actually address the question asked, independent of whether it's factually grounded
• Context precision — of the chunks retrieved, how many were actually relevant and ranked sensibly
• Context recall — did retrieval pull in everything needed to answer correctly, or did it miss a chunk that mattered

That's the appeal for a team in the support-bot's position: within an afternoon, you can run these four metrics against a set of real support transcripts and get a diagnosis. Low context recall with high faithfulness tells you retrieval is the problem, not the LLM. High context precision with low answer relevancy tells you the model is ignoring good context. That's an actionable split you cannot get from "the bot gave a wrong answer" alone.

There's a second, quieter reason Ragas fits this exact moment: it doesn't need you to have a mature test-authoring habit yet. Most early-stage teams don't have a labeled golden dataset, a CI pipeline, or an agreed-upon definition of "correct" beyond "does this look right to a human." Ragas meets you where you are — you can even run several of its metrics, like faithfulness and answer relevancy, without a ground-truth reference at all, because they compare the answer against the retrieved context rather than against a hand-written correct answer. Context recall does need a reference, but you can start with just faithfulness and answer relevancy on raw production logs, get a signal by lunchtime, and add the reference-dependent metrics once someone has time to label thirty or forty examples properly. That staged onboarding is part of why teams reach for it first — the tool's requirements grow with your dataset, not ahead of it.

Here's roughly what that first pass looks like in code.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

data = {
    "question": ["What's the refund window for annual plans?"],
    "answer": ["Annual plans can be refunded within 30 days of purchase."],
    "contexts": [["Refund policy: annual subscriptions are refundable within 30 days.",
                  "Monthly subscriptions are non-refundable after activation."]],
    "ground_truth": ["Annual plans have a 30-day refund window."],
}

dataset = Dataset.from_dict(data)

result = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)

print(result)

That's the whole surface area for a first pass. No test classes, no fixtures, no assertions to write by hand — you hand it a dataset shaped like your pipeline's inputs and outputs, and it hands back scores. For a team that needs a diagnosis by end of day, that speed to first insight is the entire pitch.

What DeepEval actually is: pytest for LLM behavior

DeepEval starts from a different premise: your RAG bot is one component in a system that also has prompt templates, safety constraints, tone requirements, and business logic, and all of it needs testing the same way your API endpoints do. DeepEval is built as a pytest plugin at its core. If your team already runs pytest in CI, DeepEval slots into that exact workflow instead of asking you to build a parallel evaluation pipeline.

DeepEval ships RAG-specific metrics too — it has its own versions of faithfulness, contextual precision, and contextual recall, so it is not weaker on the RAG axis than Ragas. What it adds is everything outside RAG: hallucination checks that don't require a retrieval step, toxicity and bias scoring, custom criteria via its GEval metric (which lets you define a rubric in plain English and have an LLM judge grade against it), conversational metrics for multi-turn flows, and red-teaming style adversarial tests. The support bot's team, once they're past "is retrieval working," starts asking questions like "does the bot ever leak an internal Slack thread it shouldn't have access to" or "does it stay polite when a customer is furious" — those are DeepEval's home turf, not Ragas's.

The structural difference shows immediately in how you write a test.

import pytest
from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric, GEval
from deepeval.test_case import LLMTestCase

def test_refund_answer_is_grounded():
    test_case = LLMTestCase(
        input="What's the refund window for annual plans?",
        actual_output="Annual plans can be refunded within 30 days of purchase.",
        retrieval_context=[
            "Refund policy: annual subscriptions are refundable within 30 days.",
            "Monthly subscriptions are non-refundable after activation.",
        ],
    )
    faithfulness = FaithfulnessMetric(threshold=0.8)
    assert_test(test_case, [faithfulness])

def test_tone_stays_professional_under_frustration():
    test_case = LLMTestCase(
        input="This is the third time your bot has given me the wrong answer!!",
        actual_output="I'm sorry for the frustration — let me get this sorted correctly this time.",
    )
    tone_check = GEval(
        name="Professional Tone",
        criteria="Response should acknowledge frustration without being defensive or dismissive.",
        evaluation_params=["input", "actual_output"],
    )
    assert_test(test_case, [tone_check])

Run it with pytest test_support_bot.py, and it behaves like any other test suite: pass/fail exit codes, JUnit XML output, the works. That's the second half of the pitch — it's not a separate reporting tool you check manually, it's a gate your CI pipeline already understands.

There's a philosophical difference underneath the tooling difference, and it's worth naming directly. Ragas treats evaluation as measurement — you're generating a score, and what you do with that score is up to you. DeepEval treats evaluation as testing — you're asserting a claim about behavior, and the framework's job is to tell you whether the claim holds, in a language your existing engineering process already speaks. Neither framing is more correct, but they lead to different habits. Teams using Ragas tend to eyeball dashboards and trends over time. Teams using DeepEval tend to write a new test the moment they find a bug, the same reflex they'd have for a backend regression. Once the support bot's team started thinking of prompt changes the way they thought of code changes — something that needs a red/green signal before merging — DeepEval's framing started to matter more than its specific metrics.

The support bot's growth-stage decision

Here's how the actual decision unfolded for a hypothetical but representative team — three engineers, a RAG bot answering billing and account questions, shipping fast and evaluating as they go.

Week one: proving the pipeline is worth shipping. The team has forty sample question-answer pairs pulled from real support tickets. They need one number: is this bot's retrieval good enough to demo to the founder. Ragas wins here without much debate. It's a script, not a project — pip install ragas, format the dataset, run evaluate(), get faithfulness and context recall back in minutes. There's no CI to integrate with yet because there's barely a codebase, just a notebook and a prompt. Trying to write pytest fixtures for this stage would be solving a problem they don't have.

Month two: the bot is live, and regressions start creeping in. Someone tweaks the system prompt to make answers shorter, and support tickets tagged "bot gave incomplete answer" tick up. Someone else swaps the embedding model to cut vector-store costs, and nobody notices context recall dropped until a customer complains publicly. This is the point where Ragas alone stops being enough — not because its metrics got worse, but because there's no mechanism forcing anyone to check the metrics before merging. The team adds a DeepEval-based pytest suite that runs on every pull request touching the prompt templates or retrieval config. Now a prompt change that tanks faithfulness fails CI the same way a broken unit test would, and it's caught before it reaches production, not after a customer complains.

Month five: the bot has grown past pure RAG. It now has a triage step (is this a billing question or a technical one), a tone layer (stay calm with frustrated customers), and a hard rule (never quote internal pricing negotiation notes even if they're accidentally in the retrieval index). None of that is expressible as faithfulness or context recall — it's exactly the custom-criteria and safety territory DeepEval was built for. The team's test suite now has three folders: test_retrieval.py (RAG metrics), test_tone.py (GEval rubrics), test_safety.py (PII and leakage checks). Ragas's role hasn't disappeared, but it's now feeding into a much bigger net.

That arc — fast diagnosis, then CI gate, then full-system safety net — is the shape of the decision more often than a static "pick one forever" choice.

It's worth being honest about what didn't happen along the way, too. The team didn't throw out their week-one Ragas scripts when they adopted DeepEval in month two — those scripts kept running as a separate nightly job against production logs, feeding a simple trend chart the team checked once a week. The CI-gated DeepEval suite and the nightly Ragas trend job answered different questions: CI asked "did this specific change break something," and the nightly job asked "is overall quality drifting even without a code change," which matters because retrieval quality can degrade silently as the underlying document set grows or goes stale, with no pull request to pin the blame on. Keeping both running, rather than replacing one with the other, is what actually happened — not a clean migration, but an accumulation of tools each answering a question the other one couldn't.

Cost and complexity of running each in production

Neither tool is free to operate, and the cost isn't really about licensing — both are open source — it's about LLM calls and engineering time.

Ragas costs scale with how many metrics you compute per example. Faithfulness and answer relevancy both require an LLM call (often more than one, since faithfulness decomposes the answer into individual claims and checks each one against the context). Running the full four-metric suite against a few hundred examples nightly is a few hundred to a couple thousand LLM calls, depending on decomposition granularity — noticeable but manageable on a smaller/cheaper judge model. The complexity cost is low: it's a script or a scheduled job, not new infrastructure.

DeepEval costs scale similarly per metric, but the complexity cost is different in kind. Because it's a full pytest suite, the "cost" also includes engineering time to write and maintain test cases as the product changes — the same tax any test suite carries. Where Ragas complexity is mostly "how many metrics do I compute," DeepEval complexity is "how many test cases do I maintain, and do they stay meaningful as behavior changes." Teams that let their DeepEval suite sprawl without pruning stale test cases end up with the same problem as any bloated test suite: slow CI, and tests nobody trusts enough to act on when they fail.

The practical implication for a startup: budget LLM-judge calls as a real line item once either tool runs on every PR, and pick a cheap-but-decent model for the judge (not necessarily the same model powering the bot) unless you specifically need judge quality to match production quality.

There's a second cost that's easy to underweight: latency in CI. A PR that triggers forty DeepEval test cases, each making one or two judge-model calls, adds real wall-clock time to every merge if those calls run sequentially. The support bot's team hit this directly — their CI run went from under a minute to nearly six once the behavioral test suite grew past thirty cases, and engineers started merging without waiting for it, which defeats the entire point of having a gate. The fix was mundane: batch the judge calls concurrently instead of one test at a time, and split the suite so only tests touching changed files run on every PR, with the full suite reserved for a nightly run. Neither Ragas nor DeepEval solves this for you automatically — it's an operational decision you make once the suite is big enough to notice.

Can you run both together

Yes, and by month five the support bot's team was effectively doing exactly this without labeling it a strategy — Ragas metrics were being computed inside a DeepEval-orchestrated suite. This isn't a hack; it's a reasonably natural seam, because Ragas metrics can be computed as plain Python functions and their scores fed into any assertion framework, including DeepEval's.

The pattern: use Ragas for what it's precisely tuned for — retrieval-stage diagnostics — and let those scores become inputs to a broader DeepEval-run pytest suite that also covers tone, safety, and non-RAG behavior. You get Ragas's speed and RAG-specific nuance without giving up DeepEval's CI-native reporting and broader coverage.

import pytest
from ragas import evaluate
from ragas.metrics import faithfulness, context_recall
from datasets import Dataset
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase

def compute_ragas_scores(question, answer, contexts, ground_truth):
    dataset = Dataset.from_dict({
        "question": [question],
        "answer": [answer],
        "contexts": [contexts],
        "ground_truth": [ground_truth],
    })
    result = evaluate(dataset, metrics=[faithfulness, context_recall])
    return result

def test_billing_answer_end_to_end():
    question = "What's the refund window for annual plans?"
    answer = "Annual plans can be refunded within 30 days of purchase."
    contexts = ["Refund policy: annual subscriptions are refundable within 30 days."]
    ground_truth = "Annual plans have a 30-day refund window."

    # Ragas handles the RAG-specific diagnosis
    ragas_scores = compute_ragas_scores(question, answer, contexts, ground_truth)
    assert ragas_scores["faithfulness"][0] >= 0.8
    assert ragas_scores["context_recall"][0] >= 0.8

    # DeepEval handles the broader behavioral check in the same CI run
    test_case = LLMTestCase(input=question, actual_output=answer, retrieval_context=contexts)
    tone_check = GEval(
        name="Concise and Direct",
        criteria="Answer should be a single direct sentence, no hedging or filler.",
        evaluation_params=["input", "actual_output"],
    )
    assert_test(test_case, [tone_check])

Run that file with pytest, and both frameworks report through the same CI gate, with Ragas doing the retrieval math it's specifically good at and DeepEval doing the orchestration and everything-else checks. This is the setup that scales past the "is retrieval fine" question without requiring a rewrite when you outgrow Ragas alone.

Where each one breaks down

Ragas gets awkward the moment your system has meaningful non-RAG behavior to test — there's no clean way to express "don't leak this string" or "match this tone" in its metric set, because that was never its job. Teams that try to force everything through Ragas end up writing custom scoring functions that duplicate what DeepEval already ships, which is wasted effort.

DeepEval, on the other hand, has more moving parts to set up if all you need on day one is a retrieval sanity check. Standing up a pytest suite, deciding on thresholds, wiring CI — that's real setup time you don't need when you just want to know if last night's chunking change helped or hurt. Teams that reach for DeepEval on day one sometimes spend the first week configuring test infrastructure instead of looking at their actual retrieval numbers.

Both tools also share a failure mode worth naming honestly: LLM-judge metrics are not perfectly deterministic, and a metric hovering right at your threshold will occasionally flip pass/fail between runs on the same input. Set thresholds with margin, not exactness, and don't treat a single borderline score as gospel — look at trend across a batch of examples instead of one.

Reading the scores without fooling yourself

Whichever tool produces the number, the number itself is easy to misread, and it's worth flagging the specific traps the support bot's team fell into before they trusted their own dashboards.

The first trap was treating a single aggregate score as the whole story. A faithfulness score of 0.85 averaged across two hundred examples can hide a bimodal reality — a hundred and eighty examples scoring near-perfect and twenty scoring near-zero, versus two hundred examples all scoring a mediocre 0.85. Those are completely different engineering problems: the first is "fix the twenty specific failures," the second is "your whole system has a consistent grounding weakness." Always look at the distribution, not just the mean, and both Ragas and DeepEval return per-example scores that make this easy to check if you bother to plot them.

The second trap was picking thresholds before having a baseline. The team's first instinct was to require faithfulness above 0.9 for every CI run, borrowed from a blog post rather than from their own data. That threshold failed almost every PR, including ones that made things better, because their judge model's natural scoring range for their specific domain (billing and account text, which is dense with numbers and dates) sat closer to 0.75. The fix was running the metric against a stable, unchanged version of the bot first, recording where the score naturally landed, and setting the CI threshold slightly below that baseline — so the gate catches regressions relative to your own system, not relative to an arbitrary number from someone else's use case.

The third trap was forgetting that the judge model itself can drift. If your CI pipeline pins a specific model version for the LLM judge and that provider silently updates the model behind the same name, your faithfulness scores can shift for reasons that have nothing to do with your bot. Pin judge model versions explicitly where the provider allows it, and re-baseline thresholds after any deliberate judge upgrade.

A simple decision rule

If you're deciding today, here's the shortcut version of everything above: start with Ragas if your immediate question is "is my retrieval any good," because it answers that question fastest with the least setup. Move to DeepEval, or add it alongside Ragas, the moment any of these become true — you have a CI pipeline you want evaluation gated on, your bot's behavior extends beyond retrieval-and-generate (tone, safety, multi-turn), or you need custom pass/fail criteria that don't fit faithfulness/relevancy/precision/recall. Most teams that ship a RAG product past the prototype stage end up needing both, just not on day one.

The support bot's team didn't choose Ragas or DeepEval — they chose Ragas, then added DeepEval, then leaned on Ragas's metrics from inside DeepEval's test runner. That's less a compromise than the honest shape of how RAG evaluation needs grow: narrow and fast first, broad and CI-gated later.

If you want the retrieval and generation fundamentals solid before you're choosing evaluation tools at all, that's exactly the gap our "Introduction to RAG" course at teachyou.ai is built to close — chunking strategy, retrieval quality, and generation grounding, taught before you need a framework to tell you it's broken.