Migrating from Manual QA to DeepEval: A Team's Journey

The Friday afternoon that started it all

Every team that ships an LLM feature eventually has the same Friday afternoon. A prompt change that looked harmless in testing quietly makes the production chatbot start hallucinating refund policies. Someone notices because a customer screenshots it and posts it in a support channel, not because a test caught it. Then everyone spends the weekend re-reading transcripts trying to figure out what changed.

That was roughly the situation for a mid-sized team building a RAG-based support assistant that we'll describe here without naming specific companies, because the pattern is close to universal. Their QA process for the LLM layer was a shared spreadsheet: forty-ish "golden" questions, a column for the expected answer, a column for what the model actually said, and a human being who read every row and marked it pass or fail. It worked, technically. It also took a full day of someone's time before every release, it did not scale past forty questions, and it caught regressions roughly a week after they shipped rather than before.

This article walks through why that team moved to DeepEval, what the migration actually looked like in code, where it got messy, and what changed about how they ship prompt and RAG updates. If you are staring at your own spreadsheet of "does this answer look right" right now, the goal is to give you a realistic picture of the before and after, not a sales pitch.

What manual QA actually looked like

It's worth being specific about the manual process, because "manual QA" undersells how much real engineering effort was going into something that still didn't scale.

The workflow before DeepEval looked like this:

• A product manager or support lead maintained a Google Sheet of representative user questions, organized loosely by feature area (billing, account settings, product troubleshooting).
• Before each deploy, an engineer would run those questions through the staging chatbot, either by hand or with a small script that hit the API and dumped responses into a new sheet column.
• A human reviewer read each response and compared it against the "expected" answer, which was itself just prose written by whoever created the row six months earlier.
• Pass/fail was subjective. "Close enough" was a real category. Two reviewers would sometimes disagree on the same row.
• Regressions in tone, hallucinated facts, or broken citation formatting were caught only if the reviewer happened to notice, since there was no structured signal for "did this response actually use the retrieved documents."

The failure mode wasn't that the process was lazy. It was that it didn't compose. Every new feature meant more rows nobody had time to add, every prompt tweak meant re-reading forty responses from scratch, and there was no way to tell whether a regression was in the retrieval step, the prompt, or the underlying model version, because the review was a single opaque judgment on the final text.

Why DeepEval specifically

The team looked at three options: keep expanding the spreadsheet with more automation glued on, build an in-house eval harness from scratch, or adopt an existing open-source framework. They picked DeepEval for a few concrete reasons that are worth calling out rather than just naming the tool.

• It's built around pytest. The team already had a CI pipeline running pytest for the rest of the backend. DeepEval's test cases are just pytest functions decorated with assertions, so there was no new test runner, no new CI config category, and no separate dashboard to check before a merge could go green.
• Metrics are pluggable and interpretable. Instead of one fuzzy "does this look right" judgment, DeepEval separates concerns: answer relevancy, faithfulness to retrieved context, contextual precision and recall, hallucination detection, and toxicity/bias checks are all independent, independently scoreable metrics. That maps almost one-to-one onto the separate things the human reviewer had been conflating in their head.
• It supports both LLM-as-judge and classic metrics. Some checks (like exact-match style structural validation, e.g., "does the response contain a valid order ID format") don't need an LLM judge at all. Others (like faithfulness) genuinely benefit from a judge model reasoning over the retrieved context and the answer. DeepEval lets you mix both in the same suite.
• Synthetic dataset generation. The forty-row spreadsheet became a real bottleneck once the team wanted broader coverage. DeepEval's Synthesizer can expand a small seed set of documents or examples into a larger, more diverse test dataset, which mattered a lot once they needed regression coverage across more feature areas than a human had time to hand-write questions for.

None of this means DeepEval was a magic fix. It means it matched the shape of the existing problem: a team that already trusted pytest, already had a RAG pipeline whose retrieval and generation steps could be evaluated separately, and needed more test cases than they had bandwidth to write by hand.

Phase one: replacing the easiest 20% first

The instinct on day one was to try to migrate the entire spreadsheet in one go. That stalled almost immediately, because writing good expected_output strings and deciding on the right metric thresholds for forty diverse rows is genuinely hard to do all at once. The team scaled it back and moved only the questions that had the clearest, most checkable expected behavior first: direct factual questions with a single correct answer sourced from a document the retrieval system was supposed to fetch.

Here's roughly what that first slice looked like, deliberately kept small:

from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

def get_chatbot_response(question: str):
    # wraps the existing RAG pipeline: retrieval + generation
    retrieved_docs = retriever.get_relevant_documents(question)
    answer = generate_answer(question, retrieved_docs)
    return answer, [doc.page_content for doc in retrieved_docs]

def test_refund_policy_question():
    question = "How long do I have to request a refund after purchase?"
    answer, retrieved_context = get_chatbot_response(question)

    test_case = LLMTestCase(
        input=question,
        actual_output=answer,
        retrieval_context=retrieved_context,
        expected_output="Refunds can be requested within 14 days of purchase."
    )

    faithfulness = FaithfulnessMetric(threshold=0.8)
    relevancy = AnswerRelevancyMetric(threshold=0.7)

    assert_test(test_case, [faithfulness, relevancy])

Running this locally with deepeval test run test_refund_policy.py produced something the spreadsheet never had: a numeric faithfulness score and a printed reason from the judge model explaining *why* it scored the way it did. The first time this ran against production traffic patterns, it caught a case where the retriever was pulling an outdated refund policy document that a human reviewer had missed twice, because the answer "looked" right on the surface even though it cited the wrong window (30 days instead of 14).

That was the moment the migration got internal buy-in. Not because DeepEval was faster (it wasn't, for that first ten-question slice), but because it caught something the manual process had already missed twice.

Phase two: mapping spreadsheet columns to metrics

Once the team had a working pattern, the bulk of the migration was mechanical but not trivial: going row by row through the old spreadsheet and deciding which DeepEval metric actually matched the intent behind each manual check.

This turned out to be the most useful part of the whole migration, because it forced the team to articulate what "correct" had actually meant for each row, instead of leaving it as an implicit judgment call.

• Rows checking factual accuracy against source documents became FaithfulnessMetric and ContextualPrecisionMetric checks.
• Rows checking "does this actually answer what was asked" (a surprising number of old failures were the bot answering an adjacent but wrong question) became AnswerRelevancyMetric.
• Rows that existed to catch the bot making up a policy that didn't exist anywhere in the docs became HallucinationMetric, using the source documents as the reference context.
• A handful of rows about tone ("don't sound robotic when apologizing for an outage") were, frankly, hard to formalize and became a smaller custom GEval metric with a natural-language rubric, since they didn't fit the standard RAG metrics.

The GEval piece deserves its own snippet, since it's the part that handled the "vibes-based" checks the spreadsheet had been quietly encoding:

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

empathetic_tone = GEval(
    name="Empathetic Tone",
    criteria=(
        "Determine whether the response acknowledges the user's "
        "frustration before providing a solution, and avoids "
        "sounding like a form letter."
    ),
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT
    ],
    threshold=0.6
)

This is a good place to be honest about a limitation: GEval, like any LLM-as-judge metric, is not perfectly deterministic. Running the same test case twice can occasionally produce a slightly different score, especially near the threshold boundary. The team dealt with this the straightforward way — by treating scores near the threshold as "needs human review" rather than a hard pass/fail, and by pinning the judge model version so at least the variance came from one source instead of two.

Phase three: wiring it into CI

Manual QA lived entirely outside CI, which was itself part of the problem — it happened right before a release, as a separate ritual, rather than on every pull request. The migration's real payoff came from making DeepEval part of the standard test suite.

The CI change was smaller than expected, because DeepEval tests are just pytest tests:

# .github/workflows/eval.yml
name: LLM Evaluation Suite
on:
  pull_request:
    paths:
      - "app/rag/**"
      - "app/prompts/**"

jobs:
  deepeval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt
      - run: deepeval test run tests/eval/ --junitxml=results.xml
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: eval-results
          path: results.xml

Two decisions here mattered more than they seemed to at first:

• Path filtering. The eval suite only runs when RAG or prompt code changes, not on every unrelated backend PR. Running the full suite costs real judge-model API calls and adds a few minutes to CI, so scoping it to the code paths that could actually affect model behavior kept the team from getting cost or patience complaints later.
• `--junitxml` output. Feeding results into the standard JUnit format meant the existing CI dashboard could show LLM eval failures next to regular unit test failures, instead of needing a separate UI. This was a small technical choice that had an outsized effect on adoption — engineers didn't have to learn a new tool to see why their PR was red.

The first month of CI integration surfaced a problem nobody had planned for: judge-model API costs. Running fifteen metrics across sixty test cases on every relevant PR adds up. The team's fix was to tier the suite — a fast subset of ten high-signal test cases ran on every PR, and the full suite ran nightly against the main branch plus once before any production deploy. That tiering is a detail worth stealing directly if you're doing this migration yourself, because "run everything on every push" is the version that gets disabled six weeks in when someone complains about the CI bill.

What broke during the migration

It would be dishonest to describe this as a clean win with no friction, so here's what actually went wrong.

• Flaky thresholds. The first pass at metric thresholds was too strict, copied from DeepEval's example docs rather than calibrated against this team's actual data. FaithfulnessMetric at 0.9 failed on answers a human would call obviously correct, because the judge model was penalizing minor phrasing that didn't map cleanly onto the retrieved sentence. The fix was boring: run the suite against fifty known-good historical responses first, look at the score distribution, and set thresholds based on that distribution rather than a default number.
• Judge model cost and latency. Nightly runs were fine. Running the full suite on every single commit to a feature branch was not, both for cost and for the three-to-five minutes it added to CI. Tiering (fast subset per-PR, full suite nightly/pre-release) fixed this, as mentioned above.
• Retrieval context wasn't always captured. Several of the older RAG endpoints didn't expose which documents had actually been retrieved, only the final answer. FaithfulnessMetric and ContextualPrecisionMetric need retrieval_context to do their job, so part of the migration was quietly plumbing that data through the existing pipeline before it could even be tested properly. If your RAG service doesn't currently log or return retrieved chunks somewhere, budget time for this step — it's not a DeepEval problem, but it will block you.
• Not every check translates. A few of the old spreadsheet rows were really testing UI behavior (does the citation link render correctly) rather than model behavior, and those stayed as ordinary integration tests instead of being forced into DeepEval. Trying to make an eval framework own every kind of test is a mistake; DeepEval is for judging model output quality, not for testing your frontend.

The result, in plain terms

No invented percentages here, just what changed operationally:

• The pre-release QA day disappeared. The equivalent checks now run automatically on every relevant pull request and again before deploy, with no one blocking a release to manually read through transcripts.
• Regressions get caught at PR review time instead of after a support ticket. The specific refund-policy document bug from phase one is the clearest example — that class of error is now caught by an automated faithfulness check before it ever reaches staging, let alone production.
• The test suite scaled past what the spreadsheet ever could. Synthetic dataset generation let the team grow from roughly forty hand-written questions to a much broader, still-growing set of test cases covering more feature areas, without needing someone to sit down and author each one by hand.
• Disagreements about "does this answer look right" turned into disagreements about "is this threshold calibrated correctly," which is a more productive argument to have, because it's backed by a score and a stated reason instead of two people's gut feelings.

Practical advice if you're starting this migration

If you're looking at your own version of that spreadsheet right now, a few things from this team's experience are worth taking directly:

1. Don't migrate everything at once. Pick the ten or twenty test cases with the clearest, most checkable expected output and start there.
2. Map each old manual check to the DeepEval metric that actually matches its intent — faithfulness, relevancy, hallucination, contextual precision/recall, or a custom GEval rubric for anything qualitative like tone.
3. Calibrate thresholds against your own historical data before trusting default numbers from documentation.
4. Make sure your RAG pipeline actually exposes retrieved context somewhere accessible, since several of the most useful metrics depend on it.
5. Tier your CI runs. A fast subset on every PR, a full suite on a schedule or before deploy, so cost and latency don't quietly become the reason the whole effort gets abandoned.
6. Treat borderline judge-model scores as a signal for human review, not as an automatic pass or fail, especially early on while thresholds are still being tuned.

The underlying shift isn't really "spreadsheet bad, framework good." It's that DeepEval gave this team a vocabulary — faithfulness, relevancy, hallucination, contextual precision — for things they had already been checking by feel, and then let them enforce that vocabulary automatically, on every change, instead of once a week by hand.

If you want a structured, hands-on walkthrough of setting this up yourself — writing your first test cases, choosing and calibrating metrics, wiring DeepEval into CI, and handling the same retrieval-context and cost issues this team ran into — that's exactly what the DeepEval Tutorial course on teachyou.ai covers, from a blank repository to a working evaluation suite you can trust before every deploy.