Ragas

Ragas in CI/CD: Automating RAG Regression Tests

Pramod Dutta · May 9, 2026 · 15 min read

Your RAG Pipeline Passed Code Review. Did It Pass Reality?

A pull request lands. It changes the chunking strategy in your retrieval-augmented generation pipeline from 512 tokens to 800 tokens. The diff is small. The unit tests are green. A reviewer skims it, sees nothing alarming in the retriever code, and approves it. Two days later, support tickets start trickling in: the chatbot is citing the wrong policy clause, hallucinating refund windows that don't exist in the underlying documents.

This is the blind spot that traditional software testing leaves wide open for RAG systems. Unit tests can confirm your vector store client connects and your prompt template renders without a KeyError. They cannot tell you whether the retrieved context actually supports the generated answer, or whether the answer is faithful to what was retrieved. That's a semantic question, not a syntactic one, and it needs a different kind of test.

Ragas (Retrieval-Augmented Generation Assessment) exists to answer exactly that question, and answer it with numbers rather than vibes. It computes metrics like faithfulness, answer relevancy, and context precision by using an LLM as a judge over your pipeline's inputs and outputs. Used interactively, Ragas is a nice-to-have for a Friday afternoon evaluation. Wired into CI/CD, it becomes a regression gate — the same category of protection pytest gives your backend, except for the parts of your system that no assertion statement can check.

This article walks through building that gate: how to structure a Ragas evaluation as a repeatable test suite, how to run it inside GitHub Actions on every pull request, how to fail builds on metric regressions instead of just logging them, and how to keep the whole thing affordable when every CI run burns LLM tokens. We'll write real workflow YAML and real Python, not pseudocode.

Why "It Works On My Laptop" Fails Harder For RAG

Traditional regression testing relies on determinism. Same input, same output, every time. RAG pipelines break that assumption in three places at once:

The retriever is stochastic in effect, if not in mechanism. Re-embedding a corpus after a model upgrade, re-indexing after a chunking change, or even a minor version bump in your vector database can shift which documents come back for the same query.
The generator is genuinely non-deterministic. Even at temperature=0, most hosted LLMs don't guarantee bit-identical outputs across calls, and most production RAG systems don't run at temperature=0 anyway because it makes answers robotic.
"Correctness" isn't binary. A generated answer can be partially right, right but poorly grounded, or right for the wrong reasons (citing a document that happens to agree by coincidence rather than by retrieval).

This means the natural unit of testing shifts from "assert equal" to "assert above threshold" on a handful of continuous metrics. Ragas gives you four that matter most for regression testing:

Faithfulness — does the generated answer only contain claims supported by the retrieved context, or is it inventing details?
Answer relevancy — does the generated answer actually address the question asked, or does it wander?
Context precision — of the chunks retrieved, how many were actually relevant to answering the question?
Context recall — did the retriever pull back everything needed to answer the question, or did it miss a necessary chunk?

Faithfulness and context precision are the two most useful for catching regressions caused by pipeline changes, because they isolate generator behavior from retriever behavior. If faithfulness drops after a PR, your generator is drifting. If context precision drops, your retriever is drifting. That separation is what makes Ragas scores actionable in a code review, rather than a single fused "quality" number nobody can act on.

Building a Golden Dataset You Can Actually Regress Against

Before any of this works in CI, you need a fixed evaluation set — the RAG equivalent of a snapshot test's expected output. This is the step teams skip, and it's the one that makes everything downstream meaningless if skipped.

A golden dataset for Ragas needs, at minimum: a question, the contexts your retriever should surface, and (for context recall) a reference answer. Store it as version-controlled data, not as something generated fresh on every run — the whole point is that the questions stay fixed so score changes are attributable to code changes, not to sampling variance in a randomly generated eval set.

# eval/golden_dataset.py
"""
Fixed evaluation set for RAG regression testing.
Every entry should be reviewed by a human before it's trusted as ground truth.
"""

GOLDEN_QUESTIONS = [
    {
        "question": "What is the refund window for annual subscriptions?",
        "ground_truth": (
            "Annual subscriptions can be refunded in full within 14 days "
            "of purchase. After 14 days, refunds are prorated based on "
            "unused months."
        ),
        "reference_contexts": [
            "refund-policy.md#annual-subscriptions",
        ],
    },
    {
        "question": "Can I transfer my course access to another email address?",
        "ground_truth": (
            "Course access can be transferred once per purchase by "
            "contacting support with proof of original purchase."
        ),
        "reference_contexts": [
            "faq.md#account-transfers",
        ],
    },
    {
        "question": "What happens to my progress if I downgrade my plan?",
        "ground_truth": (
            "Progress is preserved indefinitely. Downgrading only "
            "restricts access to premium-tier course content going "
            "forward."
        ),
        "reference_contexts": [
            "billing-faq.md#downgrades",
        ],
    },
    # Aim for 30-50 questions covering your most common query patterns,
    # plus a handful of deliberately adversarial or edge-case questions.
]

Thirty to fifty questions is a realistic starting point for most teams — enough to get statistically meaningful averages per metric, small enough that a CI run finishes in minutes rather than hours. Split them by category (straightforward factual lookups, multi-hop questions, questions with no good answer in the corpus) so you can track regressions by category rather than only in aggregate. A pipeline change that improves average faithfulness but destroys performance on edge-case questions is exactly the kind of regression an aggregate score hides.

Writing the Evaluation Script

With a golden dataset in place, the next piece is a script that runs your actual RAG pipeline against every question, collects the retrieved contexts and generated answers, and scores the result with Ragas. This script is the thing CI will actually invoke, so it needs to be runnable non-interactively and needs to produce a machine-readable result.

# eval/run_ragas_eval.py
import json
import sys
from pathlib import Path

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)

from golden_dataset import GOLDEN_QUESTIONS
from rag_pipeline import run_rag_pipeline  # your actual pipeline entrypoint


def build_eval_dataset():
    """Run the live RAG pipeline against every golden question."""
    questions, answers, contexts, ground_truths = [], [], [], []

    for item in GOLDEN_QUESTIONS:
        result = run_rag_pipeline(item["question"])

        questions.append(item["question"])
        answers.append(result["answer"])
        contexts.append(result["retrieved_chunks"])
        ground_truths.append(item["ground_truth"])

    return Dataset.from_dict(
        {
            "question": questions,
            "answer": answers,
            "contexts": contexts,
            "ground_truth": ground_truths,
        }
    )


def main():
    dataset = build_eval_dataset()

    result = evaluate(
        dataset,
        metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    )

    scores = result.to_pandas()
    summary = {
        "faithfulness": float(scores["faithfulness"].mean()),
        "answer_relevancy": float(scores["answer_relevancy"].mean()),
        "context_precision": float(scores["context_precision"].mean()),
        "context_recall": float(scores["context_recall"].mean()),
        "n_questions": len(GOLDEN_QUESTIONS),
    }

    output_path = Path("eval/results/latest.json")
    output_path.parent.mkdir(parents=True, exist_ok=True)
    output_path.write_text(json.dumps(summary, indent=2))

    print(json.dumps(summary, indent=2))

    # Row-level scores are useful for debugging which specific question
    # regressed, not just the aggregate.
    scores.to_csv("eval/results/latest_rowlevel.csv", index=False)


if __name__ == "__main__":
    main()

Two details here matter more than they look. First, run_rag_pipeline should call your production retrieval and generation code paths directly — not a mocked or simplified version of it. A regression test that exercises different code than production is worse than no regression test, because it gives false confidence. Second, dumping row-level scores to CSV alongside the aggregate JSON means that when a score drops, you don't have to re-run anything to find out which question caused it — you diff the CSV.

Setting a Regression Gate, Not Just a Report

A script that prints scores is an evaluation. A script that fails the build when scores drop is a regression test. The gap between those two is a comparison against a baseline and a non-zero exit code.

# eval/check_regression.py
import json
import sys
from pathlib import Path

# Minimum acceptable score per metric. These aren't arbitrary — they should
# be set from your own historical scores, with some margin, not copied
# from a blog post.
THRESHOLDS = {
    "faithfulness": 0.85,
    "answer_relevancy": 0.80,
    "context_precision": 0.75,
    "context_recall": 0.75,
}

# Maximum allowed drop versus the last known-good baseline, even if the
# absolute threshold above is still met. Catches slow drift.
MAX_REGRESSION = 0.05


def load_json(path):
    return json.loads(Path(path).read_text())


def main():
    latest = load_json("eval/results/latest.json")

    baseline_path = Path("eval/results/baseline.json")
    baseline = load_json(baseline_path) if baseline_path.exists() else None

    failures = []

    for metric, floor in THRESHOLDS.items():
        score = latest[metric]

        if score < floor:
            failures.append(
                f"{metric}={score:.3f} is below absolute floor {floor:.3f}"
            )

        if baseline is not None:
            drop = baseline[metric] - score
            if drop > MAX_REGRESSION:
                failures.append(
                    f"{metric}={score:.3f} dropped {drop:.3f} from "
                    f"baseline {baseline[metric]:.3f} (max allowed {MAX_REGRESSION})"
                )

    if failures:
        print("RAGAS REGRESSION CHECK FAILED:")
        for f in failures:
            print(f"  - {f}")
        sys.exit(1)

    print("RAGAS regression check passed.")
    for metric, floor in THRESHOLDS.items():
        print(f"  {metric}: {latest[metric]:.3f} (floor {floor:.3f})")


if __name__ == "__main__":
    main()

The two-layer check — absolute floor plus max-allowed-drop from baseline — matters because either check alone misses failure modes. An absolute floor alone lets a pipeline slowly decay from 0.95 to 0.86 undetected, one PR at a time, as long as it never dips below 0.85. A drop-from-baseline check alone lets a pipeline that's already mediocre stay mediocre forever, since it never "regresses" from an already-low baseline. Together, they catch both a sudden cliff and death by a thousand cuts.

The baseline file itself should only be updated deliberately, typically by a maintainer merging to the main branch after confirming a score change is an intentional improvement, not automatically on every commit. Treat it like a snapshot test's approved snapshot — updating it is a decision, not a side effect.

The GitHub Actions Workflow

Here's the part that actually wires this into CI. The workflow runs on pull requests that touch RAG-relevant paths, installs dependencies, runs the evaluation, checks it against the gate, and posts the results as a PR comment so reviewers see the numbers without digging through logs.

# .github/workflows/ragas-regression.yml
name: RAG Regression Tests

on:
  pull_request:
    paths:
      - "rag_pipeline/**"
      - "eval/**"
      - "prompts/**"
      - "requirements.txt"

jobs:
  ragas-eval:
    runs-on: ubuntu-latest
    timeout-minutes: 20

    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"
          cache: "pip"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          pip install ragas datasets

      - name: Run RAG pipeline against golden dataset
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          VECTOR_DB_URL: ${{ secrets.VECTOR_DB_URL }}
        run: python eval/run_ragas_eval.py

      - name: Pull baseline scores from main
        run: |
          git fetch origin main --depth=1
          git show origin/main:eval/results/baseline.json > eval/results/baseline.json || echo "No baseline found on main yet."

      - name: Check for regressions
        id: gate
        run: python eval/check_regression.py

      - name: Upload evaluation artifacts
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: ragas-results
          path: |
            eval/results/latest.json
            eval/results/latest_rowlevel.csv

      - name: Comment results on PR
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const summary = JSON.parse(fs.readFileSync('eval/results/latest.json', 'utf8'));
            const status = '${{ steps.gate.outcome }}' === 'success' ? '✅ Passed' : '❌ Failed';

            const body = [
              `### Ragas Regression Check: ${status}`,
              '',
              '| Metric | Score |',
              '|---|---|',
              `| Faithfulness | ${summary.faithfulness.toFixed(3)} |`,
              `| Answer Relevancy | ${summary.answer_relevancy.toFixed(3)} |`,
              `| Context Precision | ${summary.context_precision.toFixed(3)} |`,
              `| Context Recall | ${summary.context_recall.toFixed(3)} |`,
              '',
              `_Evaluated against ${summary.n_questions} golden questions._`,
            ].join('\n');

            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: body,
            });

A few choices in this workflow are deliberate. The paths filter means the evaluation only runs when something relevant to RAG behavior actually changed — no point burning LLM tokens re-scoring a pipeline because someone edited the README. The baseline is pulled from main at run time rather than committed alongside every PR branch, so it always reflects "the last thing that was actually merged," not a stale copy that drifted from a rebase. And if: always() on the artifact upload and comment steps means you get visibility into what happened even when the gate step fails and would otherwise short-circuit the job.

Note that this workflow prints "NO tables" isn't a constraint on the YAML — the constraint about no tables applies to this article's prose, not to the PR comment your CI posts, which is free to use whatever GitHub Markdown renders well for reviewers.

Handling Non-Determinism Without Making the Gate Useless

The most common objection to this setup is: "LLM judges themselves aren't deterministic, so won't the gate be flaky?" It's a fair concern, and ignoring it produces a CI check nobody trusts, which is worse than no check at all because people start merging past red builds out of habit.

Three mitigations handle most of the flakiness:

Average over the full golden set, not single questions. A 40-question average is far more stable run-to-run than any individual question's score, because per-question judge noise partially cancels out across the set.
Use a fixed, low-temperature judge model for scoring. Ragas lets you configure which LLM acts as the judge separately from which LLM powers your actual RAG pipeline. Pin the judge model version explicitly and keep its temperature near zero — you want judging to be as consistent as possible even while your production generator is not.
Give the drop threshold real margin. A MAX_REGRESSION of 0.05 (as in the script above) absorbs normal judge noise while still catching a pipeline change that meaningfully degrades quality. If you find the gate flapping on unrelated PRs, that's a signal to widen the margin or grow the golden dataset, not to delete the check.

# eval/ragas_config.py
"""Pin the judge model so evaluation runs are comparable across time."""
from langchain_openai import ChatOpenAI
from ragas.llms import LangchainLLMWrapper

JUDGE_MODEL = LangchainLLMWrapper(
    ChatOpenAI(model="gpt-4o-2024-08-06", temperature=0)
)

# Pass this explicitly into evaluate(..., llm=JUDGE_MODEL) so that a
# provider-side default-model bump doesn't silently change your scores
# out from under you.

Pinning the judge model to a dated snapshot rather than a floating alias is the detail teams miss most often. If your judge model silently updates underneath you, a score drop in next month's CI run might mean nothing about your pipeline at all — it might just mean the judge got stricter. Version the judge the same way you'd version any other dependency.

Cost Control: Making This Sustainable Long-Term

Running an LLM-as-judge evaluation over dozens of questions, on every relevant pull request, adds up in API spend if left unmanaged. A few practical guardrails keep this affordable without gutting its usefulness:

Scope the path filter tightly. The paths trigger in the workflow above already limits runs to changes that could plausibly affect RAG quality. Resist the temptation to run it on every PR "just in case."
Cache embeddings where possible. If your golden dataset's questions don't change between runs, and your corpus hasn't changed, you may be able to cache retrieval results and only re-run the generation-and-judging half of the pipeline, cutting cost roughly in half.
Run the full metric suite on PRs, but consider a lighter nightly deep-dive. PRs get faithfulness and context precision (the two most diagnostic, and cheaper to compute than context recall, which requires additional judge calls). A nightly scheduled workflow can run the complete four-metric suite plus a larger, rotating sample of questions.
Set a hard budget alert. Most LLM providers support spend alerts. Point one at whatever project or API key your CI uses, so a runaway loop (a workflow accidentally triggering on every push instead of every PR, for instance) gets caught in hours, not at month-end billing.

# Excerpt: lighter PR-time job vs. full nightly job
jobs:
  ragas-eval-pr:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - run: python eval/run_ragas_eval.py --metrics faithfulness,context_precision

  ragas-eval-nightly:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - run: python eval/run_ragas_eval.py --metrics all --sample-size 200

This split — fast, narrow checks on every PR; slower, comprehensive checks on a schedule — is the same pattern most teams already use for their broader test suites, where unit tests run on every commit and a full integration or load test runs nightly. There's no reason RAG evaluation should be architected any differently.

What This Doesn't Replace

It's worth being explicit about the limits of this setup, because oversold testing tools breed the same complacency as no testing at all. Ragas regression tests catch drift on the questions in your golden dataset. They don't catch:

Novel failure modes on questions nobody thought to add to the golden set — which is why the dataset needs periodic review, ideally seeded from real production queries that got low user feedback scores.
Latency and cost regressions, which need their own monitoring separate from quality metrics.
Safety and policy violations, which typically need a dedicated guardrail layer rather than a quality metric like faithfulness.

Treat this pipeline as one layer in a larger evaluation strategy, alongside human review of a sample of production traffic and whatever safety-specific checks your domain requires. It closes a specific, previously-open gap — silent quality regressions slipping through PR review — without pretending to be a complete answer to "is this RAG system good."

Getting This Running in Your Own Pipeline

The setup described here — golden dataset, evaluation script, regression gate, GitHub Actions workflow — is roughly a day of work for a team that already has a working RAG pipeline and a CI setup. The parts that actually take iteration are the ones that look easiest on paper: writing a golden dataset that reflects real user questions instead of the ones that are convenient to write, and tuning thresholds against your own historical scores instead of borrowing numbers from someone else's blog post.

If you want a guided, hands-on walkthrough of Ragas metrics, judge model configuration, and building this exact CI pipeline end to end, our Ragas Tutorial course on teachyou.ai covers it step by step, including the golden dataset design decisions and threshold-tuning process that this article only had room to summarize. It's built for engineers who already have a RAG system in production and want to stop finding out about quality regressions from support tickets instead of from their own CI.