Integrating DeepEval into GitHub Actions: A CI/CD Recipe

Your LLM pull request looks fine. The diff is small, the prompt tweak is "obviously" an improvement, and every reviewer approves it in ninety seconds. Then it ships, and three days later someone notices the summarizer started hallucinating dates. This is the exact failure mode that unit tests solved for regular code twenty years ago, and it is the failure mode DeepEval in GitHub Actions solves for LLM code today. If your eval suite only runs when someone remembers to run it locally, it does not exist. This post is a recipe, not a philosophy piece: the workflow YAML, the Python test file, the caching strategy, the secrets handling, and the PR comment integration, all wired together so a bad prompt change fails the build instead of failing in production.

Why CI is the only place evals actually matter

Evals that live only in a notebook or a scripts/eval.py file that nobody runs are decoration. They get skipped under deadline pressure, and deadline pressure is exactly when regressions slip through. The value of an eval suite is proportional to how automatically and how consistently it runs, not how sophisticated its metrics are.

Putting DeepEval in GitHub Actions gives you three things a local script cannot:

• A forcing function: the PR cannot merge if the build is red (once you turn on branch protection).
• A shared source of truth: every contributor sees the same eval run against the same model version, not "works on my machine."
• A historical record: every PR gets an eval score, so you can see when quality started drifting, not just that it did.

The rest of this post assumes you already have a DeepEval test suite locally (even a handful of assert_test calls using GEval or AnswerRelevancyMetric is enough to start) and you want to make it run automatically, safely, and without burning your OpenAI bill every time someone pushes a typo fix to the README.

Step 1: Structure your eval suite so CI can run it standalone

Before touching YAML, make sure your evals are runnable headlessly. DeepEval tests are just pytest tests under the hood, so the structure should look familiar to anyone who has written a Python test suite.

# tests/eval/test_support_agent.py
import os
import pytest
from deepeval import assert_test
from deepeval.metrics import GEval, AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

from app.agent import run_support_agent

correctness_metric = GEval(
    name="Correctness",
    criteria="Determine if the actual output is factually correct given the expected output.",
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
        LLMTestCaseParams.EXPECTED_OUTPUT,
    ],
    threshold=0.7,
    model="gpt-4o-mini",
)

relevancy_metric = AnswerRelevancyMetric(threshold=0.7, model="gpt-4o-mini")
faithfulness_metric = FaithfulnessMetric(threshold=0.7, model="gpt-4o-mini")


@pytest.mark.parametrize(
    "question,expected,context",
    [
        (
            "How do I reset my password?",
            "Go to Settings > Security > Reset Password, then check your email.",
            ["Password resets are handled via Settings > Security > Reset Password."],
        ),
        (
            "Can I get a refund after 30 days?",
            "No, refunds are only available within 30 days of purchase.",
            ["Refund policy: 30 days from purchase date, no exceptions."],
        ),
    ],
)
def test_support_agent_quality(question, expected, context):
    actual_output = run_support_agent(question)

    test_case = LLMTestCase(
        input=question,
        actual_output=actual_output,
        expected_output=expected,
        retrieval_context=context,
    )

    assert_test(test_case, [correctness_metric, relevancy_metric, faithfulness_metric])

Two details matter here for CI specifically. First, use a cheap, fast judge model (gpt-4o-mini or similar) for the metrics themselves — the judge model is a separate concern from the model you're testing, and CI runs pay for both. Second, keep this file isolated in tests/eval/ so you can target it independently from your regular unit tests, since eval runs are slower and cost real money per run.

Step 2: Handle API keys and secrets safely

DeepEval needs an LLM provider key to run its judge models (usually OPENAI_API_KEY), and your application under test probably needs its own key too. Never hardcode these, and never echo them in logs.

In your GitHub repo, go to Settings > Secrets and variables > Actions and add:

• OPENAI_API_KEY — used both by DeepEval's default judge and possibly your app
• CONFIDENT_API_KEY — optional, only if you're pushing results to Confident AI's dashboard
• Any provider keys your app under test needs (ANTHROPIC_API_KEY, etc.)

A critical detail that trips people up: secrets are not available to workflows triggered by pull requests from forks, by default, because GitHub blocks secret access on pull_request events from external forks for security reasons. If your repository is public and you expect outside contributors, you have two options:

1. Use pull_request_target instead of pull_request (with the caveat that you must be careful not to check out and execute untrusted code with that trigger — checkout the PR head only for read-only steps).
2. Gate the eval job so it only runs automatically for internal branches, and require a maintainer to manually approve external PR runs via the "Approve and run workflow" button (this is the default GitHub behavior for first-time contributors and is usually the safer choice).

For an internal team repo (private, or trusted contributors only), plain pull_request is fine and secrets flow through normally. That's what we'll use below.

Step 3: The GitHub Actions workflow file

Here is a complete, working workflow. It installs dependencies with caching, runs the eval suite, generates a JSON report, posts a PR comment with the results, and fails the build on regression.

# .github/workflows/deepeval.yml
name: DeepEval CI

on:
  pull_request:
    branches: [main]
    paths:
      - "app/**"
      - "tests/eval/**"
      - "prompts/**"
      - "requirements.txt"

concurrency:
  group: deepeval-${{ github.event.pull_request.number }}
  cancel-in-progress: true

permissions:
  contents: read
  pull-requests: write

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      DEEPEVAL_TELEMETRY_OPT_OUT: "YES"

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Cache pip dependencies
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: pip-${{ runner.os }}-${{ hashFiles('requirements.txt') }}
          restore-keys: |
            pip-${{ runner.os }}-

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          pip install deepeval pytest-xdist

      - name: Cache DeepEval eval results
        uses: actions/cache@v4
        with:
          path: .deepeval_cache.json
          key: deepeval-cache-${{ hashFiles('tests/eval/**') }}-${{ github.event.pull_request.base.sha }}
          restore-keys: |
            deepeval-cache-

      - name: Run DeepEval test suite
        id: run_evals
        run: |
          deepeval test run tests/eval/ \
            -n 4 \
            --use-cache \
            -m "not slow" \
            --junitxml=eval-results.xml \
            || echo "EVAL_FAILED=true" >> "$GITHUB_ENV"

      - name: Generate JSON summary
        if: always()
        run: |
          python scripts/summarize_eval_results.py eval-results.xml > eval-summary.md

      - name: Post PR comment with results
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const summary = fs.readFileSync('eval-summary.md', 'utf8');
            const marker = '<!-- deepeval-report -->';
            const body = `${marker}\n## DeepEval Results\n\n${summary}`;

            const { data: comments } = await github.rest.issues.listComments({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
            });

            const existing = comments.find(c => c.body.includes(marker));

            if (existing) {
              await github.rest.issues.updateComment({
                owner: context.repo.owner,
                repo: context.repo.repo,
                comment_id: existing.id,
                body,
              });
            } else {
              await github.rest.issues.createComment({
                owner: context.repo.owner,
                repo: context.repo.repo,
                issue_number: context.issue.number,
                body,
              });
            }

      - name: Write job summary
        if: always()
        run: cat eval-summary.md >> "$GITHUB_STEP_SUMMARY"

      - name: Fail build on regression
        if: env.EVAL_FAILED == 'true'
        run: |
          echo "DeepEval detected a quality regression. See PR comment for details."
          exit 1

A few choices here are deliberate. The paths filter means this expensive job only triggers when prompts, agent code, or the eval suite itself changes, not on every doc typo fix. The concurrency block cancels stale runs when someone pushes a new commit before the old eval finishes, which saves real money on judge-model calls. And permissions: pull-requests: write is required for the comment-posting step to work under the default restrictive token permissions GitHub now ships.

Step 4: Caching to avoid re-running expensive evals

Two different caching layers matter here, and it's easy to conflate them.

Dependency caching (the actions/cache step for ~/.cache/pip) just speeds up pip install. It saves you thirty seconds to a minute per run and is table stakes for any Python CI job.

Eval result caching is the one that actually saves money. DeepEval ships a built-in --use-cache flag for deepeval test run that skips re-invoking the LLM for test cases whose inputs, actual outputs, and metric configuration haven't changed since the last run. This is enormously valuable when your PR touches one prompt template but your eval suite has two hundred test cases — only the affected ones need fresh judge calls.

The trick in the workflow above is keying the GitHub Actions cache to both the eval test file contents (hashFiles('tests/eval/**')) and the base branch SHA, so a fresh eval cache is pulled whenever the test cases themselves change, but reused across pushes to the same PR when only unrelated app code changes. Without restore-keys, a cache miss on the primary key means starting from zero; the fallback key lets you at least partially warm-start from a near match.

Be honest with your team about what caching buys you: it does not let you skip evaluating a changed prompt. If the actual output changes, the cache key for that test case changes, and DeepEval re-invokes the judge. That's correct behavior — caching is a speed and cost optimization for the *unchanged* parts of your suite, not a way to dodge coverage.

Step 5: Failing the build on regression, not just failure

There's a subtlety worth calling out: "the build failed" and "quality regressed" are not automatically the same thing. A naive setup fails the build any time any test drops below its metric threshold, which sounds right until you realize your baseline pass rate might already be 85%, not 100%, because some categories of questions are genuinely hard for your agent.

Two approaches handle this better than a blanket pass/fail:

Absolute thresholds per metric — what we did above — are the simplest and are fine once your test suite is mature and each threshold has been deliberately tuned (threshold=0.7 on a GEval correctness check, for instance). This is the right default for most teams starting out.

Baseline comparison is more sophisticated: run the eval suite against the PR branch and against main, and fail only if the aggregate score drops by more than some tolerance (say, 3 percentage points). This catches regressions even in suites where individual test cases are noisy, at the cost of running the suite twice.

# scripts/compare_to_baseline.py
import json
import sys

def load_pass_rate(path):
    with open(path) as f:
        data = json.load(f)
    total = len(data["testCases"])
    passed = sum(1 for tc in data["testCases"] if tc["success"])
    return passed / total if total else 0.0

pr_rate = load_pass_rate("eval-results-pr.json")
base_rate = load_pass_rate("eval-results-base.json")
tolerance = 0.03

print(f"Base pass rate: {base_rate:.2%}")
print(f"PR pass rate:   {pr_rate:.2%}")

if base_rate - pr_rate > tolerance:
    print(f"REGRESSION: dropped more than {tolerance:.0%}")
    sys.exit(1)

print("No significant regression detected.")

Start with absolute thresholds. Move to baseline comparison once you have enough historical data to know what "normal" noise looks like for your suite — otherwise you're just building a second unreliable signal on top of the first.

Step 6: Posting results as a PR comment and step summary

The actions/github-script step in the workflow above does the heavy lifting: it reads a generated markdown summary and either creates a new PR comment or updates the existing one (using an HTML comment marker so repeated pushes update one comment instead of spamming ten). This matters more than it sounds — a PR with fifteen "DeepEval Results" comments from fifteen pushes is noise nobody reads, and noise nobody reads gets ignored the one time it actually matters.

The summarizer script itself just needs to turn eval-results.xml (or DeepEval's native JSON output) into readable markdown:

# scripts/summarize_eval_results.py
import sys
import xml.etree.ElementTree as ET

def summarize(xml_path):
    tree = ET.parse(xml_path)
    root = tree.getroot()

    total = int(root.attrib.get("tests", 0))
    failures = int(root.attrib.get("failures", 0))
    errors = int(root.attrib.get("errors", 0))
    passed = total - failures - errors

    lines = [
        f"**{passed}/{total} test cases passed**",
        "",
        "| Status | Count |",
        "|---|---|",
    ]
    # Note: this renders as plain lines in non-table contexts;
    # adapt to your own renderer if it doesn't support markdown tables.
    lines = [
        f"**{passed}/{total} test cases passed**",
        "",
        f"- Passed: {passed}",
        f"- Failed: {failures}",
        f"- Errors: {errors}",
        "",
    ]

    if failures or errors:
        lines.append("### Failing cases")
        for testcase in root.iter("testcase"):
            failure = testcase.find("failure")
            if failure is not None:
                name = testcase.attrib.get("name", "unknown")
                message = (failure.attrib.get("message") or "").split("\n")[0][:200]
                lines.append(f"- `{name}`: {message}")

    return "\n".join(lines)


if __name__ == "__main__":
    print(summarize(sys.argv[1]))

Also worth doing: write the same summary to $GITHUB_STEP_SUMMARY, as the workflow does. That gives you a persistent, rich-rendered report attached to the workflow run itself, which is useful once a PR merges and the comment thread is no longer the first place people look for eval history.

If your team wants a dashboard rather than a PR comment, DeepEval's maintainers (Confident AI) offer a hosted option where deepeval login plus a CONFIDENT_API_KEY secret pushes every run to a web UI with trend lines across commits. Worth evaluating once your suite outgrows "read the PR comment."

Step 7: Handling flaky, non-deterministic LLM calls in CI

This is the part that makes LLM CI genuinely harder than regular CI, and skipping it is how teams end up disabling their eval job six weeks in because "it's always flaky."

Pin temperature to zero (or as close as the provider allows) for anything you're testing deterministically. If your application code calls the model under test with temperature=0.7 in production for creativity, consider whether your eval harness should override that for reproducibility, or explicitly test at production temperature and accept some variance. Be deliberate, don't leave it as whatever the default happens to be.

# app/agent.py (excerpt)
def run_support_agent(question: str, temperature: float | None = None) -> str:
    # Allow eval harness to force temperature=0 without touching prod defaults
    temp = temperature if temperature is not None else DEFAULT_TEMPERATURE
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": question}],
        temperature=temp,
    )
    return response.choices[0].message.content

The judge model is also non-deterministic, even at low temperature, and this is the part people forget. GEval and DeepEval's other LLM-based metrics call an LLM to *score* your output, so score variance compounds output variance. Set the judge model's temperature low as well (DeepEval defaults are already reasonable here, but verify), and prefer metrics with a numeric rubric (GEval with clear evaluation steps) over vague open-ended criteria, which reduces judge variance directly.

Retry with backoff on transient API failures, which are common in CI where you're making dozens or hundreds of calls in a short burst and can hit rate limits. DeepEval handles some of this internally, but for your own application calls, wrap them explicitly:

# app/utils/retry.py
import time
import random
from functools import wraps

def retry_with_backoff(max_retries=3, base_delay=1.0):
    def decorator(fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            for attempt in range(max_retries):
                try:
                    return fn(*args, **kwargs)
                except Exception as e:
                    if attempt == max_retries - 1:
                        raise
                    delay = base_delay * (2 ** attempt) + random.uniform(0, 0.5)
                    time.sleep(delay)
            raise RuntimeError("unreachable")
        return wrapper
    return decorator

Allow a small tolerance for flaky-but-not-broken cases. Rather than treating any single failing test case as a hard build failure, consider a "quarantine" pattern: tag known-flaky test cases with a pytest marker, track their pass rate separately, and only fail the build on core (non-quarantined) test failures or on a sustained drop in the quarantined pass rate.

@pytest.mark.flaky_llm
def test_creative_response_quality():
    ...

# Run core suite strictly, quarantined suite informationally
deepeval test run tests/eval/ -m "not flaky_llm"
deepeval test run tests/eval/ -m "flaky_llm" || true

This keeps the signal meaningful. A build that cries wolf on inherent LLM variance trains your team to ignore it, which defeats the entire point of putting evals in CI.

Step 8: Scaling the pattern as your suite grows

Once this is working for one agent or one prompt, the same skeleton extends cleanly:

• Matrix the job across models if you're comparing providers or checking a prompt still works after a model upgrade — strategy: matrix: model: [gpt-4o, gpt-4o-mini, claude-sonnet] and pass the model into your test fixtures.
• Split slow and fast suites — a -m "not slow" marker for the PR-blocking suite, and a nightly scheduled workflow (on: schedule) that runs the full suite including expensive multi-turn conversation evals.
• Track cost per run — log token usage from your judge calls and surface it in the same PR comment, so a suite that quietly triples in judge-model spend doesn't go unnoticed until the invoice arrives.
• Version your eval datasets alongside your prompts — if a test case's expected output changes, that's a deliberate decision and should show up as a diff in the PR, not a silent edit to a fixture file nobody reviews.

None of this requires exotic tooling. It's the same discipline as any other test suite: keep it fast, keep it meaningful, keep the noise down, and make sure a red build actually means something is wrong.

Closing

The mechanics in this post — the workflow YAML, the caching keys, the PR comment bot, the retry wrapper — are all in service of one idea: LLM quality checks only work if they run on every PR without anyone having to remember. Wire DeepEval into GitHub Actions once, and every prompt change, every model swap, every retrieval tweak gets checked against the same bar automatically.

The part that's easy to get wrong is treating this like ordinary software testing, where correctness is binary. It isn't. Under the hood, most of what you're running is LLM-as-a-Judge: one model scoring another model's output against a rubric. That's powerful, but it inherits the judge's own blind spots and variance, which is exactly why the caching, temperature, and quarantine patterns above matter as much as the workflow file itself. Get the judge disciplined, and the CI pipeline becomes something you can actually trust.