CI/CD for LLM Testing: Building Pipelines That Catch Regressions Before Users Do

CI/CD for LLM testing means running automated evaluations against every prompt, model, or retrieval change before it reaches production, the same discipline you already apply to unit tests, just adapted for outputs that are probabilistic instead of deterministic. If you are shipping an LLM feature without a pipeline that scores outputs on every pull request, you are relying on manual spot checks and vibes, and vibes do not scale past your third prompt change. This guide walks through building that pipeline end to end: what to test, how to structure the pipeline stages, which tools to reach for, and how to keep flaky non-deterministic tests from blocking every merge.

Why LLM testing breaks traditional CI/CD assumptions

Traditional CI/CD assumes a test either passes or fails. Run the same input twice, get the same output, done. LLMs violate that assumption at the root: temperature above zero means two runs of the same prompt can produce different tokens, different lengths, and occasionally different conclusions. A pipeline built for deterministic code will either report constant false failures or, worse, get disabled by a frustrated team within a month.

The fix is not to abandon CI/CD for LLM features, it is to change what "pass" means. Instead of exact-match assertions, LLM pipelines score outputs against rubrics, similarity thresholds, and structural checks, then gate merges on aggregate pass rates rather than single-run pass/fail. A prompt change that drops your eval suite from 94% to 81% is a regression even though no individual test is "broken" in the traditional sense.

Three categories of things typically break in LLM apps, and your pipeline needs to catch all three:

• Prompt regressions: someone edits a system prompt to fix one failure mode and silently breaks three others.
• Model drift: the underlying model provider ships a new version behind the same API alias, and behavior shifts without a code change on your side.
• Pipeline regressions: a change to chunking, retrieval, or tool schemas degrades answer quality even though the prompt itself is untouched.

Designing the test pyramid for LLM applications

Borrow the shape of the classic testing pyramid, but change what lives at each layer.

Layer 1: Deterministic unit tests. These check things that should never vary regardless of model output: does the API return valid JSON matching your schema, does the function-calling response include required fields, does the retrieval step return at least one document, does the token count stay under budget. These run in seconds, cost nothing (no LLM call needed if you are testing parsing logic against fixture responses), and should run on every commit.

Layer 2: Assertion-based output checks. These call the real model with a fixed prompt and check the output against programmatic assertions: contains a keyword, matches a regex, passes a JSON schema, stays under a length limit, does not contain banned phrases. This is where tools like promptfoo and custom pytest suites live. Cheap enough to run on every pull request if you cap the eval set to 20-50 representative cases.

Layer 3: Model-graded evals (LLM-as-judge). For open-ended quality, a second LLM call scores the first model's output against a rubric: relevance, faithfulness to retrieved context, tone, refusal correctness. Slower and costlier than Layer 2, so these usually run on merge to main or nightly rather than on every PR push.

Layer 4: Golden dataset regression suite. A larger, curated set of real (or synthetic) production-like inputs with known-good reference answers, scored with a mix of exact match, semantic similarity, and LLM-as-judge. This is your safety net for catching slow drift over weeks, run nightly or on a schedule, not on every commit, because it is expensive and slow.

Structuring your suite this way keeps your PR feedback loop fast (layers 1-2, under two minutes) while still catching subtle quality regressions before they compound (layers 3-4, on a schedule).

Setting up the pipeline: a working example

Here is a concrete GitHub Actions setup that implements the four-layer pyramid for a Python-based LLM app. Assume you are using pytest for structure and promptfoo for assertion-based prompt evals.

First, the fast layer that runs on every push:

name: llm-fast-checks
on:
  pull_request:
    branches: [main]

jobs:
  unit-and-assertions:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          npm install -g promptfoo

      - name: Run deterministic unit tests
        run: pytest tests/unit -v

      - name: Run assertion-based prompt evals
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: promptfoo eval -c promptfooconfig.yaml --max-concurrency 4

      - name: Fail if pass rate drops below threshold
        run: python scripts/check_eval_threshold.py --min-pass-rate 0.90

The promptfooconfig.yaml file drives layer 2. A minimal example testing a customer support assistant prompt:

prompts:
  - "prompts/support_agent.txt"
providers:
  - anthropic:claude-sonnet-4
tests:
  - vars:
      question: "How do I reset my password?"
    assert:
      - type: contains
        value: "reset"
      - type: not-contains
        value: "I don't know"
      - type: latency
        threshold: 5000
  - vars:
      question: "Can you process a refund for order 12345?"
    assert:
      - type: llm-rubric
        value: "Response correctly states it cannot process refunds directly and routes to the refunds team"

promptfoo supports contains, regex, javascript (custom scoring functions), similar (embedding-based semantic match), and llm-rubric (model-graded) assertion types out of the box, which covers layers 2 and 3 in a single config file. That is the single biggest reason to reach for it before writing bespoke evaluation harnesses: you get assertion diversity without owning the scoring code.

The nightly regression job

Layer 4 does not belong on your PR critical path. Wire it as a scheduled workflow instead:

name: llm-nightly-regression
on:
  schedule:
    - cron: "0 3 * * *"
  workflow_dispatch: {}

jobs:
  golden-dataset-eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run golden dataset eval
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: python scripts/run_golden_eval.py --dataset data/golden_set_v3.jsonl --output results/nightly-$(date +%F).json

      - name: Compare against baseline
        run: python scripts/compare_to_baseline.py --current results/nightly-$(date +%F).json --baseline results/baseline.json --fail-on-regression 0.03

      - name: Post results to Slack
        if: always()
        run: python scripts/notify_slack.py --webhook ${{ secrets.SLACK_WEBHOOK }} --results results/nightly-$(date +%F).json

The compare_to_baseline.py script is the piece teams usually underbuild. It should not just check "did the average score drop," it should flag per-category regressions: if your golden set is tagged by intent (billing, technical support, general question), a 5-point drop concentrated entirely in "billing" questions is a more actionable signal than a 1-point drop spread evenly, and it points you straight at the retrieval index or prompt section that changed.

Handling non-determinism without disabling your pipeline

The most common failure mode teams hit is treating LLM tests like flaky UI tests: retry three times, ignore intermittent failures, eventually stop trusting the pipeline. Do this instead:

• Set temperature to 0 for assertion-based tests wherever the task allows it. Structured extraction, classification, and tool-calling tests should be near-deterministic at temperature 0; if they still flap, that is a signal the prompt itself is ambiguous, not a test infrastructure problem.
• Use pass-rate thresholds, not per-test pass/fail, for model-graded evals. Run each rubric-graded test 3-5 times and require a majority pass rather than a single-shot pass. This costs more API calls but eliminates single-sample noise.
• Separate "must never happen" assertions from "should generally happen" assertions. A response that leaks a system prompt or produces invalid JSON is a hard fail every time. A response that scores 7/10 instead of 9/10 on a tone rubric is a soft signal that should count toward an aggregate score, not block the merge on its own.
• Pin model versions in CI, not just in production. If your provider allows pinning a snapshot (rather than a rolling alias), pin it in your eval config so a provider-side model update does not silently change your baseline mid-sprint. Track the pinned version in a changelog so you know when to re-run the full golden set against a newer snapshot deliberately.

Cost control: do not let evals become your biggest API bill

A golden set of 500 examples run three times per rubric assertion against a nightly schedule adds up fast, especially once you have five or six rubric checks per example. A few practical guardrails:

• Run the full golden set nightly, but run a stratified 10% sample on every PR so contributors still get signal fast without burning the full budget on every push.
• Cache assertion-based test results keyed by a hash of the prompt template plus model version; skip re-running a test if neither changed since the last cached run.
• Use a cheaper model as the judge for layer 3 where the rubric is simple (tone, format compliance) and reserve the most capable judge model for layer 4's faithfulness and correctness checks, where nuance actually matters.
• Set a hard budget alert on your eval-specific API key so a runaway workflow_dispatch loop or a misconfigured cron does not surprise you on the monthly invoice.

Blocking merges: what threshold actually works

Teams that set the bar too high (100% pass required) end up disabling the check within weeks because any single flaky rubric result blocks unrelated PRs. Teams that set it too low (any pass rate accepted) get no protection at all. A workable middle ground:

• Hard gate: schema validation, tool-call correctness, and safety/refusal tests must pass 100% of the time. Zero tolerance, these are deterministic enough to enforce strictly.
• Soft gate with threshold: assertion and rubric-based quality scores must stay within 3-5 percentage points of the current baseline on main. A drop bigger than that blocks merge with a required manual override from a reviewer who has seen the eval diff.
• Informational only: golden dataset trend lines get posted to a dashboard or Slack channel but do not block anything in real time. Humans review weekly and decide whether a drift is acceptable or needs a prompt fix.

Encode the baseline as a committed file (results/baseline.json) that only updates via a deliberate PR, not automatically after every merge. That single decision, treating your eval baseline like a versioned artifact instead of a moving target, is what keeps the whole system honest. If baselines auto-update on every green build, a slow accumulation of small regressions becomes invisible because each individual PR looks fine relative to the (already degraded) baseline it inherited.

Tooling landscape worth knowing

• promptfoo: config-driven prompt testing with built-in assertion types, works well as the layer 2/3 workhorse and has first-class CI integration.
• DeepEval: a pytest-native framework for LLM evals, good fit if your team already lives in pytest and wants eval assertions as native test functions rather than a separate YAML config.
• Braintrust and LangSmith: hosted eval and observability platforms that pair CI-triggered eval runs with a dashboard for tracing individual failures back to specific prompt or retrieval changes, useful once your golden set outgrows a spreadsheet mental model.
• Ragas: purpose-built for RAG pipelines, scoring faithfulness, context precision, and context recall separately, which is more diagnostic than a single blended quality score when your regression is actually a retrieval problem wearing a generation-quality costume.

Pick one tool to own layer 2/3 rather than stitching together three different eval libraries; the config sprawl costs more engineering time than the marginal feature gap between any two of these tools.

Putting it together: a rollout sequence

1. Start with layer 1 (deterministic unit tests on parsing, schema, and tool-call structure) and get that green on every PR this week. This alone catches a surprising fraction of real incidents.
2. Add a 20-30 case assertion suite with promptfoo for your two or three highest-traffic prompts. Do not try to cover everything on day one.
3. Wire the fast suite into a required PR check with a generous threshold (85-90% pass rate) so it does not immediately block the team, then tighten it over a few weeks as the suite stabilizes.
4. Build the golden dataset from real production logs (anonymized) plus a handful of adversarial cases your team already knows are tricky. Run it nightly, review weekly.
5. Only after the above is stable, add LLM-as-judge rubric scoring for tone and nuance. This is the highest-maintenance layer and pays off least early on, so it belongs last, not first.

FAQ

Do I need a separate eval framework or can I just use pytest? You can absolutely stay in pytest for everything, especially for layers 1 and 2 where assertions are close to regular code assertions. Dedicated eval frameworks like promptfoo or DeepEval earn their keep once you need rubric-based scoring, test-case management across many prompt variants, or a built-in CI reporting format, but a small team can ship a solid pipeline with plain pytest and a JSON fixtures directory for a long time before outgrowing it.

How big should the golden dataset be? Start smaller than you think, 50-100 well-chosen examples that cover your main intents plus known edge cases beats 1,000 randomly sampled production logs with no curation. Grow it deliberately: every real production incident should add at least one regression test case to the golden set so the same failure can never silently ship twice.

Should evals block deploys or just PR merges? Both, but at different strictness. PR merges should gate on the fast layer 1-2 checks with a required threshold. Deploys to production should additionally gate on the most recent nightly golden set run passing within tolerance, since that is your best signal of aggregate quality right before the change reaches real users.

What is the biggest mistake teams make setting this up? Treating every eval failure as equally severe. A pipeline that blocks on any rubric-score dip below the previous run's exact number trains the team to ignore or override the check, which defeats the purpose. Split hard gates (deterministic, zero tolerance) from soft gates (statistical, threshold-based) from day one.

How do I test tool-calling and agentic workflows in CI specifically? Mock the tool execution layer and assert on the tool-call payload the model produces: correct tool name, correct argument schema, correct argument values for a fixed input. Keep a separate, smaller suite that runs against real tool implementations (or sandboxed versions of them) to catch integration-level issues, but do not run real external API calls inside every PR check, that turns your test suite into a dependency on someone else's uptime.

Can I run this pipeline without API keys checked into secrets? Yes for layer 1, since those tests should run against fixture responses rather than live model calls. Layers 2-4 need real API access, so store keys as encrypted repository or organization secrets, scope a dedicated low-limit API key just for CI usage, and rotate it separately from your production key so a leaked CI secret cannot touch production traffic or billing.