How to Build an Eval Pipeline That Blocks Bad Deploys in CI/CD

Someone on your team tweaked a system prompt at 4:47 PM on a Friday. It passed a quick manual check in the playground — looked fine, tone was right, answer was correct. It merged. By Monday morning, support tickets are up 40% because the model started hedging every answer with three paragraphs of caveats, and nobody caught it because nobody was watching. This is not a hypothetical. This is what happens in every team that treats prompts as configuration instead of code. If you would never merge a backend change without tests, you should not merge a prompt change without evals. The good news is that the infrastructure to prevent this is not exotic — it is the same CI/CD discipline you already use for application code, adapted for a system that is probabilistic instead of deterministic. This article walks through building that pipeline end to end.

Why prompt changes deserve the same rigor as code changes

A prompt is not a comment or a config value that only affects wording. It is an executable artifact that determines control flow, tool-calling behavior, refusal rates, latency, and cost. Changing a prompt can silently break function-calling schemas, change JSON output structure that downstream code parses, or shift the model's tone in a way that violates brand guidelines. None of that shows up in a git diff as an obvious bug — it shows up in production, in user complaints, or in a spike in support escalations.

The core problem is that prompt engineering has historically been treated as a creative, manual process: someone edits text in a playground, eyeballs three or four outputs, decides it "feels better," and ships it. That workflow does not scale past a single engineer, and it does not survive contact with a team. The moment more than one person touches your prompts, or your product has more than a handful of real-world edge cases, manual eyeballing stops catching regressions.

Treat every prompt, every model swap, and every RAG retrieval change as a code change because functionally it is one. That means:

• It goes through a pull request.
• It gets reviewed by a teammate.
• It runs through an automated test suite before merge.
• It has a rollback plan.

The test suite is the part most teams skip, and it is the part that actually prevents incidents. An eval suite is just a regression test suite where the assertions are fuzzier than assertEquals, but the discipline is identical: define expected behavior, run it automatically, block the merge if behavior drifts.

Building a regression test suite: golden examples and edge cases

Your eval suite needs two categories of test cases, and conflating them is the most common mistake teams make.

Golden examples are your happy-path cases — the 20 to 50 inputs that represent the core, expected use of your feature. If you are building a support-ticket classifier, golden examples are clearly-worded tickets that unambiguously belong to one category. These tests answer the question: "does the basic thing still work?" They should almost never fail. If a golden example starts failing, that is a five-alarm fire, not a nitpick.

Edge cases are where regressions actually hide. These are the inputs that are ambiguous, adversarial, or historically troublesome:

• Inputs that previously caused hallucinations or made-up citations.
• Prompt-injection attempts embedded in user content or retrieved documents.
• Multilingual or code-switched inputs if your product supports them.
• Extremely short inputs ("help") and extremely long ones that approach context limits.
• Inputs that are almost, but not quite, in-scope — designed to test whether the model over-triggers a tool call or refuses when it shouldn't.
• Every past production incident, converted into a test case the moment it is fixed.

That last point matters more than anything else in this section. Every time a user reports a bad output, or you catch one in a support ticket, that exact input becomes a permanent regression test. This is how your eval suite compounds in value over time instead of staying static. Six months in, a mature eval suite is mostly made of scar tissue from real incidents, and that is exactly what makes it valuable — it encodes institutional memory that would otherwise live in someone's head.

A practical target for a first suite: 30 golden examples, 40 edge cases, 10 adversarial/injection cases. That is enough to catch the categories of regression that actually happen, without being so large that CI runtime and cost become unmanageable.

Structuring test cases so they are gradeable

Each test case needs three things: an input, an expected behavior (not necessarily an expected exact string), and a grading method. There are three grading methods you will mix across your suite:

• Exact match / structural checks — for outputs where correctness is unambiguous, like JSON schema conformance, whether a required field is present, or whether a specific tool was called. Cheap, deterministic, fast. Use these wherever you possibly can.
• Rule-based / programmatic checks — regex for banned phrases, length bounds, checking that a citation URL actually appears in the retrieved context (a basic hallucination guard), checking that PII patterns are not present in output.
• LLM-as-a-judge — for open-ended quality dimensions like tone, helpfulness, faithfulness to source material, or whether a response appropriately declines an out-of-scope request. This is the most flexible grading method and the one teams misuse most often, which is why understanding how to build a reliable judge is its own discipline (more on that below).

Store test cases as data, not as code. A YAML or JSON file per test case (or a single structured file with an array of cases) keeps your suite reviewable in a pull request and lets non-engineers on your team contribute edge cases without touching Python.

- id: refund-policy-ambiguous-001
  input: "can I get my money back if I just changed my mind"
  category: edge_case
  checks:
    - type: must_not_contain
      values: ["I don't know", "I cannot help"]
    - type: must_call_tool
      tool_name: lookup_refund_policy
    - type: llm_judge
      rubric: "Response correctly states the 30-day window and does not promise a refund outright."
      pass_threshold: 0.8
  tags: [refunds, policy]

Setting pass/fail thresholds that actually mean something

The single biggest mistake teams make when they first wire up evals is requiring 100% pass rate and then either ignoring the suite entirely because it never goes green, or disabling half the tests to make it pass. Neither is acceptable. The fix is tiered thresholds tied to the blast radius of the test category:

• Golden examples: 100% required to merge. These represent your core contract with users. If any golden example fails, the PR is blocked, full stop, no exceptions, no "it's probably fine."
• Edge cases: 90-95% required to merge, with the specific failures visible in the CI output so a human can eyeball whether the 5-10% that failed are truly regressions or borderline judgment calls where the bar itself may need updating.
• Adversarial/injection cases: 100% required. A prompt-injection or safety bypass is not a quality nitpick — treat any regression here as equivalent to a security vulnerability.
• LLM-as-a-judge scores: aggregate score threshold, not per-case. Individual judge scores are noisy. Instead of blocking on any single judge call scoring 0.79 instead of 0.8, block on the suite-wide average dropping more than a set delta (for example, more than 3 percentage points) versus the baseline on main. This absorbs judge noise while still catching systemic quality drops.

Critically, thresholds are not "good enough forever" — they are a floor that should ratchet upward over time. When you fix a bug and add a regression test, and your current suite average is 88%, don't leave the threshold at 85% out of caution. Move it to 87% or 88%. The threshold should track your actual measured quality, trailing it slightly, so improvements get locked in and cannot silently regress later.

Wiring evals into GitHub Actions

The mechanics of CI integration are straightforward once you accept that an eval run is just another test step, except it calls an LLM API instead of running pure functions. The wrinkle is secrets management (API keys), cost control (discussed in the next section), and making results visible to reviewers directly in the PR rather than buried in logs.

A minimal workflow triggers on pull requests that touch your prompts directory or eval-relevant code, runs the eval suite, and fails the build if thresholds are not met.

name: llm-eval-gate

on:
  pull_request:
    paths:
      - "prompts/**"
      - "src/agents/**"
      - "evals/**"

jobs:
  run-evals:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r requirements-eval.txt

      - name: Restore eval cache
        uses: actions/cache@v4
        with:
          path: .eval_cache
          key: eval-cache-${{ github.event.pull_request.base.sha }}

      - name: Run golden example suite
        run: python -m evals.run --suite golden --fail-under 1.0
        env:
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
          EVAL_TEMPERATURE: "0"

      - name: Run edge case suite
        run: python -m evals.run --suite edge_cases --fail-under 0.90
        env:
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
          EVAL_TEMPERATURE: "0"

      - name: Run adversarial suite
        run: python -m evals.run --suite adversarial --fail-under 1.0
        env:
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
          EVAL_TEMPERATURE: "0"

      - name: Compare judge score against baseline
        run: python -m evals.compare_baseline --max-regression 0.03

      - name: Publish results to PR
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const summary = fs.readFileSync('eval_summary.md', 'utf8');
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: summary
            });

A few details that matter more than they look like they do:

• Path filtering keeps the eval job from running (and costing money) on PRs that only touch unrelated code, like CSS or documentation.
• Splitting into separate steps per suite means a failure in the adversarial suite shows up distinctly from a failure in golden examples, instead of one opaque "eval job failed" status check that a reviewer has to dig into.
• Posting a summary comment on the PR is what actually makes this land as a team habit. If results only exist in a CI log nobody clicks into, people learn to ignore red checks. If a bot comments a readable table showing exactly which cases regressed and why, reviewers engage with it.
• Branch protection rules should mark the eval job as a required check on your main branch, the same way you require unit tests to pass. Without this, the pipeline is advisory, and advisory checks get ignored the first time someone is in a hurry.

Handling flaky and non-deterministic outputs

This is the part that trips up engineers coming from traditional software testing, where a test either passes or it doesn't, every time, forever. LLMs do not give you that guarantee, and pretending otherwise produces a CI pipeline that cries wolf so often people stop trusting it.

Set temperature to 0 for all eval runs. This does not make outputs perfectly deterministic — most model providers still have some residual variance from batching, floating-point non-associativity across hardware, or internal sampling even at temperature 0 — but it removes the largest and most controllable source of randomness. There is no good reason to eval at any temperature other than 0 unless you are specifically testing how your product behaves at production temperature settings, which should be a separate, smaller suite.

Build retries with majority voting into the harness, not into your patience. If a single test case output is checked once and happens to land on the unlucky side of residual variance, you get a flaky red build and an engineer who reruns CI three times until it goes green, which defeats the entire point. Instead, run flaky-prone checks (particularly LLM-as-a-judge checks) three times and take the majority verdict or the median score.

import statistics
from your_llm_client import call_model, judge_score

def run_case_with_tolerance(test_case, n_samples=3, tolerance=0.05):
    """
    Runs a test case multiple times at temperature=0 to smooth out
    residual non-determinism, and applies a tolerance band instead
    of a hard equality check.
    """
    scores = []
    outputs = []

    for _ in range(n_samples):
        output = call_model(
            prompt=test_case.render_prompt(),
            temperature=0,
        )
        outputs.append(output)
        score = judge_score(
            output=output,
            rubric=test_case.rubric,
            reference=test_case.expected,
        )
        scores.append(score)

    median_score = statistics.median(scores)
    score_spread = max(scores) - min(scores)

    passed = median_score >= test_case.pass_threshold

    # Flag high-variance cases even if they technically pass —
    # this is often an early signal of prompt ambiguity.
    is_unstable = score_spread > tolerance

    return {
        "case_id": test_case.id,
        "passed": passed,
        "median_score": median_score,
        "score_spread": score_spread,
        "flagged_unstable": is_unstable,
        "outputs": outputs,
    }


def test_refund_policy_ambiguous_case(refund_test_case):
    result = run_case_with_tolerance(refund_test_case)
    assert result["passed"], (
        f"Case {result['case_id']} failed: "
        f"median score {result['median_score']:.2f} "
        f"below threshold {refund_test_case.pass_threshold}"
    )
    if result["flagged_unstable"]:
        print(
            f"WARNING: {result['case_id']} has high variance "
            f"({result['score_spread']:.2f}) across identical runs — "
            f"consider tightening the prompt or rubric."
        )

Use tolerance bands, not exact equality, for anything graded on a numeric or continuous scale. A judge score of 0.82 versus a baseline of 0.85 is not a regression — it is noise. Decide on a tolerance band up front (commonly 3-5 percentage points for aggregate judge scores) and only fail the build when a case or suite average moves outside that band, not on every single-point fluctuation.

Pin model versions explicitly in your eval config. Silent model upgrades from your provider are one of the most common causes of "nothing changed but evals started failing" incidents. Your CI should call a pinned model snapshot, and upgrading that pin should itself be a reviewed PR that runs the full eval suite, exactly like upgrading a critical dependency.

Cost management for running evals on every PR

Running a full eval suite against a live LLM API on every single PR adds up fast, especially once your suite grows past a hundred cases and your team is merging multiple times a day. A few controls keep this from becoming a surprise line item.

• Cache aggressively and key on prompt content, not just input. If a PR does not change a given prompt template or the code path that constructs it, there is no reason to re-run those specific test cases. Hash the fully-rendered prompt plus model version plus temperature, and skip the API call entirely on a cache hit from a previous run against the same base commit.
• Tier your suites by trigger, not just by content. Run golden examples and adversarial cases on every PR — they are small and cheap. Run the full edge-case suite (and any expensive multi-turn or agentic eval scenarios) on every PR too, but reserve larger stress-test batches, wider parameter sweeps, or full multilingual sweeps for a nightly scheduled run or a pre-release gate instead of every single commit.
• Use a cheaper model for judge calls where the rubric allows it. Not every judgment call needs your most expensive model. Simple binary checks (did the response contain a refusal, does the JSON parse) can often be handled by a smaller, faster model as the judge, reserving the top-tier model as judge only for nuanced quality dimensions like faithfulness or tone.
• Batch API calls where your provider supports it. Batch endpoints are frequently priced significantly lower than synchronous calls and are well suited to eval runs, which are not latency-sensitive the way production traffic is.
• Set a hard budget alert. Track monthly eval spend as its own line item and alert when it crosses a threshold, the same way you would alert on a runaway cloud bill. Eval cost creep is easy to miss because it is distributed across dozens of small PR-triggered runs rather than one obvious bill.
• Right-size sample counts. The retry-and-majority-vote pattern above triples cost for flaky-prone cases. Reserve n_samples=3 for cases that have historically shown variance, and run the rest of your suite at n_samples=1. Don't apply the expensive pattern uniformly across a suite where most cases are already stable at a single sample.

Alerting on regressions

CI gating catches problems before merge. You also need a second layer that watches production and post-merge trends, because some regressions only show up under real traffic patterns your test suite doesn't anticipate, and because prompt behavior can drift even without a prompt change — a model provider ships a silent update, a retrieval index changes, an upstream API you depend on changes its response shape.

• Track eval scores over time on `main`, not just pass/fail per PR. A dashboard showing golden-example pass rate, edge-case pass rate, and judge-score trend over the last 30 days will show slow drift that no single PR would trigger a block for. A steady 1% weekly decline is invisible in any individual CI run but obvious on a trend line.
• Alert on any golden-example failure post-merge, immediately, to a channel someone actually watches. If your gating is solid this should rarely fire, but "rarely" is not "never" — model provider updates and dependency changes can break things between PRs, not just within them.
• Sample production traffic and replay it against your eval judge on a schedule. Your test suite is a proxy for reality, not reality itself. Periodically pulling a random sample of real production inputs (scrubbed of PII) and scoring them with the same judge rubric catches the gap between what you tested and what users are actually doing.
• Set up automatic regression-to-test-case conversion where you can. When your judge or your monitoring flags a production output as low quality, route it into a review queue, and if a human confirms it is a genuine bad output, that queue should be one click away from becoming a new permanent test case, not a Slack message that gets forgotten in three days.
• Alert on cost and latency regressions alongside quality. A prompt change that maintains quality but doubles token usage or adds two seconds of latency is still a regression worth blocking or at minimum flagging for discussion — it just isn't a correctness regression, and your alerting should distinguish the two.

Rolling this out without stalling your team

If you try to build the full version of this — hundreds of test cases, a tuned LLM-as-a-judge, tiered thresholds, cost dashboards, production sampling — before shipping anything, you will spend a month building infrastructure and ship zero of it, because there is always one more edge case to add before it feels "done." Start smaller and let the suite grow with real incidents.

1. Write 10 golden examples for your single most critical user flow. Get them running in CI with a 100% threshold. Ship that this week.
2. Add a required GitHub Actions check gating merges to your prompts directory, even with just those 10 cases.
3. The next time something breaks in production, that broken case becomes test case number 11, permanently. Repeat.
4. Once you have 30-40 cases and the suite has caught at least one real regression before merge, invest in the LLM-as-a-judge layer, tolerance bands, and cost tooling described above.

The pipeline earns trust the first time it blocks a PR that would have shipped a real bug. After that, nobody on the team needs convincing anymore.

Building a reliable judge model, calibrating rubrics so they agree with human raters, and designing the scoring architecture that holds up as your product scales past a handful of prompts — that is a deeper discipline than a single article can cover, and it is exactly what we teach hands-on in the LLM-as-a-Judge course at teachyou.ai, taught by Pramod Dutta and Ira Menon.