Regression Evals in CI: Catching Model Drift

Regression evals in CI are automated checks that run your LLM application against a fixed set of test cases on every pull request, catching quality drops before they reach production. If you have ever shipped a "small" prompt edit that broke a downstream feature, or watched an underlying model upgrade change your outputs without warning, you already know why this matters. Regression evals turn that guesswork into a pass/fail gate, the same way unit tests do for regular code.

This article walks through building a regression eval suite for an LLM feature, wiring it into GitHub Actions (or any CI system), and setting thresholds that keep the pipeline useful instead of noisy.

Why LLM outputs drift even when you change nothing

Traditional software is deterministic: same input, same output, until someone changes the code. LLM applications are not that stable, for a few reasons:

• Model updates. Providers periodically update a model behind a stable-looking name, or you bump a version pin. Anthropic, OpenAI, and others version their models explicitly (for example claude-opus-4-6 vs a newer point release), but teams still get surprised when a "safe" upgrade changes formatting, tone, or reasoning depth.
• Prompt edits. A one-line change to a system prompt, meant to fix bug A, can silently regress behavior B. Prompts do not have a compiler to tell you what broke.
• Retrieval or tool changes. If your app is RAG-based or tool-using, changing the retriever, the chunking strategy, or a tool's return schema shifts the inputs the model sees, which shifts outputs.
• Temperature and sampling. Non-zero temperature means the same prompt can produce different completions run to run, which makes naive "diff the output" testing unreliable.

Regression evals in CI exist to catch all four categories before they reach a real user, using the same trigger point where you already catch code bugs: the pull request.

The anatomy of a regression eval suite

A regression eval suite for CI has four parts:

1. A frozen test set. A list of representative inputs, ideally pulled from real production traffic (anonymized) plus deliberately adversarial edge cases.
2. Expected behavior per case. Not always an exact string match. This can be a reference answer, a rubric, a set of required facts, a JSON schema, or a comparison against a "golden" model.
3. A scoring function. Deterministic checks (schema validation, keyword presence, regex) where possible, and an LLM-as-judge or embedding-similarity score where subjective quality matters.
4. A CI job that runs the suite on every PR, compares the score to a baseline, and fails the build if quality drops past a threshold.

Here is a minimal directory layout that works for most teams:

evals/
  cases/
    summarization_001.json
    summarization_002.json
    support_reply_001.json
  runner.py
  scorer.py
  baseline_scores.json

Each case file holds an input, optional reference output, and metadata:

{
  "id": "support_reply_001",
  "input": "Customer asks for a refund on an order placed 40 days ago, outside the 30-day window.",
  "must_include": ["30-day", "policy"],
  "must_not_include": ["I cannot help you"],
  "category": "policy_adherence"
}

Writing the eval runner

The runner loads every case, calls your app (not the raw model, the actual pipeline including prompts and tools), and scores the result. Keep the runner provider-agnostic so swapping models later does not mean rewriting your evals.

import json
import glob
from pathlib import Path
from your_app import run_pipeline  # the actual function your product calls

def load_cases(cases_dir="evals/cases"):
    cases = []
    for path in glob.glob(f"{cases_dir}/*.json"):
        cases.append(json.loads(Path(path).read_text()))
    return cases

def run_case(case):
    output = run_pipeline(case["input"])
    return {"id": case["id"], "output": output}

def run_all():
    results = []
    for case in load_cases():
        results.append({**case, "output": run_case(case)["output"]})
    return results

if __name__ == "__main__":
    results = run_all()
    Path("evals/last_run.json").write_text(json.dumps(results, indent=2))

Scoring: deterministic first, LLM judge second

Deterministic checks are cheap, fast, and have zero variance, so run them first and only fall back to a judge model when the check is genuinely subjective.

def score_deterministic(case, output):
    score = 1.0
    for term in case.get("must_include", []):
        if term.lower() not in output.lower():
            score -= 0.5
    for term in case.get("must_not_include", []):
        if term.lower() in output.lower():
            score -= 1.0
    return max(score, 0.0)

For cases where "must include the word policy" is not enough, use an LLM-as-judge with a strict rubric and a low temperature so the judge itself is stable:

JUDGE_PROMPT = """You are grading a customer support reply for policy adherence.

Reply: {output}

Score from 0 to 1 on:
- Does it correctly state the refund policy?
- Is the tone professional and non-dismissive?
- Does it offer a next step (escalation, exception review, etc.)?

Return only a JSON object: {{"score": <float 0-1>, "reason": "<one sentence>"}}
"""

def score_with_judge(client, output):
    response = client.messages.create(
        model="claude-opus-4-6",
        max_tokens=200,
        temperature=0,
        messages=[{"role": "user", "content": JUDGE_PROMPT.format(output=output)}],
    )
    return json.loads(response.content[0].text)

Two practical notes on LLM-as-judge:

• Use a different, stronger model for judging than the one you are testing, when budget allows. Grading your own homework with the same model invites blind spots.
• Pin the judge model version explicitly. If the judge itself drifts, your regression suite becomes unreliable in the opposite direction: green builds that should be red.

Wiring it into CI

The CI job needs three things: run the suite, compare to baseline, fail on regression. Here is a GitHub Actions workflow:

name: llm-regression-evals

on:
  pull_request:
    paths:
      - "prompts/**"
      - "src/pipeline/**"
      - "evals/**"

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run eval suite
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: python evals/runner.py

      - name: Score and compare to baseline
        run: python evals/compare_to_baseline.py

      - name: Upload eval results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-results
          path: evals/last_run.json

The paths filter matters: you only want to burn API calls and CI minutes when a prompt, pipeline, or eval case actually changed, not on every unrelated commit.

Setting the pass/fail threshold

This is where most teams get it wrong in one of two directions. Too strict, and the eval suite blocks unrelated PRs on noise from model sampling variance. Too loose, and it never catches anything.

A workable pattern:

• Run each case 3 times and average the score, to smooth out sampling noise (even at temperature 0, some providers are not perfectly deterministic).
• Compare the average score per category (not per individual case) against a stored baseline.
• Fail the build only if a category's average drops by more than a set margin, for example 10 percentage points, or if any must_not_include hard-fail case regresses at all.

def compare_to_baseline(results, baseline, hard_fail_margin=0.10):
    by_category = {}
    for r in results:
        by_category.setdefault(r["category"], []).append(r["score"])

    failures = []
    for category, scores in by_category.items():
        avg = sum(scores) / len(scores)
        base_avg = baseline.get(category, avg)
        if avg < base_avg - hard_fail_margin:
            failures.append(f"{category}: {avg:.2f} vs baseline {base_avg:.2f}")

    return failures

Hard-fail cases (things like "must never include profanity" or "must never leak the system prompt") should not use averaging at all. A single failure on those should block the merge outright, since they represent safety or correctness invariants, not fuzzy quality.

Updating the baseline deliberately

Baselines should not silently ratchet down every time a PR merges, or the whole suite becomes meaningless over a few months. Treat baseline_scores.json like a dependency lockfile:

• Only update it in a dedicated PR, reviewed by a human, with a note explaining why scores changed (intentional prompt improvement, accepted model upgrade, new stricter rubric).
• Never let the CI job auto-write the new baseline on merge. That defeats the purpose, since a real regression would just become the new normal.

# Regenerate baseline after a reviewed, intentional change
python evals/runner.py
python evals/compute_baseline.py > evals/baseline_scores.json
git add evals/baseline_scores.json
git commit -m "update eval baseline after prompt v3 rollout"

Catching model drift specifically

Prompt regressions are the easy case, since you control when the prompt changes. Model drift is sneakier because the model can change "underneath" you if you point at a rolling alias instead of a pinned version.

Two habits fix most of this:

• Pin exact model versions in production, not aliases like "latest." Bump them deliberately, through the same PR-and-eval-suite process as a prompt change.
• Run the eval suite on a schedule, not just on PRs, against your pinned model, as a canary. If scores drop between Tuesday and Wednesday with no code change, either the provider changed something server-side or your test data source (search index, cached tool responses) shifted.

on:
  pull_request:
    paths: ["prompts/**", "src/pipeline/**"]
  schedule:
    - cron: "0 6 * * *"  # daily canary run against pinned model

Alert on the schedule job separately from the PR job, since a canary failure means "something in the world changed," not "this PR is bad," and it needs a different response (open an incident, not block a merge).

Keeping the suite maintainable

A regression eval suite that nobody trusts gets ignored, which is worse than not having one. A few rules keep it healthy:

• Grow the case set from real failures. Every production bug report or bad output a user flags becomes a new eval case, so the suite never stops learning from reality.
• Keep the suite fast enough to run on every PR. If it takes 40 minutes, people will start merging without waiting for it. Parallelize case execution and cache judge calls where inputs are unchanged.
• Review eval case additions like code. A case with a wrong reference answer teaches your CI job to reward the wrong behavior.
• Separate flaky cases from the gate. If a case is genuinely ambiguous and flips pass/fail with no underlying change, move it to a "monitoring" tier that reports but does not block, until you can rewrite it to be deterministic.

FAQ

Do I need an LLM-as-judge, or are deterministic checks enough? Deterministic checks (schema validation, required keywords, forbidden phrases, length bounds) should always run first because they are cheap and stable. Use an LLM judge only for the subjective slice, like tone or reasoning quality, that a regex genuinely cannot capture.

How many test cases does a regression suite need to be useful? Fewer than you think to start. Fifty well-chosen cases covering your main user intents and known failure modes catch most regressions. Grow the set from real production incidents rather than trying to write a comprehensive suite upfront.

Should regression evals block every PR or just prompt changes? Scope the CI trigger with a path filter to prompts, pipeline code, and the eval cases themselves. Running full evals on every unrelated change (a CSS fix, a typo in a README) wastes API budget and slows down unrelated work.

What is the difference between a regression eval and an offline benchmark? A benchmark measures absolute quality against a public dataset. A regression eval measures relative change against your own baseline on your own use cases. You want both, but only the regression suite belongs in CI, since benchmarks are usually too slow and too generic to gate a PR.

How do I handle non-determinism from temperature above zero? Run each case multiple times and average, or force temperature to 0 for the eval run specifically even if production uses a higher temperature. If your feature depends on temperature-driven variety, average over more samples (5 to 10) rather than trying to force determinism that does not reflect production behavior.

Can I use the same model as both the system under test and the judge? You can, but it is weaker. A model tends to rate its own outputs more favorably and misses its own blind spots. Where budget allows, use a different or stronger model as the judge, and always pin the judge's version so its behavior does not drift independently of the system you are testing.