DeepEval

Setting Up DeepEval Alerts on Metric Regression

Pramod Dutta · Jun 15, 2026 · 12 min read

Your Eval Suite Is Only As Useful As Its Alerts

You added DeepEval to your CI pipeline three months ago. The team was excited. Every pull request now runs a batch of AnswerRelevancyMetric, FaithfulnessMetric, and a custom rubric check against your RAG pipeline. Green checkmarks everywhere.

Then last week, someone tightened a prompt template to save fifty tokens per call, faithfulness score quietly dropped from 0.91 to 0.74, the test still "passed" because the threshold was 0.7, and nobody noticed until a customer complained the bot was inventing return policy details. The eval suite ran. It just never told anyone anything had changed.

This is the gap between "having evals" and "having an alerting system." A metric score sitting in a CI log is not an alert — it's a fact nobody read. Real regression detection means comparing today's score against a baseline, deciding what magnitude of drop actually matters, and pushing that decision into a channel a human will see within minutes, not during next sprint's retro. This article walks through building that pipeline end to end: capturing baselines, computing regression deltas, wiring DeepEval's native alerting hooks, and routing failures to Slack, email, or a dashboard — with working code at every step.

Why Threshold-Only Checks Miss Regressions

Most teams start with DeepEval the same way: set a metric, set a threshold, assert it passes.

from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

def test_faithfulness():
    metric = FaithfulnessMetric(threshold=0.7, model="gpt-4o-mini")
    test_case = LLMTestCase(
        input="What is the refund window for annual plans?",
        actual_output=generate_answer("What is the refund window for annual plans?"),
        retrieval_context=get_retrieved_chunks("refund window annual plans")
    )
    assert_test(test_case, [metric])

This is a fine starting point, but it has a structural blind spot: a threshold check only tells you pass or fail against a fixed bar. It says nothing about direction or velocity. A score can drift from 0.95 down to 0.71 over six weeks of small prompt tweaks, staying "green" the entire time, until one more edit pushes it to 0.68 and the whole thing looks like a sudden failure. In reality it was regressing continuously and nobody was watching the trend.

Alerting on regression is a different question than alerting on threshold breach. Regression alerting asks: is this score meaningfully worse than our own recent baseline, independent of whether it still clears some absolute bar. That distinction matters because your absolute threshold is usually a business risk tolerance ("don't ship anything below 0.7 faithfulness"), while regression is an engineering signal ("something we just changed made things worse"). You need both, and you need the second one to fire even when the first one doesn't.

Establishing a Metric Baseline

Before you can detect regression, you need a stored baseline to regress against. DeepEval doesn't ship a baseline store out of the box, so we build a thin one — a JSON snapshot of the last known-good scores per metric, per test case, keyed by a stable test ID.

import json
import os
from datetime import datetime, timezone

BASELINE_PATH = "eval_baselines/faithfulness_baseline.json"

def load_baseline():
    if not os.path.exists(BASELINE_PATH):
        return {}
    with open(BASELINE_PATH, "r") as f:
        return json.load(f)

def save_baseline(scores: dict):
    os.makedirs(os.path.dirname(BASELINE_PATH), exist_ok=True)
    payload = {
        "updated_at": datetime.now(timezone.utc).isoformat(),
        "scores": scores
    }
    with open(BASELINE_PATH, "w") as f:
        json.dump(payload, f, indent=2)

The baseline gets refreshed only on merges to your main branch, never on feature branch runs. That's the key rule: feature branches compare against main's last recorded baseline, and only main branch runs are allowed to overwrite it. This prevents a bad PR from resetting the bar for itself.

import subprocess

def is_main_branch() -> bool:
    branch = subprocess.check_output(
        ["git", "rev-parse", "--abbrev-ref", "HEAD"]
    ).decode().strip()
    return branch == "main"

Run this check at the start of your CI eval job and branch your logic accordingly — compare-only on feature branches, compare-and-update on main.

Computing the Regression Delta

With a baseline in hand, the actual regression math is simple arithmetic, but the interesting part is deciding what counts as "regression" versus normal noise. LLM-judge-based metrics like FaithfulnessMetric or AnswerRelevancyMetric are not perfectly deterministic — the same input can score 0.88 one run and 0.84 the next purely from judge-model variance. Alerting on every single-point drop creates so much noise the channel gets muted within a week.

The fix is a two-part rule: an absolute delta floor and a relative percentage floor, both must be crossed before it counts as a regression.

from dataclasses import dataclass

@dataclass
class RegressionResult:
    test_id: str
    metric_name: str
    baseline_score: float
    current_score: float
    delta: float
    is_regression: bool

def check_regression(
    test_id: str,
    metric_name: str,
    baseline_score: float,
    current_score: float,
    absolute_floor: float = 0.05,
    relative_floor: float = 0.08
) -> RegressionResult:
    delta = current_score - baseline_score
    relative_drop = abs(delta) / baseline_score if baseline_score > 0 else 0

    is_regression = (
        delta < 0
        and abs(delta) >= absolute_floor
        and relative_drop >= relative_floor
    )

    return RegressionResult(
        test_id=test_id,
        metric_name=metric_name,
        baseline_score=baseline_score,
        current_score=current_score,
        delta=delta,
        is_regression=is_regression
    )

Tune absolute_floor and relative_floor per metric family. Faithfulness and hallucination-adjacent metrics deserve tighter floors because a drop there is a trust problem, not just a quality nit — I'd run those closer to 0.03/0.05. Something like answer relevancy or a style/tone rubric can tolerate a looser 0.08/0.12 band since judge variance is naturally higher on subjective criteria.

Running Metrics in Bulk and Diffing Against Baseline

Now tie it together: run your DeepEval suite, collect results per test case, diff each against the stored baseline, and produce a regression report.

from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

METRICS = {
    "faithfulness": FaithfulnessMetric(threshold=0.7, model="gpt-4o-mini"),
    "answer_relevancy": AnswerRelevancyMetric(threshold=0.7, model="gpt-4o-mini"),
}

def run_eval_suite(test_cases: list[LLMTestCase]) -> dict:
    results = {}
    for tc in test_cases:
        test_id = tc.input[:60]
        results[test_id] = {}
        for metric_name, metric in METRICS.items():
            metric.measure(tc)
            results[test_id][metric_name] = metric.score
    return results

def diff_against_baseline(current_results: dict, baseline: dict) -> list[RegressionResult]:
    baseline_scores = baseline.get("scores", {})
    regressions = []
    for test_id, metric_scores in current_results.items():
        for metric_name, current_score in metric_scores.items():
            key = f"{test_id}::{metric_name}"
            baseline_score = baseline_scores.get(key)
            if baseline_score is None:
                continue  # no prior baseline, nothing to regress against
            result = check_regression(test_id, metric_name, baseline_score, current_score)
            regressions.append(result)
    return regressions

Note the if baseline_score is None: continue. New test cases have no history yet — they get recorded as the first baseline entry, not flagged as a false regression against a non-existent prior score.

Wiring Alerts to Slack

Once you have a list of RegressionResult objects where is_regression is true, the alert itself is a webhook call. Slack incoming webhooks are the lowest-friction option for a team already living in Slack.

import requests
import os

SLACK_WEBHOOK_URL = os.environ["DEEPEVAL_SLACK_WEBHOOK"]

def send_slack_alert(regressions: list[RegressionResult], run_url: str):
    if not regressions:
        return

    lines = [f"*DeepEval regression detected* — {len(regressions)} metric(s) dropped"]
    for r in regressions:
        lines.append(
            f"- `{r.metric_name}` on test `{r.test_id}`: "
            f"{r.baseline_score:.2f} → {r.current_score:.2f} "
            f"(Δ {r.delta:+.2f})"
        )
    lines.append(f"<{run_url}|View CI run>")

    payload = {"text": "\n".join(lines)}
    response = requests.post(SLACK_WEBHOOK_URL, json=payload, timeout=10)
    response.raise_for_status()

Keep the message body scannable — metric name, baseline, current, delta, and a link to the run. Don't paste full test case inputs and outputs into the Slack message itself; link out to the CI artifact instead, or the channel becomes unreadable after a week of noisy PRs.

Wiring Alerts to Email

Not every team lives in Slack, and some regressions — particularly ones on your main branch, post-merge — deserve an email that lands in an inbox someone actually checks daily, separate from a busy dev channel.

import smtplib
from email.mime.text import MIMEText

def send_email_alert(regressions: list[RegressionResult], run_url: str, recipients: list[str]):
    if not regressions:
        return

    body_lines = [f"DeepEval detected {len(regressions)} metric regression(s) on main.\n"]
    for r in regressions:
        body_lines.append(
            f"{r.metric_name} | test: {r.test_id} | "
            f"{r.baseline_score:.3f} -> {r.current_score:.3f} (delta {r.delta:+.3f})"
        )
    body_lines.append(f"\nRun details: {run_url}")

    msg = MIMEText("\n".join(body_lines))
    msg["Subject"] = f"[ALERT] DeepEval regression — {len(regressions)} metric(s) affected"
    msg["From"] = os.environ["ALERT_FROM_EMAIL"]
    msg["To"] = ", ".join(recipients)

    with smtplib.SMTP(os.environ["SMTP_HOST"], 587) as server:
        server.starttls()
        server.login(os.environ["SMTP_USER"], os.environ["SMTP_PASS"])
        server.send_message(msg)

If you're already running transactional email through a provider like Resend or SES, swap the raw smtplib block for their SDK — the content structure stays identical. The point isn't the transport, it's making sure regression data reaches a channel someone is actually accountable for reading, and that main-branch regressions get a louder signal than feature-branch ones.

Hooking This Into CI

Here's the orchestration script that ties baseline loading, eval running, regression diffing, and alert dispatch into one CI step. This is what your GitHub Actions or GitLab CI job actually invokes.

import sys

def main():
    baseline = load_baseline()
    test_cases = build_test_cases_from_dataset("eval_datasets/support_qa.jsonl")

    current_results = run_eval_suite(test_cases)
    regressions = diff_against_baseline(current_results, baseline)
    failing = [r for r in regressions if r.is_regression]

    run_url = os.environ.get("CI_RUN_URL", "local-run")

    if failing:
        send_slack_alert(failing, run_url)
        if is_main_branch():
            send_email_alert(failing, run_url, ["qa-team@yourcompany.com"])

    if is_main_branch():
        flat_scores = {
            f"{test_id}::{metric_name}": score
            for test_id, metrics in current_results.items()
            for metric_name, score in metrics.items()
        }
        save_baseline(flat_scores)

    if failing:
        print(f"Regression alert sent for {len(failing)} metric(s).")
        sys.exit(1)

    print("No regressions detected.")

if __name__ == "__main__":
    main()

Exiting with status 1 on regression makes the CI job itself fail visibly on the PR, which is a second, independent alert surface beyond Slack and email — reviewers see a red X before they even open the thread. Commit eval_baselines/faithfulness_baseline.json to the repo (or push it to an S3 bucket / artifact store if you'd rather not version-control a scores file) so the baseline persists across CI runs instead of resetting every time.

Choosing What to Alert On, and How Loud

Not every metric deserves the same alert volume. A useful mental model is a three-tier severity system:

Page-worthy: Faithfulness or hallucination-related metrics on production-facing test suites. A regression here means the model is more likely to state something false with confidence. Route to Slack and email, fail the build.
Review-worthy: Answer relevancy, contextual precision/recall on RAG retrieval quality. Route to Slack only, fail the build but don't email — a human reviewing the PR should catch it before merge.
Track-only: Style, tone, verbosity-style custom metrics where judge variance is high and business impact is low. Log to a dashboard or a weekly digest, don't interrupt anyone in real time.

SEVERITY_MAP = {
    "faithfulness": "page",
    "hallucination": "page",
    "answer_relevancy": "review",
    "contextual_precision": "review",
    "tone_consistency": "track",
}

def route_by_severity(regressions: list[RegressionResult], run_url: str):
    paged = [r for r in regressions if SEVERITY_MAP.get(r.metric_name) == "page"]
    reviewed = [r for r in regressions if SEVERITY_MAP.get(r.metric_name) == "review"]

    if paged:
        send_slack_alert(paged, run_url)
        send_email_alert(paged, run_url, ["qa-team@yourcompany.com"])
    elif reviewed:
        send_slack_alert(reviewed, run_url)

This tiering is what keeps an alerting system alive past month two. Teams that alert on everything at the same volume train themselves to ignore the channel within weeks — the same failure mode as a smoke detector that goes off every time you make toast. Reserve the loudest channel for the regressions that actually change user-facing trust.

Handling Flaky Judge Scores Without Losing Signal

Because DeepEval's LLM-as-judge metrics call out to a model for scoring, you'll see run-to-run variance even with identical inputs. Before trusting any single regression flag, average across a few repeated measurements rather than acting on one noisy sample.

def measure_stable(metric, test_case: LLMTestCase, repeats: int = 3) -> float:
    scores = []
    for _ in range(repeats):
        metric.measure(test_case)
        scores.append(metric.score)
    return sum(scores) / len(scores)

This triples your judge-model API calls, so reserve it for your page-worthy tier — faithfulness and hallucination checks — rather than running it across every metric in the suite. For review-tier and track-tier metrics, a single measurement with a wider regression floor is a reasonable trade-off against eval cost and CI runtime.

Building a Trend Dashboard Instead of Just Point Alerts

Alerts tell you something broke right now. They don't show you a metric sliding slowly over three months of small commits. Append every CI run's scores to a flat log file or a lightweight table, and you get a queryable history for free.

import csv
from datetime import datetime, timezone

TREND_LOG = "eval_baselines/trend_log.csv"

def append_to_trend_log(current_results: dict, commit_sha: str):
    file_exists = os.path.exists(TREND_LOG)
    with open(TREND_LOG, "a", newline="") as f:
        writer = csv.writer(f)
        if not file_exists:
            writer.writerow(["timestamp", "commit_sha", "test_id", "metric_name", "score"])
        timestamp = datetime.now(timezone.utc).isoformat()
        for test_id, metrics in current_results.items():
            for metric_name, score in metrics.items():
                writer.writerow([timestamp, commit_sha, test_id, metric_name, score])

Point this CSV at any charting tool — even a spreadsheet pivot chart works for a first pass — and plot score-over-time per metric. This is how you catch the slow-bleed regression: not the 0.91-to-0.68 single-commit drop the alert system already caught, but the 0.91-to-0.85-to-0.79 grind across a dozen unrelated PRs where each individual delta stayed under your regression floor. A weekly glance at the trend chart is the backstop for exactly the case that per-commit alerting is structurally blind to.

Common Pitfalls

Comparing against an unstable baseline. If you refresh the baseline on every branch instead of only on main, a bad PR can silently redefine "normal" for itself and pass its own regression check.
Alerting on absolute delta alone. A metric that naturally lives in the 0.4-0.6 range has different noise characteristics than one that lives at 0.9+. Use relative drop alongside absolute drop.
No severity tiering. Treating every metric alert the same way guarantees channel fatigue and eventual muting.
Ignoring judge-model variance. A single low-repeat measurement on an LLM-judge metric is not reliable enough to gate a merge on its own for your most trust-critical metrics.
Not linking back to the run. An alert with just a number and no link to logs, retrieved context, and the actual generated output wastes the first five minutes of every investigation.

Wrapping Up

DeepEval gives you the measurement primitives — the metrics, the judge models, the test case structure — but turning those measurements into a system that actually protects production requires the layer this article covered: a versioned baseline, a regression comparison with both absolute and relative floors, tiered alert routing, and a trend log for the drift that no single alert will ever catch. None of it is exotic engineering. It's a JSON file, a diff function, a webhook call, and a CSV — the kind of pipeline you can have running by end of day.

The return on that afternoon of work is outsized. Prompt tweaks, model swaps, and retrieval changes happen constantly in any team shipping LLM features, and the only way to catch the ones that quietly make things worse is to compare, every time, against where you stood yesterday.

If you want a structured, hands-on walkthrough of DeepEval — from writing your first custom metric through building exactly this kind of CI regression pipeline — check out the DeepEval Tutorial course on teachyou.ai. It covers the full evaluation lifecycle for LLM applications, with real datasets and real CI configurations you can adapt directly into your own pipeline.