DeepEval Team Workflow: Reviewing Failed Evals Together

Why your eval failures are dying in a Slack thread

Somewhere in your codebase there's a test_hallucination.py that failed on last night's CI run. Someone on the team saw the red X, glanced at the assertion error, said "huh, weird," and moved on. Three weeks later the same failure mode shows up in production, a customer complains, and the postmortem starts with "wait, didn't we catch this in eval already?"

This is the most common failure pattern in teams adopting DeepEval, and it has nothing to do with the tool. DeepEval will faithfully compute your AnswerRelevancyMetric score, flag the threshold breach, and print a reason string explaining why the output scored a 0.42 instead of a 0.8. What it will not do is force your team to actually look at that output together, argue about whether the metric is even measuring the right thing, and agree on a fix. That part is a workflow problem, not a tooling problem.

Most teams treat evals as a solo activity: one engineer writes the test cases, runs deepeval test run, and either ships or doesn't based on whether the numbers look okay. That works fine when you have five test cases and one person touching the LLM pipeline. It falls apart the moment you have a RAG system with three contributors, a growing golden dataset, and metrics that disagree with each other. At that point, failed evals need to be reviewed the way you review a failed CI test suite or a code review comment thread — as a shared artifact the team reasons about together, not a private notification one person dismisses.

This article is about building that shared review habit around DeepEval specifically: how to structure your test suite so failures are reviewable, how to run a recurring "eval review" session that doesn't turn into a shouting match about arbitrary thresholds, and how to close the loop so a failed eval actually changes something in your prompts, retrieval, or data. None of this requires new tooling beyond DeepEval itself — it requires deciding, as a team, that a failed test case deserves the same seriousness as a failed deploy.

Why a failed eval needs more than one set of eyes

A single failed test case gives you three things: the input, the actual output, and a metric score with a reasoning string. That's enough for one person to form an opinion. It is rarely enough for that opinion to be correct, because a failure sits at the intersection of prompt design, retrieval quality, metric calibration, and domain truth — and no single person owns all four.

The engineer who wrote the prompt has a bias toward believing the prompt is fine and the metric is wrong. The engineer who wrote the metric has the opposite bias. The product owner who defined what "correct" means for this feature often isn't in the room at all, even though the failure might just mean the golden answer is outdated, not that the system misbehaved.

Picture a team spending two days debugging a FaithfulnessMetric failure that looks like a retrieval bug. Three engineers take turns staring at the retrieved context and the generated answer, each convinced the other's component is at fault. It turns out the golden answer was written against an older version of a source document — the underlying fact genuinely changed, and the model correctly reported the new state. No amount of solo debugging catches that, because catching it requires someone who remembers the document was updated. A five-minute conversation with the content owner solves what two days of test-staring couldn't.

That's the case for a group review: treating a failed eval as a single-person triage task guarantees you'll misattribute the failure to whichever component the reviewer happens to understand best.

Make failures legible before anyone looks at them

DeepEval is good at telling you something failed. It's less good, out of the box, at telling a teammate who wasn't in the room *why* it failed in a way they can act on. If your review session starts with someone squinting at a wall of JSON, you've already lost fifteen minutes.

Treat the reason field on every metric as first-class output, not a debug afterthought. Always request reasoning when you configure a metric:

from deepeval.metrics import AnswerRelevancyMetric

metric = AnswerRelevancyMetric(
    threshold=0.7,
    model="gpt-4o",
    include_reason=True,
    verbose_mode=True
)

include_reason=True is not optional in a team setting. Without it, a failed test case is just a number, and numbers don't generate useful discussion. With it, you get a sentence like "the response introduced a claim about refund timelines not present in the retrieval context" — something a teammate can evaluate immediately without re-running the pipeline in their head.

Second, standardize how test cases carry context. An LLMTestCase should always include input, actual_output, expected_output (when you have it), retrieval_context, and any extra context fields relevant to the metric. Teams that skip populating retrieval_context on RAG metrics like ContextualPrecisionMetric or FaithfulnessMetric end up with undebuggable failures, because nobody can tell if the model hallucinated or the retriever handed it garbage in the first place.

from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="What's our refund policy for annual plans?",
    actual_output=actual_response,
    expected_output=golden_answer,
    retrieval_context=retrieved_chunks,
    context=["Refund policy applies only to plans purchased after Jan 2025"]
)

That extra context field matters more than people expect. When a metric fails, the first question in any review session is "was this even answerable given what was retrieved?" Without retrieval_context attached, that question turns into an archaeology project nobody has time for.

Structure the golden dataset so ownership is obvious

A shared eval suite falls apart when nobody knows who added a given test case or why. Before your first team review, get the dataset into a shape where every entry has an owner and a reason for existing. DeepEval's EvaluationDataset and Golden objects support additional metadata, and you should use it:

from deepeval.dataset import EvaluationDataset, Golden

golden = Golden(
    input="Can I downgrade mid-cycle and get a prorated refund?",
    expected_output="Downgrades take effect next cycle; no proration.",
    additional_metadata={
        "owner": "priya",
        "source": "support-ticket-4021",
        "added_reason": "Customer confusion about proration, Nov cohort"
    }
)

dataset = EvaluationDataset(goldens=[golden])

This might feel like overhead with ten test cases. It stops feeling like overhead the day you have four hundred, half contributed by people who've left the team, and nobody can explain why a specific edge case is even in the suite. The source field is especially valuable — tying a golden back to a real support ticket, a real production incident, or a real user complaint means that when it fails, everyone in the room understands the stakes immediately instead of debating whether the test case is "realistic."

Organize goldens into named datasets that map to product surfaces or failure categories rather than one giant flat file. A dataset_billing_edge_cases, a dataset_multi_turn_context_retention, a dataset_adversarial_prompts — this lets your review session focus on one coherent slice at a time instead of jumping between unrelated failure types every three minutes.

Run evals on a shared, versioned cadence

If evals only run on someone's laptop before they feel like it, there's nothing to review as a team because there's no consistent artifact. Wire deepeval test run into CI so every pull request against your prompt templates, retrieval logic, or model config produces a comparable report.

deepeval test run test_rag_pipeline.py -n 4 --output-folder ./eval-reports

Save reports with the commit SHA and timestamp so they're diffable across time:

mkdir -p eval-reports/$(git rev-parse --short HEAD)
deepeval test run test_rag_pipeline.py --output-folder eval-reports/$(git rev-parse --short HEAD)

The point of this isn't automation for its own sake — a team review session needs something concrete to point at. "Eval score dropped from 0.81 to 0.68 on the billing dataset after the prompt change in PR #412" is a discussable fact. "Someone said evals looked worse yesterday" is not. If your team wants a hosted dashboard everyone can browse without pulling JSON files locally, DeepEval's Confident AI integration turns a report into something you can screen-share in five seconds instead of exporting and formatting by hand.

deepeval login --confident-api-key YOUR_KEY
deepeval test run test_rag_pipeline.py

Once results land somewhere shared, you also want a consistent way to filter for what actually needs a human. Not every failed test case deserves five minutes of group discussion — some are flaky, some are near-threshold noise, some are genuinely broken. Triage before the meeting, not during it.

Triage failures into categories before the review meeting

The single biggest time-waster in team eval reviews is re-litigating the same three categories of failure from scratch every time. Save everyone's attention by having whoever ran the last eval pass sort failures into buckets ahead of time:

• Real regressions — score dropped meaningfully versus a previous run on the same golden, tied to an identifiable code or prompt change
• Threshold noise — score sits just below threshold (say 0.65 vs a 0.7 cutoff) and the reason text shows a defensible, close-call output
• Metric mismatch — the actual output is fine, but the chosen metric (say, AnswerRelevancyMetric when you actually care about faithfulness) isn't measuring what you think it's measuring
• Bad golden — the expected output is stale, wrong, or ambiguous, often because product requirements shifted after the golden was written
• Flaky judge — the same test case scores differently across repeated runs with no code change, suggesting judge-model non-determinism rather than a real product issue
• Genuine new bug — a first-time failure mode nobody has seen before

A short script over your test results can do a first pass at this automatically by comparing current scores against the last run per test case:

import json

def triage(current_results, previous_results, threshold_margin=0.05):
    triaged = {"regression": [], "near_threshold": [], "new_failure": []}
    prev_by_input = {r["input"]: r["score"] for r in previous_results}

    for r in current_results:
        if r["success"]:
            continue
        prev_score = prev_by_input.get(r["input"])
        if prev_score is not None and r["score"] < prev_score - threshold_margin:
            triaged["regression"].append(r)
        elif r["score"] >= r["threshold"] - threshold_margin:
            triaged["near_threshold"].append(r)
        else:
            triaged["new_failure"].append(r)
    return triaged

If you suspect a specific failure is judge flakiness rather than a real regression, rerun that exact test case a handful of times before the meeting and look at the spread:

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

correctness_metric = GEval(
    name="Correctness",
    criteria="Determine whether the actual output is factually correct given the expected output.",
    evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
    threshold=0.7,
)

for i in range(5):
    correctness_metric.measure(test_case)
    print(f"Run {i}: score={correctness_metric.score}, reason={correctness_metric.reason}")

If the score bounces around by more than a small margin across identical inputs, that's not a product bug — it's a metric reliability problem, and it deserves its own conversation rather than being folded into "why did the model get this wrong."

Bring the pre-sorted list into the meeting. Spend zero time on threshold noise unless it's recurring across many test cases, which itself signals a miscalibrated threshold and belongs in a separate discussion. Spend most of your time on regressions and genuinely new failure modes, since those indicate something actually changed in behavior.

Run the review like an incident retro, not a status update

The meeting format matters more than most teams expect. A weekly "eval review" that turns into someone reading scores off a dashboard while everyone nods is worse than not having the meeting at all — it burns calendar time and teaches the team that eval reviews are boring, which kills the habit within a month.

Instead, structure it like a short incident retro, capped at 30 minutes for a normal week:

1. Pick two to four failures max, prioritizing regressions and new failure modes from the triage step. If fifteen test cases failed for the same underlying reason, show one representative case and batch-resolve the rest — showing all fifteen adds no new information.
2. For each one, read the full test case out loud — input, retrieval context if relevant, actual output, and the metric's reason string. Don't summarize it; read the actual model output. Paraphrasing hides the exact phrasing that caused the failure.
3. Ask the specific question: is this a model problem, a retrieval problem, a prompt problem, or a bad test case? Force a decision in the room rather than "let's look into it."
4. Explicitly separate "the metric is wrong" from "the product is wrong." These get conflated constantly under deadline pressure, because loosening a threshold is faster than fixing a prompt. Make the team say out loud which one they're choosing and why, so it becomes a documented decision instead of a quiet workaround.
5. Assign an owner and a next action, not a vague follow-up. "Priya will tighten the system prompt's instruction about proration language by Thursday" is trackable. "We should probably improve the prompt" is not.
6. Log the decision back onto the golden's metadata or a shared tracker, including cases where the team decided the golden itself was wrong and needs editing.

The habit of reading the raw output aloud is worth calling out specifically because it's the part teams skip first when they're rushed, and it's the part that actually generates insight. A relevancy score of 0.61 means nothing on its own. The sentence "the model answered a question about annual plan refunds by describing the monthly plan refund policy" tells the whole room exactly what to fix.

The threshold-change trap

Every team using DeepEval eventually hits this moment: a metric threshold gets adjusted mid-review, everyone nods, and the change ships in the same commit as an unrelated feature. Six months later nobody can explain why the faithfulness threshold is 0.4 instead of the textbook-recommended 0.7, and nobody wants to touch it because they don't know what else will break.

Treat threshold changes like schema migrations, not typo fixes. Every adjustment needs three things attached to it: the specific failure that motivated it, linked to the actual test case rather than a vague memory of "it felt too strict"; the name of the person who approved it, not necessarily who proposed it, but who signed off; and a review date, since thresholds set under pressure during an incident should get revisited once things calm down, not left as permanent policy.

Require a quick sanity check against the last ten or twenty runs before any threshold change ships, to see how many other test cases would flip status under the new configuration:

def would_flip(results, old_threshold, new_threshold):
    flipped = []
    for r in results:
        old_pass = r["score"] >= old_threshold
        new_pass = r["score"] >= new_threshold
        if old_pass != new_pass:
            flipped.append(r)
    return flipped

This turns a subjective argument ("I feel like this is too strict") into a concrete, bounded discussion: changing the threshold from 0.7 to 0.65 flips three out of forty test cases, here they are, do we actually want these three to start passing. A lightweight way to enforce the paper trail without extra tooling is keeping metric configuration in a version-controlled file with comments referencing the review that changed them:

# eval_config.py
METRIC_THRESHOLDS = {
    "faithfulness": 0.6,      # lowered from 0.75 on 2026-05-14 review, see run_0512
    "answer_relevancy": 0.7,
    "contextual_precision": 0.65,
}

That comment is cheap to write and expensive to lose. When the next person asks "why is this 0.6," the answer is right there instead of buried in someone's memory of a meeting three months ago.

Close the loop with regression tests, not just fixes

A failed eval that gets "fixed" without becoming a permanent regression test is a bug waiting to resurface. Every time your review session concludes that something was a genuine failure — not noise, not a bad golden — that exact input needs to become, or stay, a permanent entry in your golden dataset, tagged with the incident it came from.

from deepeval.dataset import Golden

regression_golden = Golden(
    input="Can I downgrade mid-cycle and get a prorated refund?",
    expected_output="Downgrades take effect next cycle; no proration.",
    additional_metadata={
        "owner": "priya",
        "source": "eval-review-2026-06-18",
        "regression_for": "PR#412-prompt-change",
        "status": "fixed"
    }
)

This is what actually compounds over time. Six months into a disciplined DeepEval practice, your golden dataset becomes an institutional memory of every real failure mode your system has ever exhibited, and your CI pipeline becomes a guardrail against reintroducing any of them. Teams that skip this step end up rediscovering the same bugs every few months because nothing was encoded permanently — the fix lived in someone's memory of a meeting, not in the test suite.

It's also worth periodically rerunning your full golden dataset against a fresh model version or updated prompt chain specifically to check whether old, "fixed" failure modes have quietly regressed. Model updates and prompt refactors are exactly the moments when previously-fixed bugs sneak back in, because nobody is thinking about them anymore.

A simple tagging convention on the test case itself makes this searchable later, rather than relying on anyone's memory of past reviews:

from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="What's the refund window for annual plans?",
    actual_output=actual_output,
    expected_output=expected_output,
    retrieval_context=retrieval_context,
    additional_metadata={
        "known_failure_pattern": "stale_golden_data",
        "last_reviewed": "2026-06-30",
    },
)

Over time this metadata becomes a searchable record: "show me every test case tagged stale_golden_data" turns into a quarterly cleanup task instead of a recurring surprise.

Make the review artifact durable, not ephemeral

Whatever your team decides in a review session needs to outlive the meeting. If decisions only live in people's heads or in a Slack thread that scrolls away, you'll re-litigate the same failure three months later with no memory of the previous conversation. At minimum, keep a lightweight running log — a shared doc, a Notion database, or a markdown file committed alongside your eval suite — with one row per reviewed failure: date, test case, category, decision, owner, and resolution status.

This doesn't need to be fancy tooling. A markdown table appended to after every session works fine for teams under fifteen people. What matters is that six months from now, when someone asks "didn't we already deal with this exact refund policy confusion," the answer is a two-second search instead of "let me ask around."

A few habits keep the whole process sustainable past the first month of enthusiasm. Rotate the eval-owner role who does pre-meeting triage, so no single person becomes a bottleneck and everyone's calibration on what counts as a real failure stays sharp. Cap the meeting; if triage routinely produces more than fits in thirty minutes, that's a signal the suite itself needs pruning rather than a signal to book a longer meeting. Track review cadence against release cadence rather than the calendar — a team shipping weekly needs a weekly review, a team shipping monthly can review biweekly. And say out loud when a review concludes nothing needs to change. That's a good outcome, not a wasted meeting, and it's worth naming so the team doesn't start feeling like reviews only matter when something's broken.

The teams that get the most value out of DeepEval long-term aren't the ones with the most sophisticated metrics. They're the ones that built a boring, repeatable habit of looking at failures together, arguing about them briefly and specifically, and writing down what they decided. The tool computes the numbers. The habit is what turns those numbers into a system that actually gets better over time instead of just generating dashboards nobody trusts.

If you want to go deeper on setting up DeepEval from scratch — custom metrics, synthetic dataset generation, CI integration, and the full RAG and agent evaluation patterns — our DeepEval Tutorial course on teachyou.ai walks through all of it hands-on, including the exact review workflow described here, so your team isn't building this process from zero.