Setting Confidence Thresholds in DeepEval: Avoiding False Failures

The 2 a.m. Slack message nobody wants

You ship a prompt change on Friday afternoon. By Monday, your CI pipeline has failed the eval suite eleven times over the weekend, paging whoever's on rotation each time. You open the logs expecting a broken feature. Instead, you find outputs that are perfectly fine — maybe even better than before — sitting at a score of 0.69 against a threshold of 0.7. One point below the line. Not wrong. Just not "confident enough" by an arbitrary cutoff nobody revisited since the day it was pasted from a tutorial.

This is the most common failure mode teams hit when they adopt DeepEval for LLM testing: the threshold is treated like a magic number instead of a calibrated decision. A threshold of 0.7 isn't a law of physics — it's a guess, usually copied from documentation examples, that happens to work for one metric on one dataset and then gets reused everywhere else. The result is a wave of false failures that erode trust in the eval suite faster than any real regression would. Once your team starts saying "oh, that test always flakes, just re-run it," you've lost the entire point of automated evaluation.

This article is about fixing that. We'll go through how DeepEval actually computes scores and applies thresholds, why the default numbers create false failures, and a practical process for setting thresholds that reflect your actual quality bar instead of someone else's example notebook.

How DeepEval scores and thresholds actually interact

Before tuning anything, it helps to be precise about what a "threshold" is doing under the hood. In DeepEval, most metrics — AnswerRelevancyMetric, FaithfulnessMetric, ContextualPrecisionMetric, GEval, and friends — return a float score, typically between 0 and 1, plus a success boolean that's simply score >= threshold. The threshold is not part of the scoring algorithm itself; it's a pass/fail gate bolted on afterward.

from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

metric = AnswerRelevancyMetric(threshold=0.7, model="gpt-4o-mini")

test_case = LLMTestCase(
    input="What's the refund policy for annual plans?",
    actual_output="Annual plans are refundable within 30 days of purchase, prorated after that.",
    retrieval_context=["Annual subscriptions can be refunded in full within the first 30 days."]
)

metric.measure(test_case)
print(metric.score)      # e.g. 0.83
print(metric.reason)     # LLM-generated explanation of the score
print(metric.success)    # True, because 0.83 >= 0.7

That looks clean in isolation. The problem is that most of DeepEval's metrics are themselves LLM-as-a-judge metrics — they call an evaluator model (often GPT-4o or GPT-4o-mini) to grade the output, extract claims, check entailment, or rate relevancy. That judge model has its own variance. Ask it to grade the same input/output pair five times and you won't get five identical scores — you'll get a small distribution, maybe 0.74, 0.81, 0.77, 0.69, 0.79. If your threshold sits at 0.7, that natural judge variance alone will occasionally flip success from True to False with zero change in actual output quality.

This is the core insight: a threshold that ignores judge variance is really a coin flip disguised as a quality bar.

Why the default thresholds fail you

DeepEval ships sensible defaults (0.5 for many metrics, 0.7 shows up a lot in tutorials) because they have to ship *something*. But defaults are calibrated for nothing in particular — they're not calibrated to your domain, your judge model, your prompt style, or your risk tolerance. A few concrete reasons the defaults produce false failures in real projects:

• Judge model variance is domain-dependent. A FaithfulnessMetric judging factual claims in a legal-document Q&A bot behaves very differently from the same metric judging a casual customer-support chatbot. Terse, fact-dense outputs often score more harshly than verbose ones, purely because there are more atomic claims to verify against context.
• Short outputs get penalized disproportionately. Metrics like AnswerRelevancyMetric break the output into statements and score how many are relevant. A one-sentence answer that's 100% correct can score lower than a rambling answer that pads itself with relevant filler, simply because of how the statement-extraction step works.
• Retrieval context quality bleeds into generation metrics. If your RAG pipeline's retriever is mediocre, FaithfulnessMetric will punish the *generator* for not hallucinating extra detail that wasn't in the (incomplete) context — even though the generator behaved exactly as it should have.
• Threshold copy-paste across metrics. Teams frequently set threshold=0.7 for every metric in a suite — relevancy, faithfulness, contextual recall, custom GEval metrics — as if it's a universal passing grade. Each metric has a different score distribution and a different meaning at 0.7.

None of this means the metrics are broken. It means treating a single hardcoded number as ground truth for "good enough" is the actual bug.

Reframe the threshold as a calibrated decision, not a constant

The fix starts with a mental model shift: a threshold is the output of a calibration process, not an input you type once. Before setting a number, answer three questions:

1. What does this metric's score distribution look like on outputs you already know are good? Run the metric against 20-30 known-good historical outputs and look at the score spread.
2. What does the distribution look like on outputs you know are bad? Same exercise with outputs your team has manually flagged as unacceptable.
3. Where's the gap between the two distributions? The threshold should sit in that gap — not at the edge of either cluster.

Here's a small script that runs this calibration pass using DeepEval directly, rather than guessing:

from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase
import statistics

def calibrate_threshold(good_cases, bad_cases, model="gpt-4o-mini"):
    metric = FaithfulnessMetric(threshold=0.0, model=model)  # threshold irrelevant here

    good_scores, bad_scores = [], []

    for case in good_cases:
        metric.measure(case)
        good_scores.append(metric.score)

    for case in bad_cases:
        metric.measure(case)
        bad_scores.append(metric.score)

    good_min = min(good_scores)
    bad_max = max(bad_scores)

    print(f"Known-good scores: {good_scores}")
    print(f"Known-bad scores:  {bad_scores}")
    print(f"Good mean: {statistics.mean(good_scores):.2f}, min: {good_min:.2f}")
    print(f"Bad mean:  {statistics.mean(bad_scores):.2f}, max: {bad_max:.2f}")

    if bad_max < good_min:
        # clean separation — pick the midpoint
        suggested = round((good_min + bad_max) / 2, 2)
        print(f"Clean separation found. Suggested threshold: {suggested}")
    else:
        print("Overlap detected between good and bad distributions.")
        print("A single scalar threshold will misclassify some cases — consider G-Eval refinement or human review buffer.")

    return good_scores, bad_scores

This isn't fancy — it's a basic separation-of-distributions check, the same logic you'd apply to any binary classifier threshold. But almost nobody runs it before shipping an eval suite. They just paste threshold=0.7 and move on. Ten minutes of calibration against a labeled set of 20-40 examples will tell you more about the right number than any documentation default ever will.

Handling metrics that overlap: no clean threshold exists

Sometimes step 3 above reveals there's no clean gap — your known-good and known-bad score distributions overlap. This is common with subjective metrics like tone, helpfulness, or brand-voice alignment scored via GEval. In that case, a single scalar threshold is fundamentally the wrong tool, and no amount of tuning the number fixes it. You have three real options:

• Tighten the `GEval` criteria. Vague evaluation steps ("check if the response is helpful") produce vague, overlapping score distributions. Specific, checklist-style evaluation steps produce tighter, more separable distributions.
• Add a human-review band instead of a hard cutoff. Treat scores in the overlap zone as "needs review" rather than pass/fail.
• Switch from a threshold gate to a trend metric. Track the score over time instead of gating individual test runs on it.

Here's what a tightened GEval metric looks like in practice — the difference between a vague criterion and a specific one is often the single biggest lever you have, bigger than threshold tuning itself:

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

# Too vague — produces a wide, overlapping score distribution
vague_metric = GEval(
    name="Helpfulness",
    criteria="Determine if the response is helpful to the user.",
    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
    threshold=0.7,
)

# Specific — produces a tighter, more separable distribution
tight_metric = GEval(
    name="Helpfulness",
    criteria="Determine if the response directly answers the user's question, "
             "includes at least one concrete next step or example, "
             "and does not ask a clarifying question when the input already "
             "contains enough information to answer directly.",
    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
    threshold=0.75,
)

The vague version forces the judge model to make subjective calls with no anchor, which is exactly where judge variance spikes. The specific version gives the judge model a near-checklist, which shrinks the variance and makes the eventual threshold far more meaningful.

Use per-metric thresholds, never a global one

If you take one tactical change from this article, make it this: stop applying one threshold value across every metric in your suite. Each metric measures something structurally different, and their score distributions don't share a scale in any meaningful sense.

from deepeval.metrics import (
    AnswerRelevancyMetric,
    FaithfulnessMetric,
    ContextualPrecisionMetric,
    ContextualRecallMetric,
)

# Calibrated independently, based on distribution analysis per metric
answer_relevancy = AnswerRelevancyMetric(threshold=0.75, model="gpt-4o-mini")
faithfulness = FaithfulnessMetric(threshold=0.85, model="gpt-4o-mini")  # hallucination risk = higher bar
contextual_precision = ContextualPrecisionMetric(threshold=0.6, model="gpt-4o-mini")
contextual_recall = ContextualRecallMetric(threshold=0.65, model="gpt-4o-mini")

Notice faithfulness sits higher than the others here — that's deliberate, not accidental. Faithfulness is your hallucination guardrail; a false negative (an unfaithful answer that slips through) is usually more costly than a false positive (a faithful answer that gets incorrectly flagged). Contextual precision and recall, by contrast, are diagnostic signals about your retriever, not user-facing safety issues, so they can tolerate a looser bar without real damage.

This is the underlying principle: set the threshold according to the cost of being wrong in each direction, per metric, not according to a single number that feels "standard."

Give judge-model variance room to breathe

Even after calibration, remember that LLM judges are non-deterministic by nature. DeepEval lets you set strict_mode on metrics, which forces a binary 0/1 outcome rather than a continuous score — useful for hard requirements, but it removes your ability to see near-misses. For most metrics, you want the opposite: visibility into how close a "failure" actually was.

A practical pattern is to run borderline cases multiple times and average, rather than trusting a single judge call:

from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase
import statistics

def stable_measure(test_case: LLMTestCase, threshold: float, runs: int = 3):
    scores = []
    for _ in range(runs):
        metric = FaithfulnessMetric(threshold=threshold, model="gpt-4o-mini")
        metric.measure(test_case)
        scores.append(metric.score)

    avg_score = statistics.mean(scores)
    spread = max(scores) - min(scores)

    result = {
        "avg_score": round(avg_score, 3),
        "spread": round(spread, 3),
        "success": avg_score >= threshold,
        "flagged_for_review": spread > 0.15,  # high variance = judge uncertainty, not a real failure signal
    }
    return result

This isn't something you run on every single CI test — that would triple your evaluation cost and latency for no reason. Reserve it for test cases that fail close to the threshold (say, within 0.05) before you let a single noisy judge call fail your build. In practice, a large share of "false failures" reported by teams turn out to be exactly this: one unlucky judge call on a borderline case, not a real regression.

Build a threshold review cadence, not a one-time setting

Thresholds decay. As your prompts evolve, your retrieval pipeline improves, or you swap the underlying model your app runs on, the score distributions shift — and a threshold calibrated for last quarter's outputs can silently become miscalibrated for this quarter's. Treat threshold values the same way you'd treat any other piece of configuration that affects production behavior: version it, review it, and revisit it on a schedule.

A few habits that keep this sane:

• Log every score, not just pass/fail. If your CI only records success: True/False, you throw away the exact data you need to notice drift. Store the raw score alongside the test run.
• Review score distributions monthly, or after any prompt/model change. A five-minute look at a histogram of recent scores tells you if the threshold is still sitting in a sensible gap.
• Track false-failure rate as its own metric. When a human reviews a "failed" test and finds the output was actually fine, log it. If that rate creeps above a few percent, your threshold (or your metric criteria) needs attention, not your prompt.
• Keep threshold changes in version control alongside prompt changes. A threshold bump from 0.7 to 0.65 is a real behavioral decision about your test suite — it deserves a commit message explaining why, just like a prompt edit does.

Putting it together: a working test suite

Here's a compact example that ties calibrated, per-metric thresholds together in an actual deepeval test, the way you'd wire it into pytest:

import pytest
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase

# Thresholds derived from calibration against known-good/known-bad sets,
# not copied from a tutorial. See docs/eval-calibration.md for the run that produced these.
RELEVANCY_THRESHOLD = 0.75
FAITHFULNESS_THRESHOLD = 0.85

def test_support_bot_refund_answer():
    test_case = LLMTestCase(
        input="Can I get a refund on my annual plan after 45 days?",
        actual_output="No — annual plans are only refundable within the first 30 days of purchase.",
        retrieval_context=["Annual subscriptions can be refunded in full within the first 30 days."]
    )

    relevancy = AnswerRelevancyMetric(threshold=RELEVANCY_THRESHOLD, model="gpt-4o-mini")
    faithfulness = FaithfulnessMetric(threshold=FAITHFULNESS_THRESHOLD, model="gpt-4o-mini")

    assert_test(test_case, [relevancy, faithfulness])

The important part isn't the syntax — it's the comment. Every threshold in a suite like this should be traceable back to a calibration decision someone made deliberately, on real data, with a documented reason. That's what turns an eval suite from a source of weekend pages into something your team actually trusts enough to block a deploy on.

Closing thoughts

False failures aren't a DeepEval bug — they're what happens when a genuinely useful tool gets configured with numbers nobody actually chose. The fix isn't complicated, but it does require slowing down at setup time: pull a handful of known-good and known-bad examples, measure the real score distributions, set thresholds per metric based on the cost of getting each one wrong, tighten vague GEval criteria before you touch the threshold at all, and revisit the numbers on a cadence instead of treating them as permanent. Do that once, properly, and your eval suite stops crying wolf — which is the only way anyone keeps listening to it when it finally does bark at something real.

If you want to go deeper on wiring DeepEval into a real CI pipeline, building custom GEval criteria for your domain, and setting up the kind of calibration workflow described here end-to-end, our DeepEval Tutorial course on teachyou.ai walks through all of it hands-on, from first metric to a production-grade eval suite.