Interpreting DeepEval Score Distributions Across a Test Run

Why a single pass rate hides more than it reveals

You run your DeepEval suite, the terminal prints "42/50 tests passed," and someone on the team says "84%, ship it." That number feels like progress, but it is one of the least informative things DeepEval gives you. A pass rate collapses every metric, every test case, and every threshold decision into a single scalar. It cannot tell you whether your failures are clustered in one hard category or spread randomly across the suite. It cannot tell you whether your passes are comfortably above threshold or scraping by at 0.51 when the bar is 0.5. It cannot tell you if last week's "improvement" from 78% to 84% actually made the model better, or just moved a handful of borderline cases across an arbitrary line.

DeepEval already computes a real number for every metric on every test case, not just a boolean. AnswerRelevancyMetric, FaithfulnessMetric, ContextualPrecisionMetric, and friends all return a float between 0 and 1 before that float gets thresholded into pass/fail. The moment you throw that float away and keep only the boolean, you throw away the signal that tells you *how* your system is behaving, not just whether it crossed a line you picked somewhat arbitrarily during setup.

This article is about treating a DeepEval run as a distribution problem, not a pass/fail problem. We will look at how scores actually spread across a test run, how to pull that data out of DeepEval's output, what shapes are healthy versus concerning, and how to use distribution thinking to set better thresholds, catch regressions earlier, and stop arguing about individual borderline test cases.

What a DeepEval test run actually produces

When you run an evaluation with DeepEval, each LLMTestCase gets scored against each metric you attach to it. If you evaluate 50 test cases against FaithfulnessMetric and AnswerRelevancyMetric, you get 100 individual score, reason pairs, not 50. Each metric object exposes .score and .reason after evaluation, and DeepEval's evaluate() function returns a result object where every test result carries its own list of metric outcomes.

Here is a minimal setup that produces the raw material for a distribution analysis:

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase

test_cases = [
    LLMTestCase(
        input="What is the refund window for annual plans?",
        actual_output="Annual plans can be refunded within 30 days of purchase.",
        retrieval_context=["Annual subscriptions are refundable within 30 days."],
    ),
    # ... 49 more test cases pulled from your eval dataset
]

relevancy_metric = AnswerRelevancyMetric(threshold=0.7)
faithfulness_metric = FaithfulnessMetric(threshold=0.7)

results = evaluate(
    test_cases=test_cases,
    metrics=[relevancy_metric, faithfulness_metric],
)

The results object contains a test_results list, and each entry has a metrics_data list with the score, threshold, success flag, and reason for every metric applied to that test case. That is the raw material. Most teams stop at printing whether each test passed. The more useful move is to pull every .score value out into a flat list per metric and actually look at how those numbers are distributed, because a metric passing 90% of the time can still be dangerously close to its threshold on every single case, and that is invisible if you only look at booleans.

Extracting scores into something you can actually analyze

Before you can reason about a distribution, you need the raw scores in a shape you can compute on. This is a few lines of code, and it is worth writing once and reusing across every project:

def collect_scores_by_metric(results):
    """Flatten a DeepEval evaluate() result into {metric_name: [scores]}."""
    scores_by_metric = {}
    for test_result in results.test_results:
        for metric_data in test_result.metrics_data:
            metric_name = metric_data.name
            scores_by_metric.setdefault(metric_name, []).append(metric_data.score)
    return scores_by_metric

scores = collect_scores_by_metric(results)
# scores == {"Answer Relevancy": [0.92, 0.88, 0.41, ...], "Faithfulness": [1.0, 0.95, ...]}

Once you have scores_by_metric, you can compute the statistics that actually describe the shape of your run: mean, median, standard deviation, min, max, and the count of scores that fall within a narrow band around the threshold. That last one matters more than people expect. A test case that scores 0.71 against a 0.7 threshold is technically a pass, but it is a pass by luck, not by margin, and a small prompt change or a slightly different retrieval chunk could flip it to a fail next week.

import statistics

def summarize(scores, threshold, margin=0.05):
    near_threshold = [s for s in scores if abs(s - threshold) <= margin]
    return {
        "n": len(scores),
        "mean": round(statistics.mean(scores), 3),
        "median": round(statistics.median(scores), 3),
        "stdev": round(statistics.pstdev(scores), 3) if len(scores) > 1 else 0.0,
        "min": round(min(scores), 3),
        "max": round(max(scores), 3),
        "pass_rate": round(sum(s >= threshold for s in scores) / len(scores), 3),
        "near_threshold_count": len(near_threshold),
    }

for metric_name, metric_scores in scores.items():
    print(metric_name, summarize(metric_scores, threshold=0.7))

Running this over a real suite gives you output like Faithfulness {'n': 50, 'mean': 0.81, 'median': 0.88, 'stdev': 0.19, 'min': 0.12, 'max': 1.0, 'pass_rate': 0.86, 'near_threshold_count': 4}. The pass rate alone said "86%, looks fine." The full summary says something different: there is a wide spread (stdev 0.19), a floor as low as 0.12 which suggests at least one genuinely broken response, and four test cases sitting within 0.05 of the threshold that could flip either direction on a rerun.

Reading the shape, not just the average

Once you have scores in hand, plotting or even just bucketing them tells you more than any single statistic. A histogram with bins like 0.0-0.2, 0.2-0.4, 0.4-0.6, 0.6-0.8, 0.8-1.0 will usually reveal one of a few recognizable shapes.

• Bimodal distribution — a cluster near 0.9-1.0 and a separate cluster near 0.1-0.3, with almost nothing in between. This is the most common and most informative shape. It means your system is either handling a case well or failing it outright, with very little "partially correct" territory. This usually points to a categorical failure mode: a specific type of question, a specific retrieval gap, or a specific prompt pattern that the model either has or doesn't have context for.
• Unimodal, centered near threshold — most scores cluster tightly around 0.65-0.75 when your threshold is 0.7. This is the shape that should worry you most even if the pass rate looks acceptable, because it means the system is consistently mediocre rather than reliably good, and small changes to the threshold or the underlying model will swing your pass rate wildly.
• Unimodal, centered high with a long left tail — most scores sit at 0.85+ with a small number of low outliers dragging the mean down. This is generally the healthiest shape. It suggests a system that works well in general with a small number of identifiable edge cases, which is a much easier problem to debug than systemic mediocrity.
• Flat or uniform spread — scores spread roughly evenly from 0 to 1 with no clear cluster. This often indicates the metric itself is noisy for your use case, or that your test cases are too heterogeneous to be evaluated by one metric configuration, or that the LLM-as-judge grading model is not getting the context it needs to be consistent.

Recognizing these shapes changes what you do next. A bimodal distribution tells you to go find the low cluster and look for what those test cases have in common. A centered-near-threshold distribution tells you the threshold itself may be poorly chosen, or that you have a genuine mediocrity problem worth escalating. A flat distribution tells you to sanity check the metric configuration before trusting any conclusion drawn from it.

Comparing distributions across runs, not just pass rates

The real payoff of distribution thinking shows up when you compare two runs, say before and after a prompt change or a retrieval config change. Comparing pass rates alone (84% versus 88%) tells you almost nothing about whether the change was actually good. Comparing distributions tells you a lot more.

def compare_runs(before_scores, after_scores, threshold=0.7):
    before_summary = summarize(before_scores, threshold)
    after_summary = summarize(after_scores, threshold)

    print(f"{'metric':<20}{'before':>10}{'after':>10}{'delta':>10}")
    for key in ("mean", "median", "stdev", "min", "pass_rate"):
        before_val = before_summary[key]
        after_val = after_summary[key]
        delta = round(after_val - before_val, 3)
        print(f"{key:<20}{before_val:>10}{after_val:>10}{delta:>10}")

compare_runs(before_run_scores, after_run_scores)

Suppose the pass rate goes from 0.84 to 0.88, a four-point improvement that looks like clear progress. But suppose the mean actually dropped slightly, from 0.79 to 0.77, and the standard deviation increased. That combination means the change pushed a few borderline failures over the threshold (which is why pass rate ticked up) while simultaneously making some previously-strong responses weaker. Net effect: your pass rate improved, but your system got less consistent. That is exactly the kind of change that looks good in a dashboard and bad in production, and you would never see it by comparing pass rates alone.

The reverse pattern also happens: a change can lower your pass rate while raising your mean and lowering your standard deviation, meaning the whole distribution shifted up but one or two edge cases that were barely passing before now barely fail. That is often a change worth keeping, because you traded one or two easily-identified regressions for a broadly more consistent system, but you would only find those specific regressions by inspecting the tails, not by watching the aggregate percentage.

Per-metric distributions tell different stories

A common mistake is looking at a single blended pass rate across all metrics in a test run, when FaithfulnessMetric, AnswerRelevancyMetric, and ContextualPrecisionMetric measure fundamentally different failure modes and often have different distribution shapes for the same underlying system.

In a RAG pipeline, for instance, it is common to see:

• ContextualPrecisionMetric and ContextualRecallMetric scores clustered high and tight, because your retrieval step is doing its job consistently.
• FaithfulnessMetric scores bimodal, because the generation step either sticks closely to retrieved context or occasionally hallucinates additions, with little middle ground.
• AnswerRelevancyMetric scores unimodal and centered a bit lower than you'd like, because the model tends to answer adjacent questions rather than the exact one asked.

If you only track a single blended score, this pattern is invisible. If you break scores out by metric and look at each distribution separately, you immediately know where to focus: retrieval is fine, faithfulness has a clear bimodal split worth root-causing, and relevancy has a systemic drift problem that needs prompt-level attention rather than a retrieval fix. This is the difference between "the eval score dropped, go look at everything" and "faithfulness bimodality increased, check whether the new context chunks are longer and giving the model more room to add unsupported details."

Using G-Eval and custom metrics with distribution awareness

If you're using GEval for a custom rubric metric, the same principles apply, but you should be more careful about the shape you expect. GEval scores are produced by an LLM judge reasoning through evaluation steps, and depending on how you write your criteria, the resulting distribution can be naturally more discrete (the judge tends to give round numbers like 0.2, 0.5, 0.8, 1.0) than metrics with more continuous underlying computation like embedding-based relevancy.

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

tone_metric = GEval(
    name="Support Tone",
    criteria=(
        "Determine whether the response is empathetic, avoids blame, "
        "and offers a concrete next step for the customer."
    ),
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
    ],
    threshold=0.7,
)

When you look at the score distribution for a metric like this across 50 test cases, expect more clumping at specific values than you'd see from an embedding-similarity metric, simply because the judge model is reasoning in discrete steps rather than producing a continuous similarity score. Do not mistake that clumping for a broken metric. Instead, use the reason field attached to each score (every DeepEval metric stores a .reason after evaluation) to read the judge's actual reasoning for a handful of cases at each cluster value. That qualitative check is what tells you whether the clumping reflects real categorical differences in your data, or whether the judge prompt itself needs tightening because it's not discriminating between genuinely different quality levels.

Turning distribution analysis into a regression gate

Once you're comfortable reading distributions, it's worth building a lightweight regression check into CI that goes beyond a simple pass rate threshold. Instead of failing a build only when pass rate drops below some number, fail it when the distribution shape degrades in a way that a single pass rate would miss.

def distribution_regression_check(baseline_scores, current_scores, threshold=0.7):
    baseline = summarize(baseline_scores, threshold)
    current = summarize(current_scores, threshold)

    failures = []
    if current["mean"] < baseline["mean"] - 0.05:
        failures.append(f"mean dropped: {baseline['mean']} -> {current['mean']}")
    if current["stdev"] > baseline["stdev"] + 0.05:
        failures.append(f"variance increased: {baseline['stdev']} -> {current['stdev']}")
    if current["min"] < baseline["min"] - 0.1:
        failures.append(f"worst case got worse: {baseline['min']} -> {current['min']}")
    if current["near_threshold_count"] > baseline["near_threshold_count"] + 2:
        failures.append("more test cases are now sitting on the threshold edge")

    return failures

regressions = distribution_regression_check(baseline_scores, current_scores)
if regressions:
    for reason in regressions:
        print(f"REGRESSION: {reason}")
    raise SystemExit(1)

This kind of gate catches the exact scenario described earlier, where pass rate improves but the underlying distribution got noisier or the worst case got meaningfully worse. It won't replace human judgment on your eval results, but it stops "pass rate went up" from being treated as the whole story in a PR review, and it gives reviewers a concrete, numeric reason to look closer at a change before merging it.

Common mistakes teams make when reading DeepEval output

A few patterns show up repeatedly once teams start actually running DeepEval at scale, and most of them come from treating the pass/fail boolean as the primary artifact instead of a derived one.

• Picking a threshold once and never revisiting it. Thresholds should be informed by where your actual score distribution sits, not by a round number that felt reasonable during initial setup. If your FaithfulnessMetric scores naturally cluster at 0.6 and 0.95 with almost nothing between, a threshold of 0.7 is doing real work separating two populations. If scores are smeared evenly from 0.5 to 0.9, that same threshold of 0.7 is an arbitrary cut through a continuum, and small changes will swing your pass rate without any real change in quality.
• Averaging across incompatible test case types. If your eval dataset mixes simple factual lookups with multi-step reasoning questions, blending their scores into one distribution hides the fact that you likely have two different systems in disguise: one that's reliably strong and one that's reliably weak. Segment your distributions by test case category before drawing conclusions.
• Ignoring the `reason` field. Every score DeepEval produces comes with a reason string explaining the judge's evaluation. When you find an interesting cluster in your distribution, whether it's the tail of failures or a suspiciously tight clump near threshold, read the reasons for a sample of those cases before acting. The number tells you where to look; the reason tells you why.
• Treating one bad run as ground truth. LLM-as-judge metrics have some inherent run-to-run variance, especially for borderline cases. Before declaring a regression based on a single run's distribution, rerun the suite once or twice and check whether the shape is stable. A distribution that changes meaningfully between two identical runs is telling you about judge noise, not model quality.

Building a habit around distributions, not just dashboards

None of this requires new tooling beyond what DeepEval already gives you through metrics_data, .score, and .reason. The shift that matters is procedural: after every eval run, before anyone reports a pass rate in standup, pull the scores into a list, compute mean, median, standard deviation, and near-threshold count per metric, and glance at the shape. It takes a few minutes and it catches the exact class of problem that pass rates are structurally blind to: quiet regressions that trade borderline failures for borderline passes, systemic mediocrity masquerading as an acceptable percentage, and categorical failure modes hiding inside an aggregate number that looks fine on average.

If you want to go deeper into building this kind of eval discipline into a real project, from writing test cases and choosing metrics to setting up CI gates that actually catch regressions instead of rubber-stamping them, our DeepEval Tutorial course on TeachYou.ai walks through the whole workflow hands-on, including the distribution analysis patterns covered here applied to a real RAG pipeline from scratch.