LLM Eval

Statistical Significance in LLM Evals: Avoiding False Confidence

Ira Menon · May 3, 2026 · 14 min read

The 2% Bump That Wasn't Real

A team we worked with swapped their prompt template, reran their eval suite, and watched accuracy climb from 84% to 86%. Champagne came out. The new prompt shipped to production. Three weeks later, a different engineer reran the exact same eval on the exact same model and got 83.5%. Nobody had changed anything. The "improvement" had been sampling noise the whole time.

This is the single most common failure mode in LLM evaluation today: teams treat a difference in accuracy numbers as if it were a fact about the world, when it is often just a fact about which 200 examples happened to be in the test set that day. Unlike traditional software testing, where a test either passes or fails deterministically, LLM outputs are probabilistic. Small eval sets, non-deterministic sampling, and judge variance all conspire to produce numbers that look meaningfully different but aren't. Statistical significance is the tool that tells you whether you're looking at signal or looking at noise, and most teams building with LLMs skip it entirely.

This article walks through why this matters, how to actually compute significance for eval score differences, and how to build a habit of skepticism into your evaluation pipeline before you ship a regression disguised as a win.

Why LLM Evals Are Noisier Than You Think

Traditional unit tests are binary and stable — the same input produces the same output every time, so a pass rate of 100/100 means something concrete. LLM evals are built on shakier ground for a few structural reasons.

Small sample sizes. Many teams run evals on 50, 100, or 200 examples because curating and labeling a golden dataset is expensive. But with 100 examples, a single flipped answer moves your accuracy by a full percentage point. The binomial confidence interval on a 100-sample eval at 85% accuracy is roughly plus or minus 7 percentage points at the 95% level. That means an "84% vs 86%" comparison is well within the range you'd expect from chance alone.

Non-deterministic generation. Even at temperature 0, many production LLM APIs are not perfectly deterministic due to floating-point non-associativity across GPU batching, mixture-of-experts routing, and backend load balancing. At temperature greater than 0, which is how most applications actually run, you get genuine run-to-run variance in outputs for the identical prompt.

Judge variance. If you're using LLM-as-a-Judge to score outputs (more on this later), the judge itself introduces another layer of randomness. Two calls to the same judge model on the same input-output pair can disagree, especially on borderline cases near a scoring threshold.

Correlated errors. Eval examples are rarely independent and identically distributed. If your test set has ten examples about SQL date formatting and your model has one systematic blind spot there, those ten failures are really one failure repeated, not ten independent data points. This inflates your effective sample size illusion — you think you have 200 independent tests, but you might have the statistical power of 40.

Put together, these effects mean that a raw score comparison — 84% versus 86%, or 91% versus 89% — tells you almost nothing on its own. You need a way to attach uncertainty to that number.

There's also a subtler version of this problem that shows up once teams get past the basics: eval leaderboards and internal dashboards tend to reward whoever ships the biggest-looking number, which creates a quiet incentive to stop digging the moment a run looks good. Nobody sits down and decides to skip statistical rigor. It happens by attrition — the eval passes, the deadline is close, and asking "would this replicate" feels like manufacturing extra work. The fix isn't more willpower, it's making the uncertainty visible by default so that skipping it requires an active decision rather than being the path of least resistance.

The Core Idea: Confidence Intervals, Not Point Estimates

The fix is conceptually simple even if the mechanics take some getting used to: never report a single accuracy number without a confidence interval around it, and never claim one system beats another unless their intervals are actually separated in a way that a proper test confirms.

For a binary metric (pass/fail, correct/incorrect), the natural tool is the Wilson score interval, which handles small samples and extreme proportions (near 0% or 100%) far better than the naive normal approximation.

import math

def wilson_interval(successes, n, z=1.96):
    """95% Wilson score interval for a binomial proportion."""
    if n == 0:
        return (0.0, 0.0)
    p_hat = successes / n
    denom = 1 + z**2 / n
    center = p_hat + z**2 / (2 * n)
    margin = z * math.sqrt((p_hat * (1 - p_hat) / n) + (z**2 / (4 * n**2)))
    lower = (center - margin) / denom
    upper = (center + margin) / denom
    return (lower, upper)

# Example: 86 correct out of 100 examples
lower, upper = wilson_interval(86, 100)
print(f"86/100 = 86.0% accuracy, 95% CI: [{lower:.1%}, {upper:.1%}]")

# Example: 84 correct out of 100 examples
lower2, upper2 = wilson_interval(84, 100)
print(f"84/100 = 84.0% accuracy, 95% CI: [{lower2:.1%}, {upper2:.1%}]")

Running this shows both intervals spanning roughly 77% to 92%. They overlap almost entirely. That 2-point "improvement" is comfortably inside the noise band. This single check — computing and eyeballing the overlap of confidence intervals — would have caught the false win from the opening story before anyone shipped it.

Paired Comparisons: The Test You Actually Want

Comparing two independent confidence intervals is a reasonable first pass, but it's conservative and throws away information. When you're comparing two systems (old prompt vs. new prompt, model A vs. model B) on the *same* set of eval examples, you have paired data, and you should use a paired test, not two separate proportions.

The right tool here is McNemar's test, which looks specifically at the examples where the two systems disagree — it ignores cases where both got it right or both got it wrong, since those provide no information about which system is better.

from scipy.stats import binomtest

def mcnemar_test(results_a, results_b):
    """
    results_a, results_b: lists of booleans (correct/incorrect),
    same length, same example order (paired).
    """
    assert len(results_a) == len(results_b)

    # b = a correct, b wrong ; c = a wrong, b correct
    b = sum(1 for a, bb in zip(results_a, results_b) if a and not bb)
    c = sum(1 for a, bb in zip(results_a, results_b) if not a and bb)

    if b + c == 0:
        return {"b": b, "c": c, "p_value": 1.0, "note": "no discordant pairs"}

    # exact binomial test on discordant pairs
    result = binomtest(min(b, c), b + c, p=0.5, alternative="two-sided")
    return {"b": b, "c": c, "p_value": result.pvalue}

# Simulated: 100 paired examples
# System A (old prompt) and System B (new prompt) results
old_prompt_correct = [True]*84 + [False]*16
new_prompt_correct  = [True]*86 + [False]*14

# In reality you'd align these per-example, not just by count.
# This is illustrative of the shape of the input.
outcome = mcnemar_test(old_prompt_correct, new_prompt_correct)
print(outcome)

The key insight McNemar's test encodes: if the new prompt fixed 5 examples the old prompt got wrong, but broke 3 examples the old prompt got right, your net gain is only 2, and with such a small number of discordant pairs, that's nowhere near significant. You need the *pattern* of disagreement, not just the aggregate score, to know whether a change is real.

Bootstrap Resampling for Non-Binary Metrics

Not every eval metric is pass/fail. Many teams score outputs on a continuous scale — a 1-to-5 rubric score, a BLEU-style overlap score, an embedding similarity, or a judge-assigned numeric rating. For these, bootstrap resampling is the most practical tool because it doesn't require you to know the underlying distribution.

The idea: resample your eval set with replacement thousands of times, compute the mean score difference each time, and look at the distribution of those differences. If 95% of the resampled differences are on the same side of zero, you have a statistically defensible result.

import random

def bootstrap_diff_ci(scores_a, scores_b, n_bootstrap=10000, seed=42):
    """
    scores_a, scores_b: paired lists of numeric scores (same eval examples).
    Returns a 95% CI for the mean difference (b - a).
    """
    random.seed(seed)
    n = len(scores_a)
    assert n == len(scores_b)

    diffs = [b - a for a, b in zip(scores_a, scores_b)]
    boot_means = []

    for _ in range(n_bootstrap):
        sample = [random.choice(diffs) for _ in range(n)]
        boot_means.append(sum(sample) / n)

    boot_means.sort()
    lower_idx = int(0.025 * n_bootstrap)
    upper_idx = int(0.975 * n_bootstrap)
    return boot_means[lower_idx], boot_means[upper_idx]

# Example rubric scores (1-5) for 50 paired examples
scores_old = [3, 4, 4, 3, 5, 2, 4, 3, 4, 5] * 5
scores_new = [4, 4, 4, 3, 5, 3, 4, 4, 4, 5] * 5

lo, hi = bootstrap_diff_ci(scores_old, scores_new)
print(f"95% CI for mean improvement: [{lo:.3f}, {hi:.3f}]")
# If this interval excludes 0, the improvement is significant at p < 0.05

If the resulting interval straddles zero — meaning some plausible resamples show the new system as worse and others show it as better — you don't have a significant result yet, no matter how good the headline number looks. This is the check most eval dashboards never run, because it requires keeping the raw per-example scores around instead of just the aggregate mean.

How Many Eval Examples Do You Actually Need

A fair question at this point is: how big does an eval set need to be before small differences become detectable? This is a power analysis question, and the answer is almost always "bigger than you think."

As a rule of thumb, to reliably detect a 3-percentage-point difference in accuracy (say, going from 85% to 88%) with standard statistical power, you typically need somewhere in the range of 800 to 1,500 examples, not 100. To detect a 1-point difference reliably, you may need several thousand.

This has a practical consequence for how teams should think about eval investment:

If your eval set has fewer than 200 examples, treat any difference smaller than 5 points as noise until proven otherwise.
If you need to detect small, incremental improvements (which is the normal regime once a system is already reasonably good), you need to either grow the eval set substantially or accept that you'll only reliably catch large regressions.
Stratify your eval set by failure category (formatting errors, factual errors, refusals, tool-call errors) so that a real regression in one category isn't diluted into invisibility by averaging across a heterogeneous set.

Growing an eval set is often cheaper than teams assume, especially if you use production traffic sampling plus human-in-the-loop labeling rather than hand-authoring every example. The cost of *not* growing it is shipping regressions you can't detect.

There's a second lever besides raw sample size: effect size matters more than most teams assume when they're deciding whether a change is worth measuring at all. If a new retrieval strategy is expected to move accuracy by half a percentage point, no eval set you can realistically build in a sprint is going to detect that reliably, and chasing it with statistical tests is a waste of engineering time. In that regime, the right move is to change the metric, not the sample size — look for a proxy signal with a bigger expected effect (latency, cost per query, a narrower task where the change should matter more) rather than trying to squeeze significance out of a noisy, marginal comparison. Save the large, expensive eval runs for changes you actually expect to move the needle by several points.

Multiple Comparisons: The Silent Killer of Eval Dashboards

Here's a failure mode that's subtler and, in our experience, even more common than small-sample noise: running many comparisons and treating any one significant-looking result as meaningful.

Say your eval dashboard tracks 20 different slices — accuracy by category, by input length, by language, by difficulty tier. If you compare old vs. new on all 20 slices independently at a 95% confidence threshold, you should *expect* roughly one slice to look "significant" purely by chance, even if nothing actually changed. This is the multiple comparisons problem, and it's exactly how teams convince themselves that "the new model is much better on long-context Spanish queries" when what actually happened is they ran 20 tests and one came up heads.

The standard correction is straightforward to apply:

def bonferroni_correction(p_values, alpha=0.05):
    """
    p_values: list of p-values from multiple independent tests.
    Returns the corrected alpha threshold and which tests remain significant.
    """
    n_tests = len(p_values)
    corrected_alpha = alpha / n_tests
    significant = [p < corrected_alpha for p in p_values]
    return corrected_alpha, significant

# Example: p-values from 20 slice comparisons
slice_p_values = [0.31, 0.44, 0.02, 0.67, 0.89, 0.15, 0.05, 0.72,
                   0.38, 0.91, 0.03, 0.58, 0.44, 0.29, 0.61, 0.08,
                   0.77, 0.19, 0.95, 0.11]

corrected_alpha, sig_flags = bonferroni_correction(slice_p_values)
print(f"Corrected alpha: {corrected_alpha:.4f}")
print(f"Slices significant after correction: {sum(sig_flags)} of {len(sig_flags)}")

Bonferroni is conservative but simple, and conservative is the right default when the cost of a false "it improved" claim is a regression shipped to production. If you're running dozens of slice comparisons every eval cycle, correcting for multiple comparisons isn't optional statistical hygiene — it's the difference between a dashboard that tells you the truth and one that generates a new false story every week.

Practical Eval Pipeline Habits That Prevent False Confidence

Beyond the specific tests, a handful of process habits do most of the work in practice.

Always report an interval, never just a point estimate. Change your eval report template so that "86% accuracy" is not an acceptable line by itself — it should always read "86% (95% CI: 79-91%, n=100)."
Fix the random seed and log it. If your eval involves any sampling — which examples get selected, which few-shot examples get used, temperature-based generation — log the seed so a "regression" can be reproduced rather than debated.
Re-run before you believe a win. A cheap, high-value habit: rerun any eval result that looks like an improvement at least once before shipping. If it doesn't replicate, you just saved yourself from the trap in the opening story.
Track per-example results across runs, not just aggregates. Keep a table of example ID, model version, and pass/fail so you can run McNemar's test or bootstrap comparisons later, instead of only ever seeing the rolled-up percentage.
Separate "did it change" from "did it change for a reason we understand." A statistically significant regression on an obscure slice still needs a root cause before you decide whether to block a release. Statistics tells you something is real; it doesn't tell you why.
Version your eval set itself. If the eval set changes between runs — even by a few added or removed examples — any comparison between "before" and "after" numbers is comparing two different experiments, not two versions of the same experiment.

None of these require exotic tooling. Most eval frameworks already store per-example results somewhere; the discipline is in actually pulling that data out and running the paired test instead of eyeballing two percentages on a dashboard.

It's also worth building a lightweight gate into your CI or release process rather than leaving significance testing as a manual step someone might forget under deadline pressure. A simple version: any eval run that claims an improvement must attach a p-value or confidence interval computed via one of the methods above, and any comparison where the interval crosses zero (or the p-value exceeds your threshold) gets labeled "not significant" automatically in the report, regardless of how good the headline number looks. Teams that wire this into their pipeline stop having the "wait, did this actually get better" conversation in a Slack thread three weeks after shipping, because the answer was already computed and attached to the pull request before anyone had to ask.

Where LLM-as-a-Judge Fits Into the Picture

Everything above assumes you have a way to score whether an output is "correct" in the first place, and increasingly that scoring itself is done by another LLM acting as a judge rather than by exact string matching or human review. LLM-as-a-Judge is enormously useful for evaluating free-form generation — summaries, explanations, code review comments, chat responses — where there's no single correct string to match against.

But a judge model is itself a noisy measurement instrument, and everything in this article applies one level up: judge scores need confidence intervals too, judge agreement with humans should be measured and reported (not assumed), and if you're comparing two systems using judge scores, you still need paired tests across the same examples rather than comparing two separately-sampled judge runs. A judge that agrees with itself only 85% of the time on repeated calls to the same input adds a whole extra layer of variance that a naive "judge said System B is better" conclusion will quietly ignore. If you're building or auditing a judge-based eval pipeline, treating the judge's verdicts with the same statistical skepticism you'd apply to the underlying model is what separates a trustworthy eval system from a very expensive random number generator — and it's exactly the kind of applied rigor we walk through hands-on in our AI engineering courses at teachyou.ai.