Evaluating Code Generation: Beyond Pass@k

Why Pass@k Became the Default, and Why That's a Problem

If you've spent any time reading model cards for code-generating LLMs, you've seen the same number quoted over and over: Pass@1 on HumanEval, Pass@10 on MBPP, some variant of "the model solved X% of problems." It's become the de facto scoreboard for evaluating code generation, and it's easy to see why. Pass@k is simple to compute, easy to compare across models, and gives you a single number to put in a table.

Here's the problem: Pass@k measures exactly one thing — whether at least one of k generated samples passes a fixed set of unit tests. That's it. It doesn't tell you whether the code is secure. It doesn't tell you whether the code would survive a code review. It doesn't tell you whether the model just memorized the benchmark problem during pretraining. And it definitely doesn't tell you whether the code will still make sense to a human maintainer six months from now.

When you're building a product that generates code — a coding assistant, an autonomous agent that opens pull requests, a tool that scaffolds boilerplate — Pass@k is a starting point, not an evaluation strategy. Teams that ship code-gen features into production learn this the hard way: the model that tops the HumanEval leaderboard is not necessarily the model that produces code your engineers will merge without three rounds of review comments.

This article is about what comes after Pass@k. We'll cover why the metric breaks down in practice, what a more complete evaluation harness looks like, and how to combine automated checks with LLM-as-a-Judge to catch the things unit tests structurally can't.

What Pass@k Actually Measures

Let's be precise about the definition, because a lot of confusion stems from people treating Pass@k as a vague "correctness score" rather than the specific statistical estimator it is.

Pass@k estimates the probability that at least one of k independently sampled code completions passes all given test cases, for a single problem, averaged across a benchmark's problem set. The original HumanEval paper introduced an unbiased estimator for this because naively generating exactly k samples and checking if any pass gives a high-variance estimate. Instead, you generate n ≥ k samples per problem, count how many (c) pass, and compute:

import math

def pass_at_k(n: int, c: int, k: int) -> float:
    """
    Unbiased estimator of pass@k.
    n: total samples generated per problem
    c: number of samples that passed all tests
    k: the 'k' in pass@k
    """
    if n - c < k:
        return 1.0
    return 1.0 - math.prod(
        (n - c - i) / (n - i) for i in range(k)
    )

This is a genuinely useful piece of engineering — it lets you estimate Pass@100 from just 200 samples instead of generating 100 completions per problem for every k you care about. But notice what it depends on: a fixed, finite set of test cases per problem, decided in advance by whoever built the benchmark.

That dependency is where most of the real-world gaps come from.

The Test Coverage Gap

HumanEval, MBPP, and most of their descendants ship with a small number of test cases per problem — often just three to five assert statements. A now well-known finding from the code-eval community is that models can pass HumanEval's official tests while producing code that fails on inputs the test suite simply never checked: empty lists, negative numbers, unicode strings, integer overflow boundaries.

Consider a canonical HumanEval-style problem: "write a function that returns the running sum of a list of numbers." A model might generate:

def running_sum(nums):
    total = 0
    result = []
    for n in nums:
        total += n
        result.append(total)
    return result

This passes the typical test cases ([1,2,3] -> [1,3,6]). But what about running_sum([])? What about running_sum(None)? What about a list with 10 million elements where you'd actually want a generator instead of building a full list in memory? The benchmark's three assert statements say nothing about any of this, and Pass@k will happily report 100% on a function that's one edge case away from a production incident.

This is why benchmarks like EvalPlus emerged — they take existing HumanEval and MBPP problems and augment them with dramatically more test cases (often 80x more) generated through a combination of LLM-based test generation and mutation testing. When teams re-ran existing model outputs against EvalPlus's expanded test suites, pass rates dropped meaningfully across the board — often by 10 to 20 percentage points — because the original test suites simply weren't looking hard enough.

The lesson for anyone building an internal eval: the ceiling of your evaluation is the thoroughness of your test cases, not the intelligence of your model. If you're evaluating a code-gen feature for your own product, do not reuse a public benchmark's test suite unmodified. Extend it. Add boundary conditions, malformed inputs, and adversarial cases specific to your domain.

Contamination: When the Model Has Seen the Answer

The second structural problem with public benchmarks is data contamination. HumanEval was released in 2021. MBPP followed shortly after. Every major foundation model trained since then has, at minimum, crawled GitHub repositories, blog posts, and papers that discuss — and in many cases directly reproduce — these exact problems and their solutions.

This isn't a hypothetical concern. Researchers have shown that you can detect contamination by testing whether a model can complete a benchmark problem's canonical solution given only a truncated prefix, and comparing perplexity on benchmark examples versus paraphrased or newly written equivalents. When contamination is present, Pass@k stops measuring "can this model write code" and starts measuring "did this model memorize this specific problem."

This matters enormously if you're choosing between models for a production feature. A model that scores 90% on HumanEval because it has memorized HumanEval will not score 90% on your company's actual coding tasks — refactoring a Django ORM query, writing a Terraform module for your specific VPC setup, or implementing a rate limiter that matches your existing codebase's conventions. None of that appeared in pretraining data in a memorizable form.

The practical mitigation is straightforward even if it's more work: build your own held-out evaluation set from tasks that look like what your product actually does, ideally written or heavily modified after the model's training cutoff, and never published publicly. Live benchmarks like LiveCodeBench address this at the research level by continuously scraping new competitive programming problems and tagging them with release dates, letting you filter to only problems released after a given model's cutoff. You can apply the same philosophy internally: date-stamp your eval set, rotate problems periodically, and treat any suspiciously high score with skepticism until you've verified it against genuinely novel tasks.

Functional Correctness Isn't the Only Kind of Correctness

Even when test coverage is solid and contamination isn't a factor, Pass@k still only captures functional correctness on the specific inputs you tested. It says nothing about several other properties that matter just as much in a real codebase:

• Security. A function that correctly returns query results while being trivially vulnerable to SQL injection will pass every functional test you throw at it. Pass@k has no concept of a security vulnerability unless you specifically write a test that tries to exploit one.
• Efficiency. A correct-but-quadratic solution to a problem that has an obvious linear solution passes the same tests as the optimal one. If your test cases use small inputs (which most benchmark tests do, for speed), you'll never notice.
• Idiomaticity and maintainability. Code that works but ignores your codebase's conventions — wrong error-handling patterns, no type hints in a strictly-typed codebase, reinventing a utility that already exists in your standard library — creates review burden and technical debt even though it "passes."
• Robustness to ambiguous specs. Real-world coding tasks (a Jira ticket, a Slack message from a PM) are rarely as precisely specified as a benchmark problem statement. A model's ability to ask clarifying questions, state assumptions, or handle underspecified requirements gracefully is invisible to Pass@k entirely, because Pass@k assumes the spec is complete and the only variable is code quality.

None of this means Pass@k is useless — it's a fast, cheap, reproducible signal, and it's genuinely good at catching gross functional failures. The mistake is treating it as sufficient rather than as one input among several.

Building a Layered Evaluation Harness

In practice, teams evaluating code-gen models or features for production use tend to converge on a layered approach, roughly in increasing order of cost and decreasing order of automatability:

1. Static analysis and linting. Run generated code through the same linters, type checkers, and security scanners (think ruff, mypy, bandit, semgrep) that you'd run in CI. This catches whole classes of problems — unused imports, type mismatches, obvious injection patterns — before you even get to execution.
2. Sandboxed execution against an extended test suite. Not just the benchmark's original asserts — your own expanded set covering edge cases, malformed inputs, and performance boundaries under a timeout.
3. Differential testing against reference implementations. Where possible, generate random or fuzzed inputs and compare the model's output against a known-correct reference implementation, rather than relying only on a fixed set of hand-written assertions.
4. Static + dynamic security scanning. For anything that touches user input, auth, or data access, run a dedicated security pass rather than folding it into general linting.
5. LLM-as-a-Judge for qualitative dimensions. For everything that resists a pass/fail unit test — readability, idiomatic style, whether the solution matches the spirit of an ambiguous request, whether error handling is sensible — use a strong LLM as a rater with an explicit rubric.
6. Human spot-checks on a sample. Even a well-designed automated harness benefits from periodic human review of a random sample, both to catch what the harness misses and to validate that the LLM judge's ratings correlate with actual human judgment.

Here's a simplified sketch of what a harness combining layers 1, 2, and 5 might look like for a single generated solution:

import subprocess
import json

def evaluate_solution(code: str, test_cases: list[dict], task_prompt: str) -> dict:
    results = {"lint_passed": None, "tests_passed": 0, "tests_total": len(test_cases)}

    # Layer 1: static lint check
    with open("candidate.py", "w") as f:
        f.write(code)
    lint = subprocess.run(
        ["ruff", "check", "candidate.py"],
        capture_output=True, text=True, timeout=10
    )
    results["lint_passed"] = lint.returncode == 0
    results["lint_issues"] = lint.stdout

    # Layer 2: extended test execution in a sandbox
    namespace = {}
    try:
        exec(code, namespace)
        for case in test_cases:
            fn = namespace.get(case["fn_name"])
            actual = fn(*case["args"])
            if actual == case["expected"]:
                results["tests_passed"] += 1
    except Exception as e:
        results["execution_error"] = str(e)

    return results

That handles the mechanical layers. The judge layer needs a different kind of scaffolding, which is worth walking through separately because it's the piece most teams get wrong on the first attempt.

Making LLM-as-a-Judge Actually Reliable

Using an LLM to grade another LLM's code sounds circular, and if you do it naively — "rate this code from 1 to 10" — you'll get noisy, inconsistent, and gameable results. But with the right constraints, LLM-as-a-Judge is genuinely one of the most useful tools for the qualitative dimensions that unit tests can't touch.

A few practices separate a judge that produces signal from one that produces noise:

• Give it a rubric, not a vibe check. Instead of "rate the code quality," break it into specific, answerable questions: Does this handle the empty-input case? Does the error handling match patterns used elsewhere in this codebase (paste an example)? Is there an obvious security issue in how user input is handled? Each question should be answerable with evidence the judge can point to in the code, not a subjective feeling.
• Show, don't just tell, the comparison baseline. When judging whether generated code is idiomatic for your codebase, give the judge two or three real examples from your repo alongside the candidate. Asking "is this idiomatic Python" in the abstract gets you generic style opinions; asking "does this match the patterns in these three files" gets you something actionable.
• Use pairwise comparison for ranking, absolute scoring for gating. LLM judges are considerably more reliable at answering "which of these two solutions is better" than at producing a stable absolute number. If you're comparing two models or two prompting strategies, prefer pairwise win-rates. If you're gating a PR ("is this good enough to auto-merge"), use a rubric with binary or low-cardinality answers per criterion rather than a single 1-10 score.
• Watch for self-preference and verbosity bias. Judge models tend to rate outputs from their own model family slightly more favorably, and tend to prefer longer, more verbose responses even when a terser one is equally correct. If you're evaluating multiple models, use a judge from a different family than any of the candidates where possible, and explicitly instruct the rubric to penalize unnecessary verbosity.
• Validate the judge against human labels periodically. Take a sample of judge-rated outputs, have a human engineer rate the same sample blind, and compute agreement. If agreement is low on a particular rubric dimension, that dimension's wording is probably ambiguous — fix the prompt, not the model.

A minimal rubric-driven judge call might look like this:

JUDGE_PROMPT = """
You are reviewing a code solution against a task and a rubric.
Answer each rubric item with PASS, FAIL, or N/A, and cite the
specific line(s) of code as evidence. Do not give an overall score.

Task:
{task_prompt}

Candidate code:
{code}

Rubric:
1. Handles empty/None inputs without crashing.
2. No unbounded memory growth for large inputs (explain reasoning).
3. Error messages are specific enough to debug from, not generic.
4. No obvious injection or unsafe deserialization patterns.
5. Naming and structure match typical idioms for this language.

Respond in JSON: {{"item_1": {{"verdict": ..., "evidence": ...}}, ...}}
"""

Parsing structured, per-criterion verdicts like this turns the judge from a black box into something you can audit, aggregate, and — critically — argue with when it's wrong.

Domain-Specific Evaluation: Beyond Generic Benchmarks

If your product generates code in a specific domain — SQL query generation, infrastructure-as-code, data pipeline transformations, front-end component scaffolding — generic benchmarks like HumanEval are close to irrelevant, because they're built almost entirely from self-contained algorithmic puzzles (reverse a string, check if a number is prime). Real production code-gen tasks look nothing like this: they involve multi-file context, existing dependencies, company-specific conventions, and integration with systems that don't exist in a benchmark's sandbox.

Building a domain-specific eval set is more work upfront but pays off fast:

• Mine your own bug tracker or PR history for real tasks that were completed by human engineers, and use the human-written solution as one reference point (not necessarily the only correct one).
• Include multi-file context in the prompt where your real tasks involve it — a benchmark that only ever asks for a single function in isolation will systematically favor models that are worse at using context from surrounding files.
• Weight your test suite toward the failure modes you've actually seen in production, not generic algorithmic edge cases. If your past incidents involved race conditions in async code, make sure your eval set has async tasks with concurrency tests, not just more sorting algorithms.
• Re-run your eval set every time you upgrade a model version or change a prompt template. Small changes upstream can shift performance on your specific distribution of tasks in ways that don't show up on public leaderboards at all.

Putting It Together: A Practical Checklist

When you're deciding whether a code-generating model or pipeline is good enough to ship, resist the urge to reduce the decision to a single Pass@k number. A more defensible evaluation covers:

• Functional correctness on an extended, edge-case-aware test suite you control — not just the benchmark's original asserts.
• An explicit check for benchmark contamination if you're relying on any public leaderboard numbers.
• Static analysis and security scanning as a baseline gate before anything else runs.
• Efficiency checks on inputs large enough to expose bad complexity, not just toy-sized test inputs.
• A rubric-based LLM-as-a-Judge pass for maintainability, idiomaticity, and alignment with ambiguous specs.
• Periodic human validation of both the automated test suite and the judge's agreement with real reviewers.
• A domain-specific task set mined from your own real engineering work, refreshed regularly.

Pass@k will keep showing up in papers and model cards because it's cheap, comparable, and not wrong — it's just incomplete. The teams that ship reliable code-generation features are the ones that treat it as the floor of an evaluation stack, not the whole stack. Static analysis catches what tests can't see structurally. Extended test suites catch what the original benchmark authors didn't think to check. And LLM-as-a-Judge fills the gap that no automated test can close: whether the code a model wrote is something a human would actually want to maintain.