Cost-Aware Evaluation: Balancing Eval Depth with API Spend

The Eval Bill Nobody Budgeted For

A team I worked with shipped a RAG assistant, wired up a 500-case golden set, and set their CI pipeline to run the full suite on every pull request. Three weeks later, the eval spend had quietly become the second-largest line item on their OpenAI invoice, right behind production traffic. Nobody had done anything wrong, exactly. Each individual eval run seemed cheap. But multiply a 500-case suite by two model calls per case (one to generate the response, one for an LLM-as-a-judge to score it), then multiply that by fifteen PRs a day, and you get a bill that looks like a second production environment.

This is the part of LLM evaluation that tutorials rarely mention: eval depth and API spend are in constant tension, and nobody hands you a formula for the tradeoff. Run a shallow eval and you ship regressions that a deeper suite would have caught. Run an exhaustive eval on every commit and you either bankrupt the project or train your team to skip the eval "just this once." Cost-aware evaluation is the discipline of designing eval suites that scale their depth to the stakes of the change being tested, so you get the confidence you need without paying for confidence you don't.

This article is a practical walkthrough of how to do that: how to categorize your evals by cost tier, how to build a tiered CI strategy, how to pick cheaper judges without losing signal, how to cache and reuse eval artifacts, and how to instrument spend so it never surprises you again.

Why Eval Costs Spiral Faster Than You Expect

The naive mental model of eval cost is "number of test cases times cost per call." The real cost model has several multipliers stacked on top of that, and each one is easy to add without noticing:

• The generation call. Every eval case needs the system under test to actually produce an output. If you're evaluating a multi-turn agent, this might mean five or ten model calls per case, not one.
• The judge call. If you're using LLM-as-a-judge, you're paying for a second model invocation per case, and judges often need larger context windows (the full transcript, the rubric, the reference answer) than the original task did.
• Retries and self-consistency. Judges are non-deterministic. Many teams run the judge two or three times per case and take a majority vote to reduce noise, tripling judge cost outright.
• Multi-dimensional scoring. A single case might be scored on faithfulness, relevance, tone, and safety separately, meaning four judge calls masquerading as "one eval."
• Frequency. Running the suite on every commit instead of once a day is an easy way to 20x your monthly spend without ever making a deliberate decision to do so.

Stack these and a suite that looks like "500 cases" is actually closer to 500 × (1 generation + 4 judge dimensions × 2 for self-consistency) = 4,500 model calls per run. If your team pushes ten PRs a day, that's 45,000 calls daily before anyone has touched production traffic. This is why cost-aware evaluation isn't a nice-to-have optimization — it's the difference between an eval practice that survives contact with a real budget and one that gets quietly disabled six weeks after launch.

Tier Your Evals by What They're Actually Protecting

The first move is to stop treating your eval suite as one monolithic thing. Split it into tiers based on what failure mode each tier catches and how often you need to catch it.

Tier 1 — Smoke tests (run on every commit, seconds to run). A handful of deterministic, cheap checks: does the output parse as valid JSON, does it avoid a banned-phrase list, does it stay under a token budget, does a known adversarial prompt get refused. These use no LLM judge at all — they're regex, schema validation, and string matching. Near-zero cost, and they catch the embarrassing regressions (broken JSON, an obvious jailbreak) before anything more expensive runs.

Tier 2 — Fast heuristic evals (run on every PR, a few dollars). A stratified sample of 30-50 cases pulled from your golden set, scored with cheap methods: exact-match or fuzzy-match for tasks with a known answer, embedding similarity for retrieval-style tasks, and a small, fast model (not your flagship) as judge for anything that needs semantic judgment. This tier exists to answer "did this PR obviously break something" without waiting for a full suite.

Tier 3 — Full regression suite (run nightly or pre-merge to main, tens of dollars). The complete golden set, scored with your production-grade judge model, across every dimension you care about. This is where you catch subtle regressions: a prompt change that improves conciseness but quietly degrades faithfulness on 8% of cases.

Tier 4 — Deep audit (run weekly or before major releases, potentially hundreds of dollars). Self-consistency judging (multiple judge passes with majority vote), human-in-the-loop spot checks on judge disagreements, and adversarial/red-team case generation. This is expensive by design — you're buying maximum confidence for a moment that matters, not routine coverage.

Here's a simplified version of what tiering looks like as config, so it's not just a policy document nobody reads:

eval_tiers:
  smoke:
    trigger: every_commit
    cases: all
    scorer: deterministic
    judge_model: none
    est_cost_per_run_usd: 0.00

  fast_heuristic:
    trigger: pull_request
    cases: sample_stratified
    sample_size: 40
    scorer: embedding_similarity
    judge_model: gpt-4o-mini
    est_cost_per_run_usd: 0.35

  full_regression:
    trigger: nightly
    cases: all
    scorer: llm_judge
    judge_model: gpt-4o
    dimensions: [faithfulness, relevance, safety]
    est_cost_per_run_usd: 22.00

  deep_audit:
    trigger: pre_release
    cases: all
    scorer: llm_judge_self_consistency
    judge_model: gpt-4o
    judge_passes: 3
    dimensions: [faithfulness, relevance, safety, tone]
    est_cost_per_run_usd: 140.00

The point of writing it this way is that cost becomes a first-class, reviewable field, not an emergent side effect discovered on the invoice.

Sampling: You Don't Need All 500 Cases Every Time

Most teams over-run their eval suite because "run the whole thing" feels safer than "run a sample." But if your golden set is well-constructed, a stratified sample gives you nearly the same signal at a fraction of the cost, especially for the fast, PR-level checks where you're looking for gross regressions, not subtle ones.

The key is stratification, not random sampling. If you randomly sample 40 cases from a 500-case set that's 60% easy cases and 10% edge cases, your sample will underrepresent exactly the cases most likely to reveal a regression. Instead, bucket your golden set by difficulty or category, then sample proportionally with a floor for rare-but-critical buckets.

import random
from collections import defaultdict

def stratified_sample(golden_set, sample_size, min_per_bucket=3):
    """
    golden_set: list of dicts, each with a 'category' key
    Returns a sample that preserves category proportions but
    guarantees at least `min_per_bucket` cases from every category,
    including rare ones like 'edge_case' or 'adversarial'.
    """
    buckets = defaultdict(list)
    for case in golden_set:
        buckets[case["category"]].append(case)

    total = len(golden_set)
    sample = []

    for category, cases in buckets.items():
        proportional_n = max(
            min_per_bucket,
            round(sample_size * len(cases) / total)
        )
        sample.extend(random.sample(cases, min(proportional_n, len(cases))))

    return sample[:sample_size] if len(sample) > sample_size else sample

# Example: a 500-case golden set, sampled down to 40 for PR-level checks
# golden_set = load_golden_set("evals/golden_set.jsonl")
# pr_sample = stratified_sample(golden_set, sample_size=40)

Re-run the full 500-case suite nightly so drift in the un-sampled cases still gets caught within 24 hours, rather than lingering for weeks. The combination — cheap stratified sample on every PR, full suite nightly — catches the vast majority of regressions at a fraction of the per-PR cost.

Picking the Right Judge for the Job

The single biggest cost lever in most eval pipelines is which model you use as the judge, and teams routinely default to their most expensive model out of habit rather than necessity. Not every judgment call needs your flagship model.

A useful mental split: objective judgments (did the answer contain the required fact, is the JSON schema valid, is the citation present) don't need an LLM judge at all — a rule-based or embedding-based check is faster, cheaper, and more consistent. Subjective-but-narrow judgments (is this tone appropriately professional, is this summary faithful to the source) often work fine with a smaller, cheaper model as judge, because the task is narrow enough that a smaller model's judgment correlates well with a larger one's. Subjective-and-nuanced judgments (is this legal explanation actually correct and not just fluent, does this medical-adjacent answer hedge appropriately) are where you actually need your strongest available judge model, because the failure mode you're guarding against is exactly the kind of subtle reasoning gap a smaller model would also miss.

The mistake to avoid is assuming judge quality scales linearly with judge cost for every task. It's worth periodically validating your cheap judge against your expensive one on a small calibration set — run both judges on the same 50 cases, compare agreement, and only "upgrade" the categories where they diverge meaningfully.

def calibrate_judge(cheap_judge, expensive_judge, calibration_set):
    """
    Runs both judges on the same cases and reports where they disagree,
    so you know which categories still need the expensive judge and
    which can safely downgrade.
    """
    disagreements = []
    for case in calibration_set:
        cheap_score = cheap_judge.score(case)
        expensive_score = expensive_judge.score(case)
        if abs(cheap_score - expensive_score) >= 2:  # on a 1-5 scale
            disagreements.append({
                "case_id": case["id"],
                "category": case["category"],
                "cheap_score": cheap_score,
                "expensive_score": expensive_score,
            })

    disagreement_rate_by_category = defaultdict(lambda: [0, 0])
    for case in calibration_set:
        disagreement_rate_by_category[case["category"]][1] += 1
    for d in disagreements:
        disagreement_rate_by_category[d["category"]][0] += 1

    return {
        category: hits / total
        for category, (hits, total) in disagreement_rate_by_category.items()
    }

# categories with high disagreement rates keep the expensive judge;
# categories with near-zero disagreement get downgraded to the cheap judge

This calibration step usually surprises teams: a cheap judge model often agrees with the expensive one on 90%+ of routine cases, and the disagreement concentrates in two or three categories you can name. That means you can run the expensive judge only on those categories and the cheap judge everywhere else, without eyeballing it — the data tells you where to spend.

Caching: The Free Lunch Most Teams Skip

If your eval harness re-runs generation and judging on every case even when nothing about that case or the model under test has changed, you're paying for the same answer twice. Deterministic caching keyed on a hash of (prompt, model version, parameters) means that re-running a suite after a documentation-only change or a retry after a flaky network error doesn't re-spend money on unchanged cases.

import hashlib
import json

def cache_key(prompt, model, params, case_id):
    payload = json.dumps(
        {"prompt": prompt, "model": model, "params": params, "case_id": case_id},
        sort_keys=True,
    )
    return hashlib.sha256(payload.encode()).hexdigest()

def get_or_generate(cache, prompt, model, params, case_id, generate_fn):
    key = cache_key(prompt, model, params, case_id)
    cached = cache.get(key)
    if cached is not None:
        return cached, True  # cache hit, no API call made

    result = generate_fn(prompt, model, params)
    cache.set(key, result)
    return result, False

The caveat: cache invalidation has to be deliberate. If you bump the system prompt, change the model version, or edit a golden-set case, the hash needs to change too, or you'll silently score against stale outputs. The safest approach is to include a version string for your prompt template and eval harness in the hash itself, so any meaningful change naturally busts the cache without you needing to remember to clear it manually.

Caching pairs especially well with the judge, not just the generation step. Judge calls are often the more expensive half of the pipeline (larger context, sometimes a stronger model), and judge outputs for a given (response, rubric) pair are just as cacheable as generation outputs.

Budgets as Guardrails, Not Afterthoughts

Once you've tiered your evals, sampled sensibly, picked appropriate judges, and cached aggressively, the last piece is making sure spend is visible before it becomes a surprise. This means setting an explicit budget per tier and failing loudly, not silently, when a run would exceed it.

class EvalBudgetGuard:
    def __init__(self, budget_usd, cost_per_call_estimate):
        self.budget_usd = budget_usd
        self.cost_per_call_estimate = cost_per_call_estimate
        self.spent_usd = 0.0

    def check_before_run(self, num_planned_calls):
        projected_cost = num_planned_calls * self.cost_per_call_estimate
        if self.spent_usd + projected_cost > self.budget_usd:
            raise RuntimeError(
                f"Eval run would cost an estimated ${projected_cost:.2f}, "
                f"pushing total spend to ${self.spent_usd + projected_cost:.2f}, "
                f"over the ${self.budget_usd:.2f} budget for this tier. "
                f"Reduce sample size or switch to a cheaper judge model."
            )
        return projected_cost

    def record_actual_cost(self, actual_usd):
        self.spent_usd += actual_usd


# guard = EvalBudgetGuard(budget_usd=25.00, cost_per_call_estimate=0.02)
# guard.check_before_run(num_planned_calls=500)  # raises if over budget

Wire this into CI so a budget breach fails the pipeline step with a clear message, the same way a failing test would, rather than quietly running up a bill that someone notices at month-end. Teams that do this well also track a rolling weekly eval spend number next to their eval pass-rate number on the same dashboard, so depth and cost are always considered together instead of cost being a surprise that shows up in a completely different conversation, usually with finance.

What to Cut First When the Budget Is Tight

When a budget cut is forced on you — and eventually one will be — cut in this order, because it preserves the most safety signal per dollar removed:

1. Reduce judge self-consistency passes first. Going from three judge passes to one loses some noise-reduction but keeps the core signal intact; it's the cheapest cut with the least information loss.
2. Shrink the nightly full-suite frequency before shrinking its coverage. Running the full 500-case suite every other night instead of every night halves that tier's cost while still catching drift within 48 hours.
3. Downgrade judge model tier-by-tier using your calibration data, not uniformly. Cut the categories where cheap and expensive judges already agree; keep the expensive judge exactly where the calibration step showed disagreement.
4. Shrink PR-level sample size last, since this is your fastest-feedback, most-frequent-use tier — the one that catches regressions before they reach main. If you must cut here, cut the sample size modestly (from 40 to 25 cases, say) rather than removing the tier entirely, since a fast_heuristic tier at zero cases means every regression sails through until the nightly run catches it, possibly a full day later.

Notice what's absent from this list: cutting the smoke tests. They're nearly free, and they're your last line of defense against the most embarrassing failures, so there's no version of a budget crunch where cutting them makes sense.

Putting It Together: A Realistic Weekly Rhythm

A team running the tiered approach described here typically settles into something like: smoke tests firing on every commit at effectively no cost, a 40-case stratified sample running on every PR for a few dollars, the full suite running nightly for the price of a nice lunch, and a deep audit with self-consistency judging running once before each release. The weekly spend is predictable, reviewable, and scales with how much the team is actually shipping rather than growing silently in the background.

The deeper shift this rhythm produces isn't just financial — it's cultural. When eval cost is visible and tiered, engineers stop treating "run the eval" as an expensive, occasional ritual reserved for big changes, and start treating it as a normal part of every PR, because the PR-level tier is cheap enough to run without a second thought. That habit, more than any single optimization in this article, is what actually catches regressions before they reach production. An eval suite that's too expensive to run often is, in practice, an eval suite that mostly doesn't run.

None of this replaces good judgment about what to measure in the first place — cost-awareness only pays off once you already know which dimensions matter for your task and have a LLM-as-a-Judge setup you trust enough to build a budget around. Get the judge right first; then make it affordable to run often.