Comparing LLM Providers: A Fair Evaluation Framework

Every few weeks a new leaderboard makes the rounds, and every few weeks engineering teams make purchasing decisions based on it. Then three months later, someone on the team asks why the model that scored highest on MMLU keeps failing at the actual task the product needs — extracting structured data from messy customer emails, or holding a coherent multi-turn support conversation, or writing SQL against a gnarly internal schema. The answer is almost always the same: the benchmark measured something adjacent to the job, not the job itself. Comparing LLM providers is harder than comparing databases or cloud vendors because the thing you're buying is a probability distribution over text, not a fixed feature set. Two providers can both claim "94% on our internal eval" and produce wildly different results on your workload, because "our internal eval" is doing all the work in that sentence.

This article lays out a framework we teach in the AI Engineering track at teachyou.ai — one built from running comparisons across OpenAI, Anthropic, Google, and open-weight models in production systems, not from reading press releases. The goal isn't to tell you which provider wins. It's to give you a repeatable process so that when you do pick one, you know it was the right call for your workload, not just the one with the best marketing deck.

The Core Problem: Benchmarks Measure the Wrong Thing

Public benchmarks like MMLU, HumanEval, GSM8K, or the newer agentic suites like SWE-bench exist for a reason — they let researchers compare model generations over time using a shared yardstick. But they have three structural weaknesses when you try to use them for a procurement decision.

Contamination. Popular benchmarks leak into training data. A model that has seen benchmark questions (or close paraphrases) during pretraining will score well without necessarily generalizing to your novel inputs. You cannot verify contamination from outside a lab, which means every public leaderboard number should be treated as an upper bound, not a prediction.

Task mismatch. MMLU tests multiple-choice academic knowledge. Your product probably doesn't ask multiple-choice trivia questions. If you're building a customer support agent, a code review assistant, or a document summarizer, the skill that actually matters — following a system prompt precisely, staying grounded in provided context, refusing gracefully when it doesn't know — is barely represented in most public suites.

Aggregate scores hide variance. A single number like "82.3%" collapses hundreds of different failure modes into one digit. A model can score well on average while reliably failing on exactly the subcategory of inputs your users send most often — say, non-English names, ambiguous dates, or ten-column tables.

The fix isn't to ignore public benchmarks entirely. It's to treat them as a first filter, then build a second layer of evaluation that actually resembles your product.

Principle 1: Define "Fair" Before You Define "Better"

A comparison is only fair if every provider is evaluated under conditions that let it do its best work. This sounds obvious, but it's the single most common way LLM comparisons go wrong. Teams routinely:

• Use a prompt tuned for one model's quirks and run it unmodified against a competitor
• Compare a model at default temperature against another tuned to temperature 0
• Test one provider's flagship model against another's mid-tier model because that's what's in the docs example
• Ignore that one API supports structured outputs / JSON mode natively and the other requires a fragile regex parse

Here's a simple rule: if you wouldn't ship a prompt to production as-is, don't use it in a comparison. That means each provider gets its own prompt-engineering pass — not a shared prompt copy-pasted across three APIs. This is more work, but it's the difference between "which model is better" and "which model tolerates lazy prompting better," which is a different, much less useful question.

Concretely, before running any comparison, write down:

1. The exact task definition (input format, output format, success criteria)
2. The model tier being compared (flagship vs flagship, not flagship vs budget)
3. Whether you're testing raw completion quality, tool-calling reliability, latency, or cost — pick one primary axis per test run
4. The decoding parameters for each provider, tuned independently

Building an Evaluation Set That Reflects Reality

The single highest-leverage thing you can do is stop using generic benchmark data and start using your own. Here's the process we walk through with students building their first eval harness.

Start with production logs, not imagination. If you have any real user traffic — even from a prototype or a beta with twenty users — mine it. Real inputs have a long tail of weirdness that you will never think to write by hand: typos, code-switched languages, incomplete sentences, adversarial phrasing. If you don't have production data yet, get as close as possible by having colleagues who aren't on the engineering team try to break your prototype.

Stratify by difficulty and category. A flat pile of 50 examples tells you less than 50 examples labeled by category (easy/medium/hard, or by intent type). This lets you see not just an aggregate pass rate but where each provider's failures cluster.

Aim for at least 100-150 examples per capability you care about. Below that, differences between providers are usually statistical noise. This is the part teams skip because it's tedious, and it's exactly why so many comparisons produce contradictory results when repeated.

Version your eval set like code. Store it in a repo, tag releases, and never silently edit examples after you've published a comparison result. If a provider "cheats" a particular example, that's useful data — don't remove it, annotate it.

A minimal eval record might look like this:

{
  "id": "support-042",
  "category": "refund_policy",
  "difficulty": "hard",
  "input": "Hey, I bought this like 2 months ago but it broke last week, do I still get my money back or what",
  "context": "refund_policy_doc_v3.md",
  "expected_behavior": "Cite the 30-day window, explain the manufacturer warranty alternative, avoid promising a refund outright",
  "must_not_contain": ["I can process your refund", "guaranteed refund"]
}

Notice that "expected_behavior" isn't a single golden string. For open-ended generation, exact-match scoring is almost always wrong — you need rubric-based or model-graded evaluation, which we'll come back to.

The Four Axes That Actually Matter

Most teams collapse everything into "quality," but a fair comparison needs to separate at least four independent axes, because providers rarely win on all of them simultaneously.

Correctness/quality. Does the output do what was asked, accurately and completely? This is the axis most benchmarks target, and it's necessary but not sufficient.

Instruction-following fidelity. Given a system prompt with five constraints, does the model honor all five, or does it drift after constraint three? This matters enormously for agents and degrades differently across providers as context grows.

Latency and throughput consistency. A model that's excellent on average but has a fat tail of 8-second responses will wreck a real-time chat UX even if its median latency looks fine. Measure p50, p90, and p99, not just the average.

Cost per successful task, not cost per token. A cheaper-per-token model that needs three retries to get valid JSON is not actually cheaper. This is the metric that most cost comparisons get wrong — they compare list price per million tokens instead of computing the effective cost of a correct, usable output.

Here's a simple harness pattern for measuring the fourth axis, which is the one most articles skip:

import time
from dataclasses import dataclass

@dataclass
class TrialResult:
    provider: str
    success: bool
    retries: int
    input_tokens: int
    output_tokens: int
    latency_s: float

def cost_per_success(results: list[TrialResult], price_per_1k_in: float, price_per_1k_out: float) -> float:
    successes = [r for r in results if r.success]
    if not successes:
        return float("inf")

    total_cost = 0.0
    for r in results:
        # retries burn tokens too — count them against the final cost
        total_cost += (r.input_tokens / 1000) * price_per_1k_in
        total_cost += (r.output_tokens / 1000) * price_per_1k_out

    return total_cost / len(successes)

def run_trial(client, prompt: str, validator, max_retries: int = 2) -> TrialResult:
    retries = 0
    start = time.time()
    while retries <= max_retries:
        response = client.generate(prompt)
        if validator(response.text):
            return TrialResult(
                provider=client.name,
                success=True,
                retries=retries,
                input_tokens=response.input_tokens,
                output_tokens=response.output_tokens,
                latency_s=time.time() - start,
            )
        retries += 1
    return TrialResult(
        provider=client.name,
        success=False,
        retries=retries,
        input_tokens=response.input_tokens,
        output_tokens=response.output_tokens,
        latency_s=time.time() - start,
    )

Running this across providers with the same validator function gives you a number that maps directly to what you're actually paying for: a usable answer, not a token.

One nuance worth calling out: the "retries burn tokens too" comment in that code is doing more work than it looks like. Teams frequently benchmark cost using only the tokens from the winning attempt, silently discarding the tokens spent on failed attempts that preceded it. That's not an accounting error you can wave away — a provider that fails validation 40% of the time on the first try but recovers on retry will look artificially cheap if you only count the successful call. Always attribute every token spent in a trial, successful or not, to that trial's final cost.

It's also worth separately tracking *why* a trial failed validation, rather than lumping all failures into one bucket. A model that fails because it returns malformed JSON is a solvable problem — you can add a repair step, or switch to a provider's native structured-output mode. A model that fails because it fabricated the answer entirely is a much more serious problem for anything with legal, medical, or financial exposure. Tag your failures at the point of validation so the aggregate numbers don't flatten a formatting bug and a hallucination into the same statistic.

Statistical Rigor: Why Five Examples Prove Nothing

A pattern we see constantly from teams new to LLM evaluation: someone runs five or ten prompts against two providers, eyeballs the outputs, and declares a winner in a Slack thread. This is the LLM-comparison equivalent of A/B testing a website with four visitors — the sample size is too small for the result to mean anything, and the "winner" is frequently just noise. LLM outputs are stochastic even at temperature 0 in many provider implementations, due to floating-point non-determinism in batched inference, so a single run of a single prompt is one sample from a distribution, not a fixed data point.

A few practical habits fix most of this:

1. Run each example multiple times per provider (three to five repetitions is a reasonable default) and report the mean and variance, not just one pass/fail outcome.
2. Use a large enough sample to detect the effect size you care about. A 30-point gap might show up clearly in a hundred examples; a 3-point difference needs an order of magnitude more data and a proper paired statistical test, not an eyeball comparison of two averages.
3. Report confidence intervals, not point estimates, whenever the comparison informs a real budget decision. "Provider A scored 78% (72-84% at 95% CI) versus Provider B's 74% (68-80% at 95% CI)" tells you the two are statistically indistinguishable on this eval — a very different, and more honest, finding than "Provider A wins."
4. Re-run your comparison periodically. Providers update models behind the same API endpoint without warning far more often than most teams expect, so an unrefreshed comparison has a shelf life.

None of this requires a statistics background. It requires resisting the urge to declare victory after the first run that confirms what you already suspected.

Testing the Full Stack, Not Just the Model

A subtle failure mode in provider comparisons is testing the model in isolation when your product will never call it in isolation. Real systems wrap the model in retries, function-calling loops, RAG retrieval, output parsers, and safety filters — and providers differ meaningfully in how well they support that surrounding infrastructure, not just in raw generation quality. Before finalizing a comparison, check each provider's behavior on:

• Tool/function calling reliability. Does the model reliably emit well-formed calls to your defined tools, including correctly typed arguments, or does it hallucinate tool names under complex multi-tool prompts? This degrades differently across providers as the number of available tools grows past five or six.
• Context window behavior under real load, not the advertised maximum. A provider might advertise a 200K-token window, but recall quality for information placed mid-context ("lost in the middle") can vary substantially between providers at the same nominal size. Test retrieval at multiple positions, not just at the start.
• Streaming and partial-output handling. If your product streams to a UI, check whether structured output can be reliably parsed incrementally — this affects perceived latency even when raw generation speed is similar.
• Rate limits and failover under your expected concurrency, not a demo script's concurrency. Providers differ in how gracefully they degrade under burst traffic.
• Safety filter false-positive rate on your specific domain. A model that refuses legitimate requests in a sensitive-but-legal domain (medical information, security research) as often as it correctly refuses harmful ones is a poor fit regardless of its quality-benchmark score.

None of these show up in a leaderboard, and any one of them can singlehandedly determine whether a provider is viable for your product. It's also worth naming a few mistakes that quietly invalidate otherwise well-designed comparisons: testing on data the model may have memorized from public sources (Stack Overflow answers, textbook problems), letting the evaluation prompt leak provider identity ("As Claude, you should..."), citing a bare model name like "GPT-4" without the exact dated version string, and comparing raw per-token pricing without accounting for hidden reasoning tokens some providers bill for and others don't expose. Each of these is a silent contamination of the result, and each is checked in minutes once you know to look.

From Comparison to Rollout

Once you've run a fair comparison and picked a provider, the evaluation work becomes the foundation for ongoing quality monitoring rather than a one-time report. Run the new provider in shadow mode alongside your current one on live traffic before serving its output to users, so you catch real-world failure modes your offline eval missed. Gate any canary rollout on the same eval suite, re-run against the exact model version being deployed rather than the version you tested weeks earlier. Keep the losing provider's client code warm in your codebase — providers have outages and deprecate models, and multi-provider abstraction is cheap insurance. And continue sampling production outputs against the same rubric used in evaluation; this is usually the first place you'll notice a silent model update before it becomes a support ticket.

Controlling for Prompt Sensitivity

LLMs are notoriously sensitive to prompt phrasing, and this sensitivity varies by provider and even by model version. If you test Provider A with a prompt that happens to match patterns heavy in its training data, and test Provider B with the same literal string, you may be measuring prompt-format luck rather than capability.

The practical fix is to test with prompt variants, not a single prompt, and report the distribution rather than a point estimate. For a given task, write three to five semantically identical prompts that vary in:

• Instruction ordering (constraints first vs. task first)
• Formatting (markdown headers vs. plain paragraphs vs. XML-style tags)
• Verbosity (terse vs. explicit step-by-step instructions)

If a provider's score swings by 20+ points across semantically identical phrasings, that's a real finding — it tells you that provider requires more careful prompt engineering in production, which is a cost even if its best-case score is high. Report both the best-case and the worst-case number, not just whichever one looks best.

This also protects you from a common failure mode: a vendor's own published benchmark numbers usually come from prompts heavily optimized for their own model. Reproducing their number with your prompt is often impossible, and that gap is diagnostic, not a bug in your test.

LLM-as-a-Judge: Necessary, But Only With Guardrails

Once your eval set includes open-ended tasks — summarization, tone, reasoning explanations, multi-turn dialogue — exact string matching stops working. This is where LLM-as-a-Judge becomes essential: using a strong model to grade the outputs of the models under test against a rubric, rather than trying to hand-write a regex for every possible acceptable answer.

Done carelessly, LLM-as-a-Judge just relocates the bias problem instead of solving it. Judges have well-documented tendencies: they favor longer answers, they favor answers stylistically similar to their own outputs, and if the judge is made by the same lab as one of the candidates, that candidate has a structural home-field advantage. So the framework needs three guardrails.

Use a judge model from a different provider than any model under test, where feasible. If you're comparing three providers, a fourth, independent judge (or a panel of two different judges with disagreement flagged for human review) reduces self-preference bias substantially.

Write an explicit, structured rubric — never "rate this response 1-10." Vague scoring prompts produce noisy, unreproducible judgments. Score each dimension separately:

Rubric for support-response quality (score each 0-2, then sum):

1. Groundedness: Does the response only state facts present in the provided
   context document? (0 = fabricates policy, 1 = mostly grounded with minor
   overreach, 2 = fully grounded)

2. Constraint adherence: Does the response avoid every item in
   "must_not_contain" and follow the required tone? (0 = violates a hard
   constraint, 1 = follows tone loosely, 2 = fully compliant)

3. Completeness: Does the response address the user's actual question,
   not just a nearby topic? (0 = off-topic, 1 = partial, 2 = complete)

Total: 0-6. Report the score AND a one-sentence justification citing the
specific text that earned or lost points.

Validate the judge against human labels before trusting it. Take 30-50 examples, have a human score them with the same rubric, and compute agreement with the judge. If agreement is below roughly 80%, the rubric is too ambiguous or the judge model is too weak for this task — fix the rubric before you trust a single automated score at scale. This validation step is the one most teams skip, and it's the reason "we used LLM-as-a-judge" sometimes produces results nobody can defend under scrutiny.

Used this way, LLM-as-a-Judge isn't a shortcut around rigor — it's what makes rigorous evaluation of open-ended tasks affordable at all. Human grading alone doesn't scale to the hundreds of examples per category this framework calls for, and judge models, properly rubric-constrained and validated, get you 90% of the reliability at a fraction of the cost and time. That combination — real eval data, controlled prompting, multi-axis measurement, and a validated judge — is what separates a defensible provider comparison from a leaderboard screenshot, and it's exactly the kind of evaluation engineering we build hands-on in the AI Engineering course at teachyou.ai.