Pairwise Model Comparison for LLM Selection

Pairwise llm comparison means showing a judge (human or model) two outputs side by side and asking which one is better, instead of asking it to rate each output alone on a 1-10 scale. You run enough of these head-to-head matchups, feed the results into a rating system like Elo or Bradley-Terry, and get a ranked list of models or prompts that is far more stable than averaging absolute scores. This article walks through why pointwise scoring falls apart at scale, how to build a pairwise evaluation harness in Python, and how to turn the results into a leaderboard you can defend.

If you are choosing between two or three candidate models for a production feature, coding assistant, or RAG pipeline, pairwise comparison is the fastest way to get a defensible answer instead of guessing from vibes or a single demo prompt.

Why pointwise scoring breaks down

The obvious way to evaluate an LLM is to ask a judge model to rate each response on a scale, say 1 to 10, then average the scores across a test set. This is called pointwise or absolute scoring. It is simple to implement and it is also the source of most bad model-selection decisions.

The core problem is that "quality" has no fixed anchor. If a judge model sees only one response, it has to invent a scale on the fly. That scale drifts across the run: the judge might call a decent answer a 7 early in the eval, then call an equally decent answer a 5 later because the previous ten examples were unusually strong. This is the same anchoring effect that shows up in human grading, and LLM judges inherit it because they are trained on human-labeled data.

Concretely, pointwise scoring has three recurring failure modes:

• Scale drift. Scores are not comparable across batches, sessions, or even different orderings of the same batch.
• Score compression. Judges cluster scores in a narrow band (commonly 6-9 out of 10), which flattens real quality differences and makes statistical comparison between two models weak.
• Poor correlation with preference. A study of judge behavior consistently finds that asking "which is better, A or B" produces judgments that agree with human preference more often than asking "rate A" and "rate B" separately and comparing the numbers.

Pairwise comparison sidesteps all three. The judge only has to answer a relative question: given this prompt, which of these two responses would a user prefer? That is a much easier and more consistent judgment than picking an absolute number, and it maps directly to what you actually care about: would switching models make your users' experience better or worse.

The basic pairwise loop

At its simplest, pairwise evaluation is:

1. Take a set of prompts representative of your real workload.
2. Generate a response from Model A and a response from Model B for each prompt.
3. Show a judge the prompt and both responses (in a randomized order) and ask which is better, or if it's a tie.
4. Aggregate the win/loss/tie counts into a win rate, or feed them into a rating system for more than two models.

Here is a minimal harness using the Anthropic Python SDK as the judge. It works the same way if you swap in a different judge model, the pattern does not depend on which vendor you use.

import random
from dataclasses import dataclass
from anthropic import Anthropic

client = Anthropic()
JUDGE_MODEL = "claude-sonnet-4-5"

@dataclass
class PairwiseResult:
    prompt: str
    winner: str  # "A", "B", or "tie"
    raw_judgment: str

JUDGE_PROMPT = """You are comparing two AI responses to the same user prompt.
Judge only on correctness, helpfulness, and clarity. Ignore length unless
verbosity actually hurts clarity.

User prompt:
{prompt}

Response A:
{response_a}

Response B:
{response_b}

Reply with exactly one line in this format:
WINNER: A
or
WINNER: B
or
WINNER: TIE

Then on a new line, give a one-sentence reason.
"""

def judge_pair(prompt: str, response_a: str, response_b: str) -> PairwiseResult:
    # Randomize position to cancel out position bias (see below)
    flipped = random.random() < 0.5
    first, second = (response_b, response_a) if flipped else (response_a, response_b)

    text = JUDGE_PROMPT.format(prompt=prompt, response_a=first, response_b=second)
    reply = client.messages.create(
        model=JUDGE_MODEL,
        max_tokens=200,
        messages=[{"role": "user", "content": text}],
    ).content[0].text

    first_line = reply.strip().splitlines()[0].upper()
    if "TIE" in first_line:
        winner = "tie"
    elif "WINNER: A" in first_line:
        winner = "B" if flipped else "A"
    elif "WINNER: B" in first_line:
        winner = "A" if flipped else "B"
    else:
        winner = "tie"  # fail safe on unparseable output

    return PairwiseResult(prompt=prompt, winner=winner, raw_judgment=reply)

Run this over your prompt set, tally the wins, and you already have something more useful than a pointwise average: a win rate for Model A vs Model B on your actual workload.

def run_pairwise_eval(prompts, responses_a, responses_b):
    results = [judge_pair(p, a, b) for p, a, b in zip(prompts, responses_a, responses_b)]
    wins_a = sum(1 for r in results if r.winner == "A")
    wins_b = sum(1 for r in results if r.winner == "B")
    ties = sum(1 for r in results if r.winner == "tie")
    total = len(results)
    print(f"A wins: {wins_a}/{total} ({wins_a/total:.1%})")
    print(f"B wins: {wins_b}/{total} ({wins_b/total:.1%})")
    print(f"Ties:   {ties}/{total} ({ties/total:.1%})")
    return results

Fixing position bias

The randomization in judge_pair above is not optional decoration, it is the single most important control in a pairwise harness. LLM judges reliably favor whichever response appears first (or, less commonly, second) in the prompt, independent of actual quality. This is called position bias and it can swing win rates by 10-20 percentage points if you do not correct for it.

The fix is straightforward: run every pair twice, once with A first and once with B first, and only count a win if the judge picks the same response both times. If the judge flips its answer depending on order, treat that comparison as a tie, because the judge effectively had no real preference.

def judge_pair_debiased(prompt: str, response_a: str, response_b: str) -> str:
    # Run in both orders
    result_ab = judge_pair(prompt, response_a, response_b)
    result_ba = judge_pair(prompt, response_a, response_b)  # random() re-flips internally

    if result_ab.winner == result_ba.winner and result_ab.winner != "tie":
        return result_ab.winner
    return "tie"

This doubles your judge API calls but it is the difference between a leaderboard that means something and one that just measures which model's outputs happen to render first in your template.

A second, cheaper mitigation: if your judge model supports it, ask it to output the reasoning before the verdict (chain-of-thought before the final answer), which measurably reduces position bias compared to verdict-first prompting. Combine both if you can afford the extra tokens.

Length bias and other judge quirks

Position bias is not the only systematic error. LLM judges also tend to:

• Favor longer responses, even when the extra length is padding rather than substance. Mitigate by instructing the judge explicitly to ignore length unless it affects clarity (as in the prompt above), and periodically spot-check with a human on a sample where the lengths differ a lot.
• Favor their own family's writing style. A judge model tends to rate outputs that "sound like itself" more highly. If you are comparing your own model family against a competitor, run at least part of the eval with a judge from a third, unrelated family to sanity-check.
• Struggle with domain-specific correctness (math, code execution, legal citations) if the task requires verification the judge cannot do from the text alone. For code, always execute test cases and inject pass/fail results into the judge prompt rather than trusting the judge to eyeball correctness.

None of these fully disappear no matter how you word the prompt. The practical answer is to treat the LLM judge as a fast approximation and periodically calibrate it against a small set of human-labeled pairs, checking agreement rate rather than assuming the judge is ground truth.

Scaling beyond two models: Bradley-Terry and Elo

Win rates work fine for a head-to-head between two candidates. Once you are comparing three or more models, or you want to fold in results from many different pairings over time, you want a rating system that turns win/loss records into a single comparable number per model. Two standard choices:

Elo (the same system used for chess ratings, and by public LLM leaderboards) updates each model's rating after every match based on the surprise of the outcome: beating a much stronger model earns more points than beating a weaker one.

def update_elo(rating_a, rating_b, winner, k=32):
    expected_a = 1 / (1 + 10 ** ((rating_b - rating_a) / 400))
    expected_b = 1 - expected_a

    if winner == "A":
        score_a, score_b = 1, 0
    elif winner == "B":
        score_a, score_b = 0, 1
    else:
        score_a, score_b = 0.5, 0.5

    new_rating_a = rating_a + k * (score_a - expected_a)
    new_rating_b = rating_b + k * (score_b - expected_b)
    return new_rating_a, new_rating_b

Seed every model at 1500, replay your match results through update_elo in order, and you get ratings you can rank directly. Elo is simple but order-dependent: shuffle the match sequence a few times and average the resulting ratings to reduce noise, especially with small sample sizes.

Bradley-Terry is the model most public LLM arenas actually fit under the hood, because it estimates each model's win probability from the full set of pairwise outcomes at once, rather than updating sequentially match by match. It is more stable than Elo when your match counts are uneven across models (which they usually are), because it does not depend on the order matches happened to run in.

import numpy as np
from scipy.optimize import minimize

def fit_bradley_terry(models, match_results):
    """
    match_results: list of (model_a, model_b, winner) where winner is
    model_a, model_b, or None for a tie.
    """
    idx = {m: i for i, m in enumerate(models)}
    n = len(models)

    def neg_log_likelihood(log_strength):
        strength = np.exp(log_strength)
        ll = 0.0
        for a, b, winner in match_results:
            if winner is None:
                continue  # drop ties, or split-weight them if you prefer
            i, j = idx[a], idx[b]
            p_a_beats_b = strength[i] / (strength[i] + strength[j])
            ll += np.log(p_a_beats_b if winner == a else 1 - p_a_beats_b)
        return -ll

    result = minimize(neg_log_likelihood, x0=np.zeros(n), method="BFGS")
    strengths = np.exp(result.x)
    strengths = strengths / strengths.sum()  # normalize
    return dict(zip(models, strengths))

For a small model-selection project (say, deciding between 3-4 candidates for one feature), Elo with a shuffled replay is good enough and much easier to explain to a non-technical stakeholder. For anything you plan to run repeatedly as a standing leaderboard across many models and prompt versions, Bradley-Terry is worth the extra setup because it handles sparse, uneven match data correctly.

Building a real evaluation set

The rating math is the easy part. The harder part, and the part that actually determines whether your conclusion is right, is the prompt set you run the comparison on.

• Sample from real traffic, not invented examples. Pull 50-200 real prompts (or realistic paraphrases if you cannot use production data directly) from the actual use case: support tickets, code review comments, RAG queries, whatever your feature does. A model that wins on textbook examples can lose badly on the messy, ambiguous inputs users actually send.
• Stratify by difficulty and category. Don't let one easy category dominate the sample. If half your prompts are simple factual lookups, the eval will mostly measure who is better at simple factual lookups.
• Include edge cases deliberately: empty context, contradictory instructions, adversarial phrasing, requests that should be refused. A model that wins the average case but fails badly on edge cases may be the wrong choice for a production system with real users.
• Keep the set fixed across comparisons. If you change the prompt set between evaluating Model A and Model B, you are no longer comparing models, you are comparing prompt sets. Version the eval set the same way you version code.

A reasonable minimum for a model-selection decision that will affect production is 100 prompts with the debiased two-pass judging described above, which means 200 judge calls per model pair. That is inexpensive relative to the cost of shipping the wrong model.

Reading statistical significance into the result

A 55%-45% win rate over 100 prompts is not a strong signal, it is close to noise. Before you act on a pairwise result, run a quick significance check. A binomial test against the null hypothesis of a 50/50 split is enough for most model-selection decisions:

from scipy.stats import binomtest

def significance_check(wins_a, wins_b):
    total_decisive = wins_a + wins_b  # excluding ties
    if total_decisive == 0:
        return None
    result = binomtest(wins_a, total_decisive, p=0.5)
    return {
        "win_rate_a": wins_a / total_decisive,
        "p_value": result.pvalue,
        "significant_at_05": result.pvalue < 0.05,
    }

If the p-value is above 0.05, treat the two models as statistically tied on this eval set and decide on secondary factors instead, latency, cost per token, context window, or how well each integrates with your existing tooling. Do not round a 52%-48% win rate up to "Model A is better" and ship it.

Putting it together as a leaderboard

Once the harness exists, extending it from a one-off A-vs-B decision to a standing leaderboard across every model you evaluate is mostly bookkeeping: store each match result (prompt id, model A, model B, winner, judge reasoning, timestamp) in a table, refit Bradley-Terry whenever new matches come in, and expose the current ratings alongside sample size per model so nobody mistakes a 3-match rating for a settled result.

import sqlite3

def log_match(db_path, prompt_id, model_a, model_b, winner, reasoning):
    conn = sqlite3.connect(db_path)
    conn.execute(
        """CREATE TABLE IF NOT EXISTS matches (
            prompt_id TEXT, model_a TEXT, model_b TEXT,
            winner TEXT, reasoning TEXT, created_at TEXT DEFAULT CURRENT_TIMESTAMP
        )"""
    )
    conn.execute(
        "INSERT INTO matches VALUES (?, ?, ?, ?, ?, CURRENT_TIMESTAMP)",
        (prompt_id, model_a, model_b, winner, reasoning),
    )
    conn.commit()
    conn.close()

This is deliberately low-tech. You do not need a dedicated eval platform to get value from pairwise comparison, a SQLite table and the two functions above cover most single-team use cases. Reach for a hosted eval tool once you need multi-user review, dataset versioning at scale, or a shared dashboard across teams.

FAQ

Is pairwise comparison always better than pointwise scoring? Not for every task. If you need an absolute quality bar independent of any competitor, for example "does this response violate our content policy: yes or no," pointwise (or better, a binary classifier) is the right tool. Pairwise comparison is specifically for the question "which of these options is better," which is exactly the shape of a model-selection decision.

How many prompts do I need for a reliable result? There is no fixed number, it depends on how close the two models are. If one model is clearly stronger, even 30-50 prompts will show a lopsided, statistically significant win rate. If the models are close, you may need several hundred pairs to reach significance. Run the binomial significance check as you go and stop once you have a stable, significant result rather than picking a sample size up front.

Can I use the same model as both the judge and one of the candidates? You can, but be aware of self-preference bias: judges tend to rate outputs from their own model family slightly higher, even when the underlying quality is equal. If one of your candidates shares a lineage with your judge model, cross-check a subset of results with a judge from a different provider before trusting the leaderboard.

Do I need a human in the loop at all? Yes, at least for calibration. Take a sample of 20-30 pairwise judgments the LLM judge made and have a human re-judge them blind. If human-judge agreement is high (roughly 80%+), you can trust the LLM judge for the bulk of the eval. If agreement is low, the judge prompt needs work before you scale up the automated run.

What's the difference between this and A/B testing in production? Pairwise LLM comparison happens offline against a fixed prompt set before you ship anything. Production A/B testing measures real user behavior (click-through, task completion, retention) after you've already deployed both variants to live traffic. Use pairwise comparison to narrow candidates down to one or two finalists, then use production A/B testing to confirm the choice against real user outcomes, since judge preference and actual user preference don't always match perfectly.

Should ties count as half a win for each model? For Bradley-Terry fitting, the cleanest approach is to drop ties entirely and only fit on decisive outcomes, since the standard Bradley-Terry model doesn't have a native tie term (there are tie-aware variants like Rao-Kupper if you need them). For a simple win-rate readout, report tie rate separately rather than folding it into either side's score, a high tie rate is itself useful information: it tells you the two models are close enough that other factors like cost or latency should decide.