A/B Testing Your Prompts

Prompt a/b testing is the practice of running two or more prompt variants against the same inputs, scoring the outputs on metrics that matter to your product, and picking the version that wins on evidence instead of gut feeling. Most teams skip this step and just ship whichever prompt "feels" better after a few manual tries in a playground. That works until the prompt touches real users, at which point small wording changes start moving conversion, support ticket volume, or hallucination rates in ways nobody predicted. This guide walks through a complete, runnable setup for testing prompts the way you would test a UI change or a pricing page.

Why Guessing at Prompts Stops Working

A single prompt change can shift several outcomes at once: response length, tone, factual accuracy, latency, and token cost. When you eyeball five outputs in a chat window, you are sampling a tiny, non-random slice of the input space and judging it with a brain that is already biased toward whichever version you wrote most recently. That is fine for early exploration. It falls apart the moment you have a customer support bot, a summarizer, or a coding assistant where a 5% regression in accuracy translates into real support load or real bugs.

Prompt a/b testing fixes this by making three things explicit that manual review leaves implicit: the input set, the scoring method, and the sample size needed to trust a result. Once those three are pinned down, comparing prompts becomes a repeatable engineering task rather than a vibe check.

What a Prompt A/B Test Actually Needs

Before writing any code, get four pieces in place:

• A fixed evaluation set. A list of representative inputs, ideally pulled from real production traffic or support logs, not hand-picked "nice" examples. Twenty to fifty inputs is a workable starting point; more if your traffic is highly varied.
• Two or more prompt variants that differ in exactly one dimension. Changing the system prompt's tone AND adding a new instruction AND reordering the few-shot examples in the same test tells you nothing about which change mattered.
• A scoring function. Either an automated check (does the output contain valid JSON, does it match a regex, is the length under a limit) or a model-graded rubric where a second LLM call scores the first model's output against criteria you define.
• A way to compare the two score distributions, not just the averages. A prompt that wins on average but has a fat tail of terrible outputs might be worse for your product than a consistently mediocre one.

Step 1: Freeze Your Evaluation Set

Pull real examples instead of inventing them. If you are testing a support-ticket triage prompt, export fifty real tickets. If you are testing a code-review prompt, pull fifty real pull request diffs. Store them as a plain JSON file so both variants run against the identical inputs every time.

[
  {"id": "case-001", "input": "My invoice shows a charge I don't recognize from last Tuesday."},
  {"id": "case-002", "input": "The app crashes every time I try to export a PDF on mobile."},
  {"id": "case-003", "input": "Can you explain why my subscription renewed early this month?"}
]

Keep this file under version control. Every prompt variant you test from now on runs against the same file, which is what makes the comparison fair. If you change the eval set later, treat that as a new experiment, not a continuation of the old one.

Step 2: Write the Variants as Data, Not Hardcoded Strings

Store prompts as named entries in a config file rather than burying them inside application code. This makes the diff between variant A and variant B visible in one place, and it means your test harness can loop over variants without touching business logic.

{
  "variant_a": {
    "system": "You are a support triage assistant. Classify the ticket into one of: billing, bug, account, other. Respond with only the category name."
  },
  "variant_b": {
    "system": "You are a support triage assistant. Read the ticket carefully, then respond with only the category name from this list: billing, bug, account, other. If the ticket mentions money, a charge, or a refund, prefer billing over other categories."
  }
}

Variant B adds a single disambiguation rule. That is the one thing this test is measuring. Resist the urge to also rewrite the tone or add examples in the same pass, or you will not know which change caused the result.

Step 3: Build the Test Runner

Below is a runnable Python harness using the Anthropic SDK. It loops over every eval case, runs both variants, and stores raw outputs for scoring. Swap in whichever model client you use; the structure stays the same.

import json
import time
import anthropic

client = anthropic.Anthropic()
MODEL = "claude-sonnet-4-5"

def load_json(path):
    with open(path) as f:
        return json.load(f)

def run_variant(system_prompt, user_input):
    response = client.messages.create(
        model=MODEL,
        max_tokens=200,
        system=system_prompt,
        messages=[{"role": "user", "content": user_input}]
    )
    return response.content[0].text.strip()

def run_test(eval_path, prompts_path, output_path):
    cases = load_json(eval_path)
    prompts = load_json(prompts_path)
    results = []

    for case in cases:
        row = {"id": case["id"], "input": case["input"]}
        for variant_name, variant in prompts.items():
            output = run_variant(variant["system"], case["input"])
            row[variant_name] = output
            time.sleep(0.2)
        results.append(row)
        print(f"done: {case['id']}")

    with open(output_path, "w") as f:
        json.dump(results, f, indent=2)

    return results

if __name__ == "__main__":
    run_test("eval_cases.json", "prompt_variants.json", "raw_results.json")

Run this once per test. It produces a raw_results.json file with every case's output from both variants side by side, which is the raw material for scoring.

Step 4: Score the Outputs

There are two honest ways to score: rule-based checks for tasks with a verifiable answer, and model-graded rubrics for tasks that need judgment.

For the triage example, you likely have the correct category for each historical ticket (support agents already labeled it when they closed it). That makes this a rule-based check: exact match against ground truth.

def score_exact_match(results, ground_truth):
    scores = {"variant_a": [], "variant_b": []}
    for row in results:
        truth = ground_truth[row["id"]]
        for variant in scores:
            predicted = row[variant].strip().lower()
            scores[variant].append(1 if predicted == truth.lower() else 0)
    return scores

For tasks without a clean ground truth, such as "is this customer reply empathetic and correct," use a model-graded rubric. Have a separate LLM call score each output on a fixed scale against explicit criteria, and always ask for a short justification alongside the score so you can spot-check disagreements.

GRADER_SYSTEM = """You are grading a customer support reply for quality.
Score from 1 to 5 on this rubric:
5: Accurate, empathetic, resolves the issue, correct tone.
3: Accurate but generic or missing empathy.
1: Inaccurate or unhelpful.
Respond as JSON: {"score": <int>, "reason": "<one sentence>"}"""

def grade_output(customer_message, reply):
    response = client.messages.create(
        model=MODEL,
        max_tokens=150,
        system=GRADER_SYSTEM,
        messages=[{
            "role": "user",
            "content": f"Customer message: {customer_message}\n\nReply: {reply}"
        }]
    )
    return json.loads(response.content[0].text)

Run the grader on both variants' outputs, using the same rubric and the same grading model each time. Log the raw score and reason for every row so you can audit any surprising result by hand later.

Step 5: Read the Results Without Fooling Yourself

Once you have per-case scores for each variant, resist the temptation to just compare averages. A few cheap checks catch most false conclusions:

• Look at the distribution, not just the mean. Print a quick histogram or at least the min, median, and max for each variant. A variant that wins on average but has more 1-out-of-5 scores than the other is riskier to ship.
• Check the win rate per case, not the aggregate score. Count how many individual cases variant B beat variant A on, tied on, and lost on. If it wins 60% of individual comparisons but the average is close, that is a more convincing signal than the average alone.
• Run a basic significance check for small samples. With evaluation sets under a hundred cases, a paired comparison (same input, two outputs) is far more sensitive than an unpaired one. A simple sign test works well here: count wins, losses, and ties per case, and check whether the win/loss split is more extreme than you would expect from a fair coin.

from math import comb

def sign_test(scores_a, scores_b):
    wins = losses = ties = 0
    for a, b in zip(scores_a, scores_b):
        if b > a:
            wins += 1
        elif b < a:
            losses += 1
        else:
            ties += 1

    n = wins + losses
    if n == 0:
        return {"wins": wins, "losses": losses, "ties": ties, "p_value": None}

    k = min(wins, losses)
    p_value = 2 * sum(comb(n, i) * (0.5 ** n) for i in range(k + 1))
    p_value = min(p_value, 1.0)

    return {"wins": wins, "losses": losses, "ties": ties, "p_value": round(p_value, 4)}

A p-value under 0.05 with a reasonably sized win/loss split (not 3 wins and 2 losses) is a reasonable bar for "this difference is probably real, not noise." With fewer than twenty non-tied cases, treat any result as directional and plan to widen the eval set before fully trusting it.

Step 6: Add Cost and Latency to the Comparison

A prompt that scores 5% higher on accuracy but doubles token usage or adds 800ms of latency is not automatically the winner. Log token counts and response time alongside quality scores for every call.

def run_variant_with_metrics(system_prompt, user_input):
    start = time.time()
    response = client.messages.create(
        model=MODEL,
        max_tokens=200,
        system=system_prompt,
        messages=[{"role": "user", "content": user_input}]
    )
    elapsed = time.time() - start
    return {
        "text": response.content[0].text.strip(),
        "input_tokens": response.usage.input_tokens,
        "output_tokens": response.usage.output_tokens,
        "latency_seconds": round(elapsed, 3)
    }

Report quality, cost, and latency together in your results summary. A three-line table per variant (accuracy, average tokens, average latency) is usually enough to make the tradeoff obvious to whoever signs off on shipping the change.

Step 7: Ship With a Rollback Plan

Once a variant wins on your offline eval set, do not flip 100% of production traffic to it immediately. Route a small percentage of live traffic to the new variant, log outcomes the same way you did in your offline test, and compare after a few days of real usage. Offline eval sets are a proxy for production; they are never a perfect match, especially for tasks where user phrasing drifts over time.

Keep both prompt versions in your config file even after you pick a winner. If the new variant's live metrics degrade, you want to flip back with a config change, not a redeploy.

Common Mistakes to Avoid

• Testing on inputs you wrote yourself. Synthetic examples tend to be cleaner and more polite than real user input, which flatters whichever prompt handles clean text well and hides failures on messy real-world phrasing.
• Changing more than one variable per test. If variant B has a new instruction, a different tone, and reordered examples, a win tells you nothing about which change to keep.
• Using the same model to grade its own output with no rubric. Model-graded scoring without explicit criteria tends to reward verbosity and confident phrasing rather than actual correctness. Always give the grader a concrete rubric and ask for a reason, not just a number.
• Ignoring ties. A large tie count usually means your eval set or scoring method is not sensitive enough to detect the difference you are testing for. Sharpen the rubric or add harder cases before trusting a result.
• Declaring a winner from ten cases. Small samples produce noisy win rates. Treat anything under twenty non-tied comparisons as a hint to investigate further, not a decision.

A Minimal End-to-End Workflow

Putting the pieces together, a full test run looks like this:

1. Export fifty to a hundred real inputs into eval_cases.json, with ground truth labels where available.
2. Define two prompt variants that differ in exactly one way, in prompt_variants.json.
3. Run the test harness to collect raw_results.json.
4. Score each row with a rule-based check or a rubric-based grader, storing per-case scores for both variants.
5. Run the sign test on paired scores, and separately compare token usage and latency.
6. If the winner clears the significance bar and the cost/latency tradeoff is acceptable, roll it out to a small percentage of live traffic before a full switch.

This is the same shape as any other controlled experiment: fixed inputs, isolated variable, explicit scoring, and a significance check before you trust the result. The only thing specific to prompts is that your "metric" often needs a second model call to compute, which is why the grading rubric step matters as much as the test harness itself.

FAQ

How many test cases do I need for a reliable prompt a/b test? Twenty to fifty is a reasonable starting point for rule-based scoring where the ground truth is unambiguous. For model-graded rubrics, aim higher, fifty to a hundred, since rubric scores are noisier than exact-match checks and you need more paired comparisons for the sign test to produce a meaningful p-value.

Can I use the same model to both generate and grade outputs? Yes, but give the grader a separate, explicit rubric and a distinct system prompt from the one being tested. Without a rubric, a model grading its own family of outputs tends to favor longer, more confident-sounding responses over accurate ones. Where possible, spot-check a sample of the grader's scores by hand to confirm it agrees with human judgment.

What's the difference between prompt a/b testing and general prompt evaluation? Prompt evaluation usually means scoring a single prompt against a benchmark to see if it's "good enough." Prompt a/b testing is specifically a paired comparison between two or more variants on the same inputs, designed to answer "which one is better," not "is this one acceptable." A/b testing is the right tool once you already have a working prompt and are deciding between a change and the status quo.

Should I test prompt variants in a playground or through the API? Through the API, with the exact system prompt, temperature, and max token settings your production code uses. Playground UIs often apply different defaults, and manual testing in a chat window makes it too easy to unconsciously nudge the conversation toward the answer you expect.

How do I handle non-deterministic outputs when comparing variants? Set temperature to a low, fixed value for the test run so outputs are as consistent as possible between runs, and consider running each case two or three times per variant to average out remaining randomness. Note the temperature and any other sampling settings in your results file so a rerun months later starts from the same conditions.

What if my two variants tie on most cases? Treat a high tie rate as a signal that either the eval set isn't varied enough to expose the difference, or the scoring method is too coarse. Try adding harder or more ambiguous cases to the eval set, or switch from a binary right/wrong check to a graded rubric that can detect smaller quality differences between otherwise-correct answers.