A/B Testing LLM Models in Production

LLM model A/B testing is the practice of routing a fraction of live traffic to a candidate model, comparing it against your current model on real metrics, and promoting the winner only when the data says so. You do it because offline evals never fully predict production behavior: prompts drift, users ask things your test set never covered, and a model that scores higher on a benchmark can still be slower, pricier, or worse at your one task that matters. This guide shows how to build LLM model A/B testing end to end with deterministic bucketing, honest logging, and a stats check you can defend in a review.

Why offline evals are not enough

Offline evaluation is necessary but it is a lab. You curate a few hundred examples, run both models, and eyeball the scores. That catches obvious regressions. It misses everything about the live distribution: the long tail of weird inputs, the interaction with retrieval context that changes hourly, the way real users react to a slightly different tone.

LLM model A/B testing closes that gap by measuring on the exact traffic you care about. The tradeoff is that production experiments are slower, noisier, and can hurt real users if the candidate is bad. So the whole discipline is about limiting blast radius while gathering enough signal to decide. Three rules keep you honest:

• Never route a user to a different model mid-session. Consistency beats curiosity.
• Log the inputs and outputs you need to reconstruct any decision later.
• Decide the metric and the sample size before you look at results, not after.

The core loop of LLM model A/B testing

Every production model experiment has the same five parts, no matter the framework:

1. Assignment: decide which variant a request gets, deterministically.
2. Invocation: call the assigned model with the same prompt and params.
3. Logging: record variant, latency, tokens, cost, and outcome.
4. Aggregation: roll up per-variant metrics over a window.
5. Decision: run a stats test, then promote, hold, or roll back.

Get assignment and logging right and the rest is arithmetic. Get them wrong and no amount of dashboards will save you.

Deterministic bucketing so a user always sees one model

The single most common mistake is random assignment per request. If you call random() on every request, one user hits model A, then B, then A again. Their experience is incoherent and your per-user metrics are meaningless. Bucket on a stable key instead: user id, account id, or session id. Hash the key, take it modulo a large number, and compare against your rollout percentage.

import hashlib

def bucket(key: str, experiment: str, buckets: int = 10_000) -> int:
    # Salt with the experiment name so two experiments do not
    # correlate their assignments on the same user.
    digest = hashlib.sha256(f"{experiment}:{key}".encode()).hexdigest()
    return int(digest, 16) % buckets

def assign_variant(user_id: str, experiment: str, rollout_pct: float) -> str:
    # rollout_pct is the share going to the candidate, e.g. 0.10 for 10%.
    threshold = int(rollout_pct * 10_000)
    return "candidate" if bucket(user_id, experiment) < threshold else "control"

Two properties matter here. First, the same user always lands in the same variant for the life of the experiment, so their sessions stay consistent. Second, salting the hash with the experiment name means overlapping experiments do not accidentally assign the same users to the same side every time, which would confound your results.

To widen the rollout you just raise rollout_pct. Because the hash is stable, a user in the candidate bucket at 10 percent stays in the candidate bucket at 20 percent. You never reshuffle assignments, you only add users.

A minimal router you can drop into a service

Here is a small router that ties assignment, invocation, and logging together. It is provider-agnostic: you pass in a call_model function so the same harness works whether you are on Claude, GPT, Gemini, or a self-hosted model. The point is the structure around the call, not the SDK.

import time
import uuid

MODELS = {
    "control": "current-production-model",
    "candidate": "new-candidate-model",
}

def handle_request(user_id, prompt, call_model, log_event, experiment="q3-model-swap"):
    variant = assign_variant(user_id, experiment, rollout_pct=0.10)
    model_id = MODELS[variant]
    request_id = str(uuid.uuid4())

    start = time.monotonic()
    error = None
    try:
        result = call_model(model_id=model_id, prompt=prompt)
        text = result["text"]
        input_tokens = result["input_tokens"]
        output_tokens = result["output_tokens"]
    except Exception as exc:
        error = str(exc)
        text, input_tokens, output_tokens = None, 0, 0

    latency_ms = (time.monotonic() - start) * 1000

    log_event({
        "request_id": request_id,
        "experiment": experiment,
        "variant": variant,
        "model_id": model_id,
        "user_id": user_id,
        "latency_ms": round(latency_ms, 1),
        "input_tokens": input_tokens,
        "output_tokens": output_tokens,
        "error": error,
        "ts": time.time(),
    })
    return text, request_id

Notice what is logged: the variant, the exact model id, token counts, latency, and any error. That is the raw material for every metric you will compute. The request_id is the join key that lets you attach a later outcome (a thumbs-up, a resolved ticket, a completed purchase) back to this exact call.

Pick metrics before you start

An experiment without a pre-registered metric is a fishing trip. You will find something that looks better and convince yourself it matters. Decide up front what "better" means and split it into three tiers.

• Primary metric: the one thing that decides the experiment. For a support bot it might be resolution rate. For a coding assistant, the accepted-suggestion rate. Pick one.
• Guardrail metrics: things that must not get worse even if the primary improves. Almost always p95 latency and cost per request. A candidate that lifts quality but doubles latency is usually a no.
• Diagnostic metrics: refusal rate, empty-response rate, output length, tool-call success. These explain why the primary moved.

Cost and latency you get for free from the log above. Quality is the hard part, because for most LLM tasks there is no label at request time.

Measuring quality when there is no ground truth

You have three practical options, in rough order of trust.

Direct user signal is the gold standard. A thumbs-up, a copied answer, a completed checkout, a ticket that did not get reopened. Attach it to the request_id so it flows into the same table as the variant. It is sparse and delayed but it is real.

Implicit behavioral signal is the next best. Did the user rephrase the same question three times (bad)? Did the conversation end quickly after the answer (often good for support, ambiguous for chat)? Did they accept the code suggestion? These are proxies, so treat them as diagnostics, not verdicts.

LLM-as-judge fills the gap where human labels are too sparse. You take a sample of production request-response pairs from both variants, strip the variant label, and ask a separate judge model to score each one against a rubric. Blinding matters: the judge must not know which model produced the answer, or you bake in bias.

JUDGE_RUBRIC = """You are grading a support answer. Score 1-5 on:
- Correctness: is the factual claim right?
- Completeness: does it fully address the question?
- Tone: is it clear and professional?
Return only JSON: {"correctness": n, "completeness": n, "tone": n}."""

def judge(call_model, question, answer, judge_model="a-strong-eval-model"):
    prompt = f"{JUDGE_RUBRIC}\n\nQuestion:\n{question}\n\nAnswer:\n{answer}"
    result = call_model(model_id=judge_model, prompt=prompt)
    return parse_json(result["text"])

Two cautions with LLM-as-judge. Judges have known biases: they tend to prefer longer answers and answers that echo their own style, so a candidate from the same family as the judge can score unfairly high. Mitigate by keeping the judge fixed across variants, randomizing the order when comparing pairs, and periodically checking a slice of judge scores against human labels to confirm the judge still tracks reality.

Running the numbers without fooling yourself

Once data is flowing, resist the urge to check hourly and stop the moment the candidate looks ahead. That is peeking, and it inflates false positives badly, because with enough looks a random walk will cross your threshold eventually. Two defenses.

First, compute the sample size you need before you start, based on the smallest improvement worth shipping (the minimum detectable effect). If lifting resolution rate by less than two points would not change any decision, do not power the test to detect half a point. A rough power calculation for a rate metric:

from math import sqrt
from statistics import NormalDist

def sample_size_per_arm(baseline_rate, min_detectable_effect,
                        alpha=0.05, power=0.80):
    p1 = baseline_rate
    p2 = baseline_rate + min_detectable_effect
    z_alpha = NormalDist().inv_cdf(1 - alpha / 2)
    z_beta = NormalDist().inv_cdf(power)
    p_bar = (p1 + p2) / 2
    numerator = (z_alpha * sqrt(2 * p_bar * (1 - p_bar))
                 + z_beta * sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
    return int(numerator / (min_detectable_effect ** 2)) + 1

# e.g. baseline 0.60 resolution, want to detect a 3-point lift:
print(sample_size_per_arm(0.60, 0.03))

Second, when you do compare, use a real test rather than a raw difference of averages. For a binary metric like resolution rate, a two-proportion z-test gives you a p-value and a confidence interval on the lift.

def two_proportion_ztest(success_a, n_a, success_b, n_b):
    p_a = success_a / n_a
    p_b = success_b / n_b
    p_pool = (success_a + success_b) / (n_a + n_b)
    se = sqrt(p_pool * (1 - p_pool) * (1 / n_a + 1 / n_b))
    z = (p_b - p_a) / se
    p_value = 2 * (1 - NormalDist().cdf(abs(z)))
    return {"lift": p_b - p_a, "z": z, "p_value": p_value}

If the p-value clears your threshold and the lift is large enough to matter, and the guardrails held, you have a winner. If not, you either keep collecting to reach the planned sample size or you stop and conclude no difference. Both are legitimate outcomes. "No difference" is useful: if the cheaper, faster candidate is statistically tied with control on quality, ship it for the cost win alone.

For continuous metrics like latency, compare percentiles, not means. Averages hide tail blowups. Track p50, p95, and p99 per variant and treat a p95 regression as a guardrail breach even if the mean looks fine.

Rollout mechanics: canary, ramp, and rollback

Do not jump from 0 to 50 percent. Ramp in stages and watch the guardrails at each step.

• Start at a 1 to 5 percent canary. This is a smoke test for crashes, timeouts, and cost blowups, not a quality read. Sit here for a day.
• Move to 10 percent if guardrails hold. Now you are gathering quality signal.
• Ramp to 50 percent once the primary metric trends positive and the sample is approaching your target.
• Promote to 100 percent when the stats clear, then keep the old model wired for one more cycle in case you need to revert.

Make rollback a config change, not a deploy. Keep rollout_pct and the model ids in a config store or feature-flag service so you can set the candidate to 0 percent in seconds. If you have to ship code to roll back, you will hesitate at exactly the wrong moment.

Instrumentation and tooling

You do not have to build the plumbing from scratch. Feature-flag platforms such as LaunchDarkly, Statsig, GrowthBook, or Unleash give you percentage rollouts and stable bucketing out of the box; you supply the model-specific logging. On the LLM observability side, tools like Langfuse, Helicone, Arize Phoenix, and Braintrust capture prompts, responses, token counts, latency, and cost, and several support experiment or variant tagging so you can slice metrics by arm directly.

Whatever you use, the non-negotiables are the same: stable bucketing on a user key, a per-request log with variant and cost and latency, and a way to attach delayed outcomes back to the original request_id. If your stack has those three, you can run LLM model A/B testing with a spreadsheet and the functions above.

Common failure modes

A few traps catch teams repeatedly.

• Prompt drift between arms. If the candidate secretly gets a tweaked prompt or different temperature, you are testing two things at once and cannot attribute the result. Hold everything constant except the model id.
• Sample ratio mismatch. If you set 10 percent but see 7 percent of traffic in the candidate, your assignment is buggy or something is dropping requests. Check the split before you trust any metric.
• Novelty and learning effects. Users react to change itself. A dip or spike in the first days can be adjustment, not the model. Let the experiment run past the initial reaction.
• Mixing experiments. Two overlapping model tests on the same users interfere. Salt each experiment's hash and, where you can, keep users in one experiment at a time.
• Ignoring cost variance. Output length varies by model, so cost per request can swing even at the same price per token. Log token counts, not just estimated dollars, so you can see why cost moved.

A concrete end-to-end example

Suppose you run a support assistant and want to test a newer model. Your primary metric is resolution rate, defined as a conversation that got a thumbs-up and was not reopened within 24 hours. Guardrails are p95 latency under a fixed budget and cost per resolved conversation.

You pre-register: baseline resolution is 0.62, the smallest lift worth shipping is 3 points, so sample_size_per_arm(0.62, 0.03) tells you roughly how many conversations you need per arm. You start a 2 percent canary for a day, confirm no crashes and cost in range, then ramp to 10 percent. Every conversation logs its variant, latency, tokens, and later its thumbs-up and reopen status keyed by request_id.

After you hit the planned sample, you run two_proportion_ztest on resolutions. Say the candidate shows 0.66 versus 0.62 with a p-value under 0.05, p95 latency within budget, and lower cost because its answers were shorter. That is a clean win: promote to 50 percent, watch for a day, then 100 percent, keeping control warm for one cycle. If instead the candidate tied on quality but cut cost, you still ship it, for the cost. If it regressed p95 latency past the guardrail, you hold or roll back even if quality nudged up, because the guardrail was the pre-agreed line.

FAQ

How much traffic do I need for LLM model A/B testing? It depends on your baseline rate and the smallest effect worth detecting, not on a fixed number. Use the sample_size_per_arm calculation. Rare events and small effects need a lot of traffic; big, obvious regressions show up in a canary almost immediately. If you cannot reach the required sample in a reasonable window, either raise the minimum effect you care about or lean harder on offline evals and LLM-as-judge on a sampled slice.

Can I A/B test more than two models at once? Yes, and the assignment code extends naturally: split the bucket range into more than two segments. The catch is statistics. Comparing many arms inflates the chance of a false winner, so correct for multiple comparisons (for example a Bonferroni adjustment on your threshold) or run a single primary comparison and treat the rest as exploratory.

Should I bucket on user id or on request? Bucket on a stable key, usually user id or session id, so a given user always sees the same model. Per-request random assignment gives incoherent experiences and breaks any per-user metric. The only time request-level assignment is acceptable is a stateless, single-shot API where there is no notion of a returning user.

How is this different from offline evaluation? Offline evaluation runs both models on a fixed, curated dataset before shipping and catches obvious regressions cheaply. LLM model A/B testing measures on live traffic with real users and real outcomes. They are complementary: gate with offline evals first so you never canary something clearly broken, then use production A/B testing to confirm the improvement holds on the real distribution.

Is LLM-as-judge reliable enough to decide an experiment? As the sole decider, be careful. Judges favor longer and stylistically similar answers and can drift. Use LLM-as-judge for volume on sampled pairs, keep the judge model fixed and blinded to the variant, randomize pairwise order, and calibrate periodically against human labels. When you have direct user signal like thumbs-up or resolution, weight that above the judge.

What do I do when the result is "no difference"? Treat it as information, not failure. If a cheaper or faster candidate is statistically tied with control on quality, ship it and bank the cost or latency win. If the candidate is more expensive and only ties, keep control. A tie only feels bad if you expected the experiment to always crown a new champion; its real job is to stop you shipping regressions and to justify changes with data.

How long should an experiment run? Long enough to reach your pre-computed sample size and to span at least one full weekly cycle, since traffic and user behavior differ across weekdays and weekends. Do not stop early just because the candidate looks ahead; peeking inflates false positives. Do stop early if a guardrail clearly breaks, that is a safety valve, not a quality decision.