A/B Testing LLM Changes in Production

LLM A/B testing is how you find out whether a prompt tweak, a model swap, or a retrieval change actually made your product better instead of just different. You run the old version and the new version side by side on real traffic, measure outcomes that matter, and keep the winner. Offline evals tell you if a change is plausible; LLM A/B testing tells you if it works when real users, real inputs, and real money are on the line.

This guide covers the parts that are easy to get wrong: how to split traffic deterministically, which metrics survive contact with production, how to log enough to debug later, and how to read the numbers without lying to yourself. Everything here is runnable and provider-agnostic. If you can call a chat completion endpoint and write to a database, you can ship this.

Why offline evals are not enough

Offline evaluation runs your candidate against a fixed dataset with a scoring function, usually an LLM judge or a set of assertions. It is fast, cheap, and catches obvious regressions. You should absolutely have it. But it lies to you in three predictable ways.

First, your eval set drifts from reality. The prompts users actually send this month are not the ones you froze into a golden set six months ago. A change that scores +8% offline can be flat or negative live because the input distribution moved.

Second, offline evals measure proxy quality, not product outcomes. An LLM judge might prefer the more verbose answer while your users abandon it because it takes too long to read. The judge and the user disagree, and the user is the one paying.

Third, offline evals cannot see downstream behavior. Did the support answer actually deflect the ticket? Did the code suggestion get accepted? Did the summary get copied? Those signals only exist in production. LLM A/B testing is the only way to connect a model change to the behavior you actually care about.

The rule of thumb: gate every change with offline evals so you never ship an obvious regression, then confirm with an online A/B test before you roll it out to everyone.

Anatomy of an LLM A/B test

An online experiment for an LLM feature has five moving parts.

• A unit of assignment: the thing you bucket. Usually a user id, sometimes an account id or a session id. Pick one and never mix them within a test.
• A variant assignment function: deterministic, so the same unit always gets the same variant for the life of the experiment.
• A treatment: the actual change. A new prompt, a different model, a reranker, a temperature change, a tool added or removed.
• Instrumentation: the logs and events that let you compute metrics and debug individual cases.
• A decision rule: the metric, the direction you expect it to move, and the threshold at which you ship or kill.

Get the first two right and the rest is bookkeeping. Get them wrong and every number downstream is contaminated.

Deterministic traffic splitting

Do not assign variants with random(). If a user gets the new prompt on one request and the old one on the next, their experience is incoherent and your metrics mix treatments within a single unit. You want sticky assignment: hash the unit id together with the experiment name and bucket on the hash.

import hashlib

def assign_variant(unit_id: str, experiment: str, weights: dict[str, float]) -> str:
    # Hash unit + experiment so the same user is stable within an experiment,
    # but independent across experiments (no cross-test correlation).
    key = f"{experiment}:{unit_id}".encode()
    digest = hashlib.sha256(key).hexdigest()
    # Take 8 hex chars -> integer in [0, 2^32), normalize to [0, 1).
    bucket = int(digest[:8], 16) / 0xFFFFFFFF

    cumulative = 0.0
    for variant, weight in weights.items():
        cumulative += weight
        if bucket < cumulative:
            return variant
    return list(weights)[-1]  # floating point guard

# Usage
weights = {"control": 0.5, "treatment": 0.5}
variant = assign_variant(user_id, "summary_prompt_v3", weights)

Three properties make this correct. It is deterministic, so a user is pinned to one variant for the whole experiment. It is uniform, because SHA-256 spreads evenly across the bucket space. And it is independent across experiments, because the experiment name is in the hash, so running five experiments at once does not correlate their assignments. That last property is what lets you run overlapping tests without them polluting each other.

Include the experiment name in the hash even if you think you will only ever run one test. You will run more, and retrofitting isolation after the fact means rehashing live users into new buckets mid-flight.

Wiring the variant into the call

Keep the variant definition in one place: a config object, a feature flag service, or a small table. The call site should be dumb. It asks for the config for a variant and executes it. This keeps the experiment logic out of your business logic.

VARIANTS = {
    "control": {
        "model": "your-current-model",
        "system_prompt": "You are a concise support assistant.",
        "temperature": 0.2,
    },
    "treatment": {
        "model": "your-current-model",
        "system_prompt": "You are a support assistant. Answer in under 80 words, "
                         "lead with the fix, then a one-line why.",
        "temperature": 0.2,
    },
}

def generate_answer(user_id: str, question: str) -> dict:
    variant = assign_variant(user_id, "summary_prompt_v3", {"control": 0.5, "treatment": 0.5})
    cfg = VARIANTS[variant]

    response = client.chat.completions.create(
        model=cfg["model"],
        temperature=cfg["temperature"],
        messages=[
            {"role": "system", "content": cfg["system_prompt"]},
            {"role": "user", "content": question},
        ],
    )
    answer = response.choices[0].message.content

    log_generation(
        experiment="summary_prompt_v3",
        variant=variant,
        unit_id=user_id,
        question=question,
        answer=answer,
        model=cfg["model"],
        prompt_tokens=response.usage.prompt_tokens,
        completion_tokens=response.usage.completion_tokens,
    )
    return {"answer": answer, "variant": variant}

Notice the only difference between control and treatment lives in the config. That discipline matters: when you read results, you want to be able to say exactly one thing changed. If your treatment also happens to swap the model and add a tool, a win tells you nothing about which change earned it.

Change one thing per experiment. If you must change several, that is a different, coarser question ("is the whole new pipeline better?"), and you should frame it that way and not pretend you learned anything about the individual parts.

Choosing metrics that survive contact with users

This is where most LLM A/B tests go wrong. Teams measure what is easy (token count, latency, an LLM judge score) and ignore what matters (did the user get what they came for). Structure your metrics in three tiers.

Primary metric. One number that defines success, tied to user value. Pick it before you look at any data. Examples: ticket deflection rate, suggestion acceptance rate, task completion rate, thumbs-up rate, conversion. If you cannot name a primary metric, you are not ready to run the test.

Guardrail metrics. Things that must not get worse even if the primary improves. Latency (p50 and p95), cost per request, error and refusal rate, safety flags. A treatment that lifts acceptance 3% but doubles p95 latency is usually a loss. Guardrails stop you from shipping a local win that is a global loss.

Diagnostic metrics. Things that explain the primary metric but are not decision criteria on their own. Output length, retrieval hit rate, tool-call frequency, LLM judge scores. Useful for understanding why a result happened, dangerous as the thing you optimize.

A concrete set for a support assistant:

• Primary: fraction of conversations that end without the user opening a human ticket.
• Guardrails: p95 response latency, cost per conversation, refusal rate.
• Diagnostics: average answer length, retrieval recall, LLM judge helpfulness.

The trap is optimizing a diagnostic because it is convenient to measure. An LLM judge score is a diagnostic, not a primary metric. Judges are useful and they are also biased toward length, formatting, and confident tone. Use them to explain, not to decide.

Instrumentation: log for the argument you will have later

When results come in, someone will ask "why did treatment win?" or "why is this one answer terrible?" You need the logs to answer both. Log at generation time, not after, because the inputs are ephemeral.

For every generation, capture: a stable request id, the experiment name, the variant, the unit id, a hash or trimmed copy of the input, the output, the model, token counts, latency, and a timestamp. Then, separately, log the outcome events keyed by the same request id: the thumbs-up, the ticket-opened, the suggestion-accepted, whatever your primary metric is built from.

create table llm_generations (
    request_id   text primary key,
    experiment   text not null,
    variant      text not null,
    unit_id      text not null,
    input_hash   text,
    output       text,
    model        text,
    prompt_tokens int,
    completion_tokens int,
    latency_ms   int,
    created_at   timestamptz default now()
);

create table llm_outcomes (
    request_id   text references llm_generations(request_id),
    event_type   text not null,   -- 'ticket_opened', 'thumbs_up', 'accepted'
    value        double precision,
    created_at   timestamptz default now()
);

Two things save you later. Keep generations and outcomes in separate tables joined by request id, because outcomes arrive minutes or hours after generation and you do not want to block the response path waiting for them. And log the variant at generation time rather than recomputing it during analysis, so that if you ever change the assignment function you do not retroactively rewrite history.

If you use an LLM observability tool (LangSmith, Langfuse, Helicone, Braintrust, and similar all do this), let it own the generation trace and attach the experiment and variant as metadata. Wire your outcome events back to the same trace id. The pattern is identical; you are just not hand-rolling the storage.

Reading the results without fooling yourself

You have two variants and a primary metric. Now you compare. The instinct is to eyeball two conversion rates, see 24% versus 26%, and declare a winner. Resist it. That gap can be pure noise.

For a rate metric (converted or not, deflected or not), the question is whether the difference between two proportions is larger than the noise you would expect from sample size alone. A two-proportion z-test answers exactly that.

from statsmodels.stats.proportion import proportions_ztest

# control: 240 successes of 1000; treatment: 268 of 1000
successes = [268, 240]
totals = [1000, 1000]

stat, p_value = proportions_ztest(successes, totals)
print(f"treatment rate: {268/1000:.3f}")
print(f"control rate:   {240/1000:.3f}")
print(f"p-value:        {p_value:.4f}")

A small p-value (the common threshold is 0.05) means the difference is unlikely to be noise. But statistical significance is not the same as "ship it." Two more checks.

Effect size. A difference can be statistically real and practically meaningless. If treatment lifts deflection by 0.3 points and that is significant only because you have two million samples, ask whether 0.3 points is worth the added cost or latency. Always report the absolute and relative lift next to the p-value.

Practical thresholds on guardrails. Even a real, meaningful primary lift loses if a guardrail breaks. Compute latency and cost deltas the same way and hold them to a pre-agreed budget.

Then there is the sin that undoes everything above: peeking.

Peeking, sample size, and the sequential-testing fix

If you check the p-value every hour and stop the moment it dips below 0.05, you will "find" significant results that are pure noise. The math assumes you decide the sample size in advance and look once. Repeated looks inflate your false-positive rate badly, often to 20-30% instead of the 5% you think you have.

Two honest ways out.

Fixed-horizon: decide the sample size before you start, using a power calculation, run until you hit it, then look once. To size the test you need your baseline rate, the minimum lift worth detecting, and your desired power (usually 80%).

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

baseline = 0.24            # current deflection rate
target = 0.27             # smallest lift worth shipping
effect = proportion_effectsize(target, baseline)

n = NormalIndPower().solve_power(
    effect_size=effect, alpha=0.05, power=0.80, ratio=1.0
)
print(f"needed sample size per variant: {int(n) + 1}")

Run the number before the experiment. If it says you need 40,000 users per arm and you get 500 a day, you now know this test takes months, and you can decide whether the change is worth that wait before you spend it.

Sequential testing: if you genuinely need to look continuously (to stop a harmful treatment early), use a method built for it, such as an always-valid confidence sequence or a group-sequential design with alpha spending. These keep the false-positive rate controlled while allowing repeated looks. Many experimentation platforms implement one of these; if you are hand-rolling, use a library rather than inventing the correction yourself.

The one thing you must not do is run a fixed-horizon test and then peek. Pick a discipline and hold to it.

A safe rollout sequence

Wiring and statistics in hand, here is the order of operations that keeps a bad change from reaching everyone.

1. Gate the candidate through offline evals. If it regresses the golden set, stop here.
2. Ship it behind the experiment flag at a small exposure, for example 5% treatment, mostly to confirm nothing is on fire (error rates, latency, obvious garbage output).
3. If guardrails hold, raise to a 50/50 split and let it run to the pre-computed sample size.
4. Read the primary metric once, with the guardrails. Ship, kill, or iterate.
5. If you ship, keep a small holdback (a few percent left on control) for a week or two to catch slow-burn effects the short test missed.

The small-exposure canary in step 2 is not the experiment; it is a smoke test. The real read happens at 50/50 with enough samples. Do not skip the smoke test on the theory that offline evals caught everything, because offline evals never see production input shapes.

Common failure modes

• Assigning on the wrong unit. Bucketing per-request instead of per-user makes the same user flip variants, which both wrecks their experience and biases metrics. Assign on the durable id.
• Changing several things at once. New model plus new prompt plus new retriever. A win tells you the bundle helped, not which piece. Fine if the bundle is the question; wrong if you wanted to learn about the prompt.
• Optimizing an LLM judge. Judge scores are diagnostics. Ship on user outcomes, use the judge to understand them.
• Peeking and stopping early. Covered above, and worth repeating because it is the most common and most invisible mistake.
• Ignoring novelty and primacy effects. Existing users may react to any change simply because it is new, and new-user cohorts may behave differently. A holdback and a long enough window smooth these out.
• Skipping guardrails. A primary-metric win that quietly doubled cost or latency is a loss you shipped because you did not measure it.

FAQ

How much traffic do I need for an LLM A/B test? It depends entirely on your baseline rate and the smallest lift you care about. Run the power calculation shown above before you start. Rarer events and smaller lifts need more samples. If the required sample size is larger than your traffic can supply in a reasonable window, either aim to detect a bigger effect or accept that you are making a judgment call, not a measured one.

Can I A/B test with an LLM judge instead of user metrics? Use the judge offline to gate changes and online as a diagnostic, but decide on real user outcomes. Judges are biased toward length, formatting, and confident tone, and they cannot see whether the user's actual problem got solved. A judge-only "win" often does not show up in behavior.

How do I run several LLM experiments at once? Include the experiment name in your assignment hash so buckets are independent across experiments. That lets overlapping tests coexist without correlating. The exception is when two experiments touch the same surface in conflicting ways; those you should sequence, not overlap.

Should I split by user or by session? Default to the most durable id you have, usually the user or account. Session splitting lets one user experience both variants, which contaminates any metric that depends on consistent experience and inflates your effective correlation. Only split by session when you truly have no stable user id.

How long should the experiment run? At least until you reach the pre-computed sample size, and long enough to cover your natural usage cycle, typically a full week or two, so weekday and weekend behavior are both represented. Stopping the moment significance appears is the peeking trap.

What if offline evals and the A/B test disagree? Trust the A/B test for the ship decision, because it measures real outcomes, then investigate the gap. A disagreement usually means your eval set drifted from live traffic or your offline metric is a poor proxy for the thing users care about. Fold what you learn back into the eval set so it tracks reality next time.

Do I need a full experimentation platform? No. The pattern in this guide (deterministic hashing, config-driven variants, two logging tables, a z-test, and a power calculation) runs on a database and a few dozen lines of code. Reach for a platform when you are running many concurrent experiments, need sequential testing done correctly, or want non-engineers to launch tests. Start simple and let the pain tell you when to upgrade.