LangSmith for A/B Testing Prompt Variants in Production

You rewrote a prompt last week and it "felt better" in the playground, so you shipped it. Two days later, a customer complained that the assistant started ignoring their formatting instructions, and now nobody on the team can say with confidence whether the new prompt is actually an improvement or a regression. This is the default state of prompt engineering at most companies: changes are judged by vibes, deployed on gut feeling, and rolled back in a panic. LangSmith A/B testing gives you a way out. By versioning your prompt variants, splitting live traffic between them, attaching feedback to every trace, and comparing the results with real metrics, you can treat prompt changes with the same rigor that product teams apply to a new checkout flow. In this guide, we will build a complete A/B testing workflow for prompt variants using LangSmith, from variant creation through traffic routing, feedback capture, and final analysis, with production-ready Python code at every step.

Why A/B Testing Prompts Is Harder Than A/B Testing UI

Classic A/B testing is a solved problem in web analytics. You show half your users a green button and half a blue one, count the clicks, run a significance test, and ship the winner. Prompt variants break several of the assumptions that make that workflow simple.

First, the output of an LLM call is not a binary event like a click. A response can be partially correct, correct but verbose, correct but slow, or wrong in a way that only surfaces three turns later in the conversation. Your metric layer has to capture quality signals, not just conversion events, which means you need explicit feedback mechanisms wired into your application.

Second, LLM outputs are non-deterministic. The same prompt with the same input can produce different responses across calls, so a single spot check tells you almost nothing. You need volume, and you need every single call traced and attributed to the variant that produced it, or your analysis dissolves into noise.

Third, prompt changes have entangled side effects. A rewrite that improves answer accuracy might double token usage and blow up your latency budget. A tighter system prompt might reduce hallucinations on one intent while degrading tone on another. This means a proper prompt experiment tracks multiple metrics per variant simultaneously: correctness, latency, cost, and user satisfaction.

Finally, there is the sample attribution problem. In a multi-step chain or agent, the "prompt variant" is buried inside one node of a larger pipeline. Without a tracing system that records which variant served which request, you literally cannot reconstruct the experiment after the fact. This is exactly the gap LangSmith fills: it is a tracing and evaluation platform where every run carries metadata, every run can receive feedback scores, and runs can be sliced, filtered, and compared by any dimension you attach to them.

The Core LangSmith Primitives You Will Use

Before writing any experiment code, it helps to understand the four LangSmith building blocks that make prompt A/B testing work.

Traces and runs. Every call through your application produces a trace: a tree of runs capturing inputs, outputs, latency, token counts, and errors. You get tracing either automatically (if you use LangChain with the LANGSMITH_TRACING environment variable set) or explicitly with the @traceable decorator from the LangSmith SDK. Runs are the atomic unit of your experiment; each one is a data point.

Metadata and tags. Runs accept arbitrary key-value metadata and string tags. This is the mechanism for variant attribution. When request 4812 is served by variant_b, you write {"prompt_variant": "variant_b"} into that run's metadata. Later, you filter and group by that key in the LangSmith UI or through the SDK. Metadata is the backbone of the whole workflow, and forgetting to attach it is the single most common way prompt experiments fail.

Feedback. LangSmith lets you attach named feedback scores to any run: a thumbs up from a user, a 1-to-5 rating from a human reviewer, or a score emitted by an automated evaluator. Feedback is how raw traces become measurable outcomes. A run with user_score: 1 and resolved: true is a success; group those by variant and you have your experiment readout.

Datasets and experiments. Beyond live traffic, LangSmith supports offline experiments: you build a dataset of representative inputs (often harvested from production traces), run each prompt variant over the dataset with the evaluate function, and compare the resulting experiments side by side. Offline comparison is how you sanity-check a variant before it ever touches a real user, and it pairs naturally with the online A/B test that follows.

There is one more piece worth naming: the LangSmith Prompt Hub, which stores versioned prompts with commit hashes. Treating prompts like code, with versions you can pull by tag or hash, is what makes the word "variant" mean something precise instead of "whatever string happens to be in the config file today."

Creating and Versioning Your Prompt Variants

An A/B test is only as trustworthy as its variant definitions. If engineers can hot-edit the prompt string mid-experiment, your results are garbage. The fix is to store each variant as a versioned prompt in LangSmith and pull it by an immutable reference.

Suppose you run a support assistant and want to test whether a more structured system prompt improves resolution quality. You push both variants to the Prompt Hub:

from langsmith import Client
from langchain_core.prompts import ChatPromptTemplate

client = Client()

variant_a = ChatPromptTemplate.from_messages([
    ("system",
     "You are a helpful support assistant for an online course platform. "
     "Answer the customer's question clearly and concisely."),
    ("human", "{question}"),
])

variant_b = ChatPromptTemplate.from_messages([
    ("system",
     "You are a support assistant for an online course platform.\n"
     "Follow these rules strictly:\n"
     "1. Answer only from the provided context. If unsure, say so.\n"
     "2. Keep answers under 120 words.\n"
     "3. End with one concrete next step the customer can take.\n"
     "Context: {context}"),
    ("human", "{question}"),
])

client.push_prompt("support-assistant-a", object=variant_a)
client.push_prompt("support-assistant-b", object=variant_b)

Each push returns a URL containing a commit hash. From this point on, your application pulls prompts by name (optionally pinned to a specific commit with the name:hash syntax), which gives you three important properties. The variant is immutable for the duration of the experiment. The exact text of what you tested is permanently recorded next to its results. And when the experiment concludes, promoting the winner is a one-line change rather than an archaeology project through git history and Slack threads.

A practical note on variant design: change one meaningful thing at a time. Variant B above changes grounding rules, length constraints, and response structure together, which is fine for a first coarse experiment, but if B wins you will not know which ingredient mattered. Mature teams run coarse experiments to find a winning direction, then follow up with narrower experiments isolating individual changes.

Routing Production Traffic Between Variants

With variants versioned, you need a router that assigns each request to a variant and records the assignment. Two rules matter here. Assignment should be deterministic per user (so a returning user does not flip between prompt personalities across a session), and the assignment must be written into the trace metadata at call time.

Hash-based bucketing handles both cleanly:

import hashlib
from langsmith import Client, traceable
from langchain_openai import ChatOpenAI

client = Client()
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

VARIANTS = {
    "variant_a": "support-assistant-a",
    "variant_b": "support-assistant-b",
}
ROLLOUT_B = 0.5  # fraction of traffic sent to variant B

def assign_variant(user_id: str) -> str:
    digest = hashlib.sha256(user_id.encode()).hexdigest()
    bucket = int(digest[:8], 16) / 0xFFFFFFFF
    return "variant_b" if bucket < ROLLOUT_B else "variant_a"

@traceable(name="support_answer", run_type="chain")
def answer_question(user_id: str, question: str, context: str) -> dict:
    variant = assign_variant(user_id)
    prompt = client.pull_prompt(VARIANTS[variant])
    messages = prompt.invoke({"question": question, "context": context})
    response = llm.invoke(messages)
    return {
        "answer": response.content,
        "variant": variant,
    }

result = answer_question(
    user_id="user_1234",
    question="How do I get a refund for a course I bought yesterday?",
    context="Refunds are available within 14 days of purchase...",
    langsmith_extra={
        "metadata": {
            "prompt_variant": assign_variant("user_1234"),
            "experiment": "support-prompt-2026-07",
        }
    },
)

The langsmith_extra argument stamps the run with the variant name and an experiment identifier. That experiment key matters more than it looks: you will run many experiments over the life of this endpoint, and scoping each one with its own identifier keeps analyses from bleeding into each other.

Notice also that the SHA-256 bucketing makes ramping trivial. Start with ROLLOUT_B = 0.05 to expose the new variant to five percent of users as a safety canary, watch the traces for a day, then raise it to 0.5 for the real experiment. Because bucketing is a pure function of the user ID, users already assigned to B stay in B as the rollout expands.

If you deploy through LangGraph or a gateway layer, the same pattern applies: compute the assignment at the edge, pass it down as metadata, and make sure every LLM run in the request's trace tree inherits it.

Capturing Feedback: Turning Traces Into Metrics

Traffic splitting without outcome measurement is just chaos with extra steps. The experiment becomes real when every run can be scored. LangSmith supports three complementary feedback channels, and strong experiments usually use at least two of them.

Explicit user feedback is the gold standard where you can get it. Thumbs up and down buttons, star ratings, or a "did this solve your problem?" prompt all map directly to feedback scores on the run that produced the response:

from langsmith import Client

client = Client()

def record_user_feedback(run_id: str, helpful: bool, comment: str = ""):
    client.create_feedback(
        run_id=run_id,
        key="user_helpful",
        score=1.0 if helpful else 0.0,
        comment=comment,
    )

# In your API layer, return the run_id to the frontend with the
# response, then post it back when the user clicks thumbs up/down.
record_user_feedback(
    run_id="f3a1c2e0-8d5b-4c77-9e21-0b6f4d9a1c55",
    helpful=True,
    comment="Answered on the first try",
)

The wiring detail that trips people up is run ID plumbing. Your backend must return the trace's run ID alongside the LLM response so the frontend can attach feedback to the right run later, sometimes minutes after the response was rendered. Generating the run ID yourself with uuid4() and passing it in langsmith_extra={"run_id": ...} makes this deterministic.

Implicit behavioral signals cover the majority of users who never click a feedback button. Did the user rephrase the same question immediately (a strong negative)? Did they end the session after the answer (weakly positive)? Did the conversation escalate to a human agent (strong negative for a support bot)? Each of these can be computed by your application and logged as feedback with its own key, such as escalated or rephrased_within_2_turns.

Automated LLM-as-judge scoring fills the coverage gap. You define a judge prompt that scores responses on dimensions like groundedness or instruction adherence, run it asynchronously over a sample of production traces, and log the scores as feedback. LangSmith's online evaluators can do this server-side on a sampled percentage of incoming runs, which means you get continuous quality scoring on both variants without adding latency to the user-facing path. Judges have known failure modes, including a bias toward longer and more confident-sounding answers, so calibrate the judge against a set of human-labeled examples before trusting it as a primary metric.

Validating Variants Offline Before the Ramp

Sending an untested prompt to live users is an avoidable risk. Before any production ramp, run both variants through an offline experiment against a dataset that reflects real traffic. The cleanest source for that dataset is production itself: filter recent traces to the endpoint in question, pick a diverse sample including known hard cases, and add them to a LangSmith dataset with reference answers where you have them.

The evaluate function then runs each variant over the dataset and applies your evaluators:

from langsmith import Client, evaluate

client = Client()

def make_target(prompt_name: str):
    prompt = client.pull_prompt(prompt_name)
    def target(inputs: dict) -> dict:
        messages = prompt.invoke({
            "question": inputs["question"],
            "context": inputs.get("context", ""),
        })
        response = llm.invoke(messages)
        return {"answer": response.content}
    return target

def concise_enough(outputs: dict) -> dict:
    words = len(outputs["answer"].split())
    return {"key": "under_120_words", "score": float(words <= 120)}

def has_next_step(outputs: dict) -> dict:
    text = outputs["answer"].lower()
    signals = ["you can", "next step", "try", "visit", "click"]
    return {"key": "actionable", "score": float(any(s in text for s in signals))}

for variant, prompt_name in VARIANTS.items():
    evaluate(
        make_target(prompt_name),
        data="support-questions-golden",
        evaluators=[concise_enough, has_next_step],
        experiment_prefix=f"support-2026-07-{variant}",
        metadata={"prompt_variant": variant},
    )

In the LangSmith UI, the two experiments appear side by side on the dataset page, with per-evaluator averages and per-example diffs. The comparison view is genuinely useful here: you can select both experiments, see exactly which examples one variant handled and the other fumbled, and read the paired outputs next to each other. Regressions concentrated in a specific slice, say refund questions versus certificate questions, show up immediately in a way aggregate scores hide.

For a head-to-head judgment rather than independent scores, evaluate_comparative runs a pairwise evaluator that sees both variants' outputs for the same input and picks a winner. Pairwise judging tends to be more reliable than absolute scoring for subjective qualities like tone and helpfulness, because the judge only has to rank, not calibrate.

Offline results are a gate, not a verdict. A variant that loses badly offline should not ship at all. A variant that wins offline still needs the online test, because golden datasets never fully capture the strangeness of real users.

Analyzing the Live Experiment

Once both variants have been serving traffic, analysis happens along two tracks: the LangSmith UI for exploration, and the SDK for programmatic readouts.

In the UI, the tracing project's filter bar accepts metadata queries, so filtering runs by your experiment key and grouping the monitoring charts by metadata.prompt_variant splits every dashboard chart, including trace counts, error rates, latency percentiles, token usage, and feedback score averages, into one line per variant. This turns the abstract question "is B better?" into a set of concrete, separable questions. Is B's user_helpful average higher? Is its P95 latency acceptable? Did its cost per request move?

For a scripted summary you can drop into a weekly report, pull the runs and aggregate:

from collections import defaultdict
from langsmith import Client

client = Client()
stats = defaultdict(lambda: {"runs": 0, "helpful": [], "latency": []})

runs = client.list_runs(
    project_name="support-bot-prod",
    filter='and(eq(metadata_key, "experiment"), eq(metadata_value, "support-prompt-2026-07"))',
    is_root=True,
)

for run in runs:
    variant = (run.extra or {}).get("metadata", {}).get("prompt_variant", "unknown")
    stats[variant]["runs"] += 1
    if run.end_time and run.start_time:
        stats[variant]["latency"].append((run.end_time - run.start_time).total_seconds())
    for fb in client.list_feedback(run_ids=[run.id]):
        if fb.key == "user_helpful" and fb.score is not None:
            stats[variant]["helpful"].append(fb.score)

for variant, s in stats.items():
    n = len(s["helpful"])
    rate = sum(s["helpful"]) / n if n else float("nan")
    lat = sum(s["latency"]) / len(s["latency"]) if s["latency"] else float("nan")
    print(f"{variant}: {s['runs']} runs, helpful={rate:.3f} (n={n}), avg latency={lat:.2f}s")

Two statistical cautions apply. Feedback is sparse and self-selected: users who click thumbs down are not a random sample of users, so compare like with like across variants and lean on implicit signals for volume. And resist the urge to call the experiment the moment one variant edges ahead. Decide before the experiment how many scored runs you need per arm and what difference you consider meaningful, then let it run. Peeking at a live dashboard and stopping when the gap looks good is the classic way to ship noise.

When a variant wins, close the loop properly: promote its prompt in the Prompt Hub, move the rollout fraction to send full traffic to the winner, archive the experiment identifier, and write down the result where the next engineer will find it. The losing variant's traces are not waste. They are labeled failure data, exactly what you need to seed the dataset for the next experiment.

Common Pitfalls That Quietly Ruin Prompt Experiments

Having watched teams adopt this workflow, a handful of failure modes come up again and again.

• Missing attribution. The variant is chosen in one service but the metadata is attached in another, and a refactor silently breaks the link. Make variant metadata a required field in your tracing wrapper and alert when runs arrive without it.
• Contaminated variants. Someone edits the "B" prompt halfway through the experiment to fix a typo. Now your data describes two different prompts averaged together. Pin prompt versions by commit and treat any edit as a new experiment.
• Session splitting. Random per-request assignment means a single conversation mixes both variants, corrupting multi-turn quality signals. Always bucket by user or session ID.
• Single-metric tunnel vision. B improves helpfulness by a hair while doubling token spend and P95 latency. Track cost and latency per variant from day one, not as an afterthought when the invoice arrives.
• Judge worship. An LLM judge preferring B does not mean users prefer B. Use judges for coverage and regression detection; use human and behavioral signals for the final call.
• Underpowered experiments. Ten thumbs-up clicks per arm decide nothing. If explicit feedback volume is low, extend the experiment window, add implicit signals, or increase judge sampling rather than declaring a winner from anecdotes.
• Testing everything at once. New prompt, new model, and new retrieval settings shipped together tells you only that "the bundle" changed things. Hold everything except the prompt constant, or you are not running a prompt experiment.

None of these are LangSmith limitations; they are experiment design mistakes that tracing makes visible. The platform's job is to make the right behavior cheap, and with versioned prompts, metadata filters, and feedback APIs, it mostly does.

From One Experiment to a Continuous Evaluation Loop

The real payoff of LangSmith A/B testing is not any single winning prompt. It is the flywheel that forms once the pieces are in place. Production traces reveal failure cases. Failure cases become dataset examples. Datasets power offline experiments that filter out weak variants cheaply. Surviving variants graduate to online A/B tests with real users. The winner ships, its traces generate the next round of failure cases, and the loop turns again.

Teams that run this loop stop arguing about prompts in meetings, because the question "which prompt is better?" has an empirical answer with a dashboard behind it. Prompt changes go from the scariest deploys in the codebase to some of the safest, because every change is canaried, attributed, measured, and reversible. And the accumulated datasets become an asset in their own right: when you eventually evaluate a new model or migrate providers, you already have the golden sets and evaluators to do it in an afternoon instead of a quarter.

Start small. Version the one prompt that scares you most, split ten percent of traffic, wire a thumbs-up button to create_feedback, and look at the grouped dashboard after a week. The first experiment teaches you more about your system than the previous six months of eyeballing outputs.

If you want to go deeper into building this workflow end to end, from your first traced run through datasets, custom evaluators, LLM-as-judge calibration, online evaluation, and full production experiment design, the LangSmith Tutorial course on teachyou.ai walks through every step with hands-on projects, so you can bring rigorous prompt experimentation to your own stack this week.