LangSmith Experiment Comparison: Reading the Diff That Matters

The Moment You Actually Need a Comparison

You changed one line in a prompt. Maybe you bumped the temperature, swapped a model, or rewrote the system message to be a little firmer about JSON. You run your evaluation, the aggregate score ticks up by two points, and you feel that small rush of progress. Then a teammate asks the question that ruins your afternoon: "Two points on what? Which examples got better, and did anything get worse?"

This is the moment a LangSmith experiment comparison earns its keep. A single experiment score is a headline. A comparison is the article. It tells you not just that something moved, but where it moved, by how much, and whether the movement is the kind you can trust or the kind that will embarrass you in production next week.

Most teams treat evaluation as a scoreboard. You run version A, you run version B, you check which number is bigger, and you ship the winner. That works right up until it does not, and it stops working precisely when the stakes are highest: when two versions are close, when one metric improves while another quietly degrades, or when your average is being propped up by a handful of easy examples while the hard ones rot. Reading the diff that matters is the skill that separates people who guess from people who know. This article is about building that skill.

What a LangSmith Experiment Actually Is

Before we compare experiments, we should be precise about what one is, because the vocabulary trips people up constantly.

A dataset in LangSmith is a fixed collection of examples. Each example has inputs and, usually, a reference output, sometimes called the expected output or the ground truth. The dataset is your stable measuring stick. You do not change it between runs, because if the ruler changes, the measurement means nothing.

An experiment is what you get when you run a target function over every example in that dataset and score the results. The target function is whatever you are testing: a chain, an agent, a single LLM call, a retrieval pipeline. The scoring comes from evaluators, which are functions that look at the output (and often the reference) and return a score. One experiment produces one row per example, each row carrying the input, the generated output, the reference, and every evaluator score attached to that run.

Here is a minimal experiment in code so the shape is concrete.

from langsmith import Client
from langsmith.evaluation import evaluate

client = Client()

def target(inputs: dict) -> dict:
    # your system under test
    question = inputs["question"]
    answer = call_your_pipeline(question)
    return {"answer": answer}

def correctness(outputs: dict, reference_outputs: dict) -> dict:
    predicted = outputs["answer"].strip().lower()
    expected = reference_outputs["answer"].strip().lower()
    return {"key": "correctness", "score": float(predicted == expected)}

results = evaluate(
    target,
    data="qa-golden-set",
    evaluators=[correctness],
    experiment_prefix="baseline",
)

When this finishes, LangSmith stores an experiment. Run it again with a tweaked target and a different experiment_prefix, and you have a second experiment over the identical dataset. Two experiments, same ruler. Now, and only now, a comparison is meaningful.

The Comparison View and How to Read It

Open two or more experiments that share a dataset and LangSmith gives you a comparison view. The layout is deceptively simple: one row per example, one column per experiment, and the evaluator scores laid side by side. But there is a lot packed into that grid, and reading it well is a learned habit.

Start at the top, with the aggregate. Each experiment shows its summary scores: the mean of every evaluator across all examples, plus latency and token or cost figures if you tracked them. This is your headline, and it answers exactly one question: on average, did the number go up or down? Note the word "average." Averages hide as much as they reveal, which is why you do not stop here.

Drop into the per-example rows. This is where the comparison stops being a scoreboard and becomes a diff. Each row shows you the same input scored under both versions, so you can see whether example number forty-seven got better, got worse, or stayed flat. LangSmith highlights the deltas, so regressions do not hide in a wall of green. Your eye should go straight to the red first. A change that raises your average while introducing three brand-new failures is not obviously a good change, and you would never learn that from the headline alone.

Use the sort and filter controls aggressively. Sort by the delta on your primary metric, descending, and you see your biggest wins. Sort ascending and you see your biggest regressions. Filter to rows where version B scored lower than version A and you have an instant list of everything the change broke. This filtered set is the single most valuable artifact in the whole view, because fixing regressions is almost always more urgent than celebrating improvements.

Then open individual runs. Click into a row and you get the full trace: the exact prompt sent, the raw model response, intermediate chain steps, retrieved documents, tool calls, token counts, and the evaluator's reasoning. A number tells you something changed. The trace tells you why. You cannot fix what you cannot see, and the trace is where the seeing happens.

Regression Versus Noise: The Distinction That Saves Your Sanity

Here is the trap that catches almost everyone. You run the same version twice and the scores are not identical. Did something break? No. LLMs are stochastic, evaluators that use an LLM judge are stochastic, and small datasets amplify every wobble. Some of the movement you see between experiments is real signal, and some is pure noise. Confusing the two costs you either false confidence or wasted days chasing ghosts.

The discipline that fixes this is establishing a noise floor. Before you compare A against B, compare A against A. Run your baseline twice with no changes and measure how much the score moves on its own. If two identical runs differ by three points, then a three-point difference between A and B means nothing. You need to clear the noise floor before you claim a real effect.

# Run the identical baseline twice to measure run-to-run variance
run_1 = evaluate(target, data="qa-golden-set",
                 evaluators=[correctness], experiment_prefix="baseline-a")
run_2 = evaluate(target, data="qa-golden-set",
                 evaluators=[correctness], experiment_prefix="baseline-b")
# Compare baseline-a vs baseline-b in the UI.
# The delta you see here is your noise floor. Any real
# change must beat it before you trust it.

A few practical rules follow from this. First, dataset size matters enormously. Ten examples cannot distinguish a two-point improvement from randomness. The smaller the set, the larger the swing you need before you believe it. Second, pin what you can. Set temperature to zero when the task allows it, fix your seeds where the API supports them, and version-lock the model, because "the latest model" silently changing under you is a comparison-killer. Third, look at direction and consistency, not just magnitude. A change that improves forty examples and worsens two is a different animal from one that improves twenty-one and worsens twenty, even if the averages land in the same place. The first is a real, coherent effect. The second is noise wearing a costume.

Reading Deltas Across Multiple Metrics at Once

Real systems are almost never scored on a single dimension. You care about correctness, but also about format validity, latency, cost, refusal rate, and maybe a groundedness or hallucination check. The comparison view shows all of these together, and the interesting story usually lives in the tension between them.

Suppose your new prompt lifts correctness from 0.81 to 0.87. Clear win, ship it. But scroll right and latency climbed from 900 milliseconds to 2.4 seconds because you added a chain-of-thought step, and cost doubled because the responses are three times longer. Is that still a win? The comparison cannot answer that for you, but it puts every number in front of you so you can make the trade deliberately instead of discovering the latency regression from an angry user.

This is why you should resist collapsing everything into one composite score too early. A single blended number is convenient for a dashboard, but it destroys exactly the information a comparison exists to surface. When correctness and cost are fused into one figure, you lose the ability to see that you bought six points of accuracy with a doubling of your bill. Keep your metrics separate in the comparison, and only combine them, if you must, once you have seen the individual movements with your own eyes.

Watch especially for anti-correlated metrics, the pairs that trade against each other. Verbosity often buys correctness at the cost of latency and money. Stricter formatting instructions can raise JSON validity while lowering answer quality, because the model spends its attention obeying the schema instead of thinking. A firmer safety prompt can cut hallucinations and simultaneously spike refusals on perfectly benign inputs. The comparison view is where these tensions become visible. If you only ever look at your headline metric, you are flying blind on all the others.

Regression Hunting: A Repeatable Workflow

Reading a comparison well is not improvisation. It is a routine you run the same way every time, so nothing important slips past you. Here is the sequence I use.

1. Confirm both experiments share the exact same dataset. If the ruler differs, stop. The comparison is invalid before it begins.
2. Read the aggregate deltas for every metric, not just the one you care about. Write down which went up and which went down.
3. Filter to regressions on your primary metric. Sort ascending by delta so the worst offenders sit at the top.
4. Open the three worst regressions and read their full traces. Look for a pattern: same input type, same failure mode, same missing retrieval.
5. Categorize each regression. Is it a genuine capability loss, an evaluator quirk, or a noise-floor flicker that would vanish on a rerun?
6. Repeat steps three through five for your secondary metrics, especially latency and cost, which regress quietly and get noticed late.
7. Only after the regressions are understood, enjoy the improvements. Verify they are real by spot-checking a couple of the biggest positive deltas.

The order is the whole point. Amateurs open a comparison and scroll straight to the wins, because wins feel good. Professionals go hunting for what broke first, because a shipped regression costs far more than a delayed improvement. Discipline here is the difference between a rollback at 2 a.m. and a quiet, confident deploy.

When the Evaluator Is the Thing That Changed

A subtle failure mode deserves its own section, because it fools smart people. Sometimes the diff you are staring at is not caused by your system changing at all. It is caused by your evaluator changing, or by your evaluator being unreliable in the first place.

If you edited your evaluator between two experiments, adjusted the LLM-judge prompt, changed the grading rubric, tightened a string match, then any delta you see is hopelessly confounded. You cannot tell whether the output got better or the grader got stricter. The two effects are tangled and you cannot pull them apart. Change one variable at a time. If you must update the evaluator, rerun the old experiments under the new evaluator so both sides are graded by the identical judge, then compare.

Even when the evaluator is unchanged, an LLM-based judge introduces its own noise and its own biases. Judges have been shown to favor longer answers, to prefer their own family of models, and to grade the same output differently on repeat calls. So when a comparison shows movement, ask a second question before you trust it: do I believe the score, or do I believe the judge? Open a few traces and read the judge's actual reasoning. If example twelve got marked wrong, read why it got marked wrong. Half the time you will find the output was fine and the judge was confused, or the reference answer itself was stale. The comparison view surfaces the discrepancy; your own eyes have to adjudicate it.

# An LLM-judge evaluator returns reasoning, not just a score.
# Always read the reasoning when a delta looks suspicious.
def llm_judge(inputs: dict, outputs: dict, reference_outputs: dict) -> dict:
    verdict = judge_model.evaluate(
        question=inputs["question"],
        answer=outputs["answer"],
        reference=reference_outputs["answer"],
    )
    return {
        "key": "judged_correctness",
        "score": verdict.score,          # numeric, drives the aggregate
        "comment": verdict.explanation,  # human-readable, drives your trust
    }

That comment field is not decoration. In a comparison, when two scores disagree, the reasoning behind each is what lets you decide which experiment actually deserves to win.

Turning a Comparison Into a Decision

A comparison you read but never act on is a diary entry. The point is to reach a decision you can defend, and there are only a handful of honest outcomes.

You ship version B when it clears the noise floor on your primary metric, introduces no regressions you are unwilling to accept, and does not blow your latency or cost budgets. Note the compound condition. All of it has to hold, not just the first clause. A correctness bump that ships three new failures and doubles your spend is not a clean win, and pretending otherwise is how bad changes reach production.

You hold and investigate when the aggregate improved but the per-example view shows a worrying cluster of regressions. The average lied, or at least oversimplified, and you owe it to yourself to understand the failures before you commit. This is the outcome the comparison view was built for, and it is the one a scoreboard mentality never reaches.

You reject and revert when the regressions outweigh the gains, when the improvement sits inside the noise floor, or when the trade against a secondary metric is unacceptable. Rejecting a change after reading a comparison is not a failure. It is the entire system working. You caught a bad change before your users did, which is precisely what all this measurement was for.

The habit worth building is writing the decision down next to the comparison. One or two sentences: what moved, why you believe it, what you are choosing, and what you will watch after deploy. Six weeks later, when someone asks why the prompt looks the way it does, that note is the answer. Evaluation without a recorded decision is just expensive curiosity.

Making Comparisons a Habit, Not an Event

The teams that get real value from LangSmith do not reach for a comparison only when something feels wrong. They wire it into the normal flow of work, so every meaningful change is measured against the last known-good version before it merges.

Practically, that means running your evaluation on every pull request that touches a prompt, a model choice, a retrieval setting, or a chain structure, and comparing the resulting experiment against your current production baseline. The diff becomes a required reading, the same way you would read a code diff before approving it. Nobody merges a code change sight unseen. A prompt change deserves the same scrutiny, and the comparison view is where that scrutiny happens.

It also means keeping your datasets alive. Every time a real regression slips through, capture the offending input, attach the correct reference, and add it to your dataset. Your golden set should grow scar tissue over time, each example a memory of something that once broke. A comparison is only as sharp as the dataset behind it, and a stale dataset produces confident, meaningless diffs. Feed it, curate it, and prune the examples that no longer reflect what you care about.

None of this is exotic. It is the same instinct that made version control and code review non-negotiable, applied to the probabilistic layer of your stack where the old tools go blind. A comparison is a diff for behavior instead of source, and reading it well is the same craft as reading a pull request well: look past the summary, hunt for what broke, understand the trade-offs, and decide on purpose.

If you want to go deeper on the mechanics, from wiring up datasets and evaluators to building the regression-hunting muscle memory this article describes, that is exactly what the LangSmith Tutorial course on teachyou.ai is built to give you. It walks through real experiments, real regressions, and the exact reading habits that turn a wall of numbers into a decision you can stand behind. The score tells you something changed. Learning to read the diff that matters tells you whether to trust it.