LangSmithEvaluation

LangSmith Evaluations Explained: Datasets, Evaluators and Experiments

Pramod Dutta · Jun 12, 2026 · 14 min read

You shipped a prompt change last week because it "felt better" in a few manual tests. Then a user reported the bot started hallucinating citations it never used to. You have no idea if this is your change, a model update, or bad luck, because you never measured anything before you shipped. This is the default state of most LLM teams, and it is exactly the problem LangSmith evaluations exist to solve. If you cannot compare version A of your app against version B on the same inputs with the same scoring logic, you are not engineering — you are guessing with extra steps.

LangSmith gives you three primitives to fix this: datasets, evaluators, and experiments. None of them are exotic ideas. They are the same testing discipline you already use for regular software — fixtures, assertions, test runs — adapted for the fact that LLM outputs are non-deterministic and often ungradeable by simple equality checks. Once you understand how the three pieces fit together, evaluation stops being a research project and becomes a normal part of your development loop.

Why Vibe Checks Stop Working

Early in a project, "vibe checking" is fine. You run a prompt against three examples, read the outputs, and decide if it looks reasonable. This breaks down the moment you have more than a handful of edge cases to worry about, because:

Manual review does not scale past a dozen examples, and real production traffic has hundreds of distinct failure modes.
You cannot remember what the output looked like before your last change, so every comparison is against your memory, not against data.
Different team members eyeball outputs differently, so "looks good to me" is not a repeatable signal.
Regressions in one part of the input space get masked by improvements in another, and vibes cannot tell you the net effect.

The fix is the same fix software engineering already applied to this problem decades ago: write down your test cases, define what "correct" means, run the code against the cases, and track the score over time. LangSmith evaluations are just that pattern, wired directly into the same platform where you already have your traces.

Datasets: Your Test Fixtures

A dataset in LangSmith is a collection of examples, where each example is typically an input and, optionally, a reference or expected output. Think of it as the fixture file in a traditional test suite — except instead of asserting add(2, 2) == 4, you are asserting something fuzzier, like "given this customer question, the answer should mention the refund policy and should not promise a timeline we cannot guarantee."

Datasets can come from two very different places, and you will end up using both:

Hand-authored examples — cases you write deliberately to cover known edge cases: empty inputs, adversarial prompts, ambiguous phrasing, multi-turn context, non-English queries, or anything a domain expert flags as tricky.
Production traces — real inputs your app has already handled, pulled directly from LangSmith's trace logs. These are gold because they represent the actual distribution of what users throw at your system, including the weird stuff you never thought to write a test case for.

A healthy dataset usually mixes both. Hand-written cases guarantee you cover known risks; trace-derived cases guarantee you cover unknown ones. As your app matures, the dataset should grow — every production incident is a candidate new example, because if it broke once, it can break again silently in a future release.

Building Your First Dataset From Real Traces

The fastest way to get real signal is not to sit down and invent hypothetical test cases. It is to mine your own traces. Here is the practical workflow:

Filter your trace logs to a time window and a run type you care about — for example, the last two weeks of your RAG chatbot's top-level chain runs.
Sample deliberately, not randomly. Pull a mix: some traces flagged with low user feedback scores, some flagged with errors or retries, some picked completely at random to represent the "normal" case, and some that took unusually long or used unusually many tokens.
Add runs to a dataset. Each selected trace becomes an example: the input is whatever your chain received, and the output — if you already know it was good, or if you correct it — becomes the expected reference.
Clean up the reference outputs. Raw production outputs are not automatically "correct." A human should review each one and either accept it as the target answer, edit it into what the answer should have been, or leave it without a reference and rely on a rubric-based evaluator instead of exact matching.
Tag the dataset by capability. Instead of one giant undifferentiated blob, split examples into named datasets or use metadata fields — "refund-policy-questions," "multi-turn-follow-ups," "adversarial-jailbreak-attempts" — so a regression later can be traced to a specific capability, not just "something broke somewhere."

The mistake teams make here is waiting until the dataset is "complete" before using it. Start with twenty solid examples pulled from real traces. That is already more signal than a hundred vibe checks, and you can keep growing the dataset every time something goes wrong in production.

Evaluators: Turning Outputs Into Scores

A dataset tells you what to run. An evaluator tells you whether the result was any good. Structurally, an evaluator is just a function: it takes a run (the actual input/output/trace produced by your app) and, optionally, the reference example from the dataset, and it returns a score.

There are three broad families of evaluators you will reach for, roughly in order of how much nuance they can capture:

Exact-match and heuristic evaluators. Cheap, deterministic, and fast. Did the output contain a required substring? Does the returned JSON parse and match a schema? Is the output length under some limit? These are great for structural correctness — "did the tool call use valid arguments" — but useless for judging whether an explanation was actually good.
LLM-graded evaluators. You use a second model call to grade the first model's output against a rubric — checking for correctness, relevance, tone, or faithfulness to retrieved context. This is where most nuanced evaluation ends up living, because natural-language quality resists hardcoded rules.
Custom code evaluators. Arbitrary logic you write yourself: calling an external API to verify a fact, running a regex over structured fields, computing a similarity score against a reference embedding, or combining several signals into one composite score. This is your escape hatch whenever neither exact-match nor an LLM judge captures the thing you actually care about.

The important design decision is not which family to pick in the abstract — it is matching the evaluator to what "correct" means for that specific dataset. A support bot's refund policy answer needs a different notion of correctness than a code-generation task's "does this compile and pass tests" check.

Writing a Custom Evaluator Function

Custom evaluators are just Python (or TypeScript) functions with a predictable shape: accept the run and the reference example, return a score — often a boolean, a 0–1 float, or a small structured dict with a key metric plus a comment explaining the reasoning. Here is the conceptual shape of one that checks whether a support-bot answer stayed grounded in the retrieved context and hit the required policy points:

def groundedness_and_policy_evaluator(run, example):
    """
    Custom evaluator: scores an answer on two axes —
    1) whether it stays grounded in the retrieved context (no invented facts)
    2) whether it mentions the required policy disclaimer

    run: the actual execution trace, including inputs, outputs, and
         any intermediate steps like retrieved documents
    example: the dataset example, including the expected/reference output
    """
    output_text = run.outputs.get("answer", "")
    retrieved_docs = run.outputs.get("retrieved_context", [])
    expected = example.outputs.get("expected_answer", "")

    # Heuristic check: did the answer include the mandatory disclaimer?
    has_disclaimer = "refunds are processed within" in output_text.lower()

    # Groundedness check: naive containment test against retrieved context.
    # In practice you'd swap this for an embedding similarity check or
    # an LLM-graded faithfulness check for anything non-trivial.
    context_blob = " ".join(doc.lower() for doc in retrieved_docs)
    grounded = all(
        phrase.lower() in context_blob
        for phrase in extract_claims(output_text)
    )

    passed = has_disclaimer and grounded

    return {
        "key": "groundedness_and_policy",
        "score": 1.0 if passed else 0.0,
        "comment": (
            f"disclaimer_present={has_disclaimer}, "
            f"grounded_in_context={grounded}"
        ),
    }

Note the pattern: the function signature is stable (run, example in — a score dict out), which means you can register a dozen of these, run them all against the same experiment, and get a dashboard of independent signals instead of one opaque pass/fail number. When one metric regresses, you know exactly which axis broke.

A practical tip: keep evaluators narrow. A single evaluator that tries to score "overall quality" on a 1–10 scale is much harder to trust and debug than three evaluators — groundedness, policy compliance, tone — each scoring a narrow, well-defined thing. Narrow evaluators are also easier to unit test themselves, which matters because a buggy evaluator silently invalidates every experiment that uses it.

Experiments: Running Your App Against the Dataset

An experiment is the act of actually executing your application — the real chain, agent, or prompt you are testing — against every example in a dataset, then running each configured evaluator against every resulting run. The output is a single comparable score (or set of scores) attached to that specific version of your app.

This is the piece that turns datasets and evaluators from static artifacts into an actual feedback loop. The workflow looks like this:

Pick a dataset that represents the capability you are testing.
Point an experiment at your current app configuration — a specific prompt version, model, temperature setting, or retrieval pipeline.
Attach the evaluators relevant to that dataset.
Run the experiment. LangSmith executes your app against every example and stores each run alongside its evaluator scores.
Repeat step 2 with your candidate change — new prompt, new model, new chunking strategy — against the same dataset and evaluators.
Compare the two experiment results side by side.

Because both experiments used the identical dataset and identical evaluators, the only variable that changed is your app. That is the entire point: it isolates your change as the cause of any score delta, the same way changing one variable in a controlled experiment isolates causality anywhere else in science.

Comparing Two Experiment Runs

This is where the payoff shows up. Say you changed your system prompt to be more concise, hoping it reduces latency without hurting answer quality. You run the same dataset through both the old and new prompt as two experiments, then compare:

Aggregate score movement. If your groundedness score dropped from 0.94 to 0.81 while your policy-compliance score stayed flat, you know precisely what got worse — the model got looser with facts, not that it forgot the disclaimer.
Per-example diffs, not just averages. An aggregate number can hide the real story. Maybe the average barely moved, but ten specific examples flipped from pass to fail — often clustered around one capability, like multi-turn context handling. Averages will not tell you that; row-by-row comparison will.
Regressions versus improvements, netted out. A change can genuinely help some inputs and hurt others. The question is never "did the average go up" — it is "did it go up enough to be worth the regressions it introduced," which is a product decision, not just a number.
Latency and cost alongside quality. A prompt that scores marginally higher but doubles token usage and adds 2 seconds of latency per call might not be a net win. Track these next to your quality metrics in the same experiment comparison, not in a separate spreadsheet.

The discipline this creates is simple but powerful: no prompt change, model swap, or pipeline refactor ships without a before/after experiment comparison on a dataset that represents real usage. That single habit eliminates most of the "it felt better in my three manual tests, but users are complaining" incidents.

Annotation Queues: Bringing Humans Back In

Automated evaluators are necessary but not sufficient. Some judgments genuinely need a human — tone appropriateness for a sensitive support conversation, whether a legal disclaimer is actually adequate, whether a generated summary lost something that mattered even though it "looks fine" structurally. Annotation queues are how LangSmith routes runs to humans for exactly this kind of review.

The pattern is straightforward: instead of a human randomly opening the trace explorer and reading logs, runs get pushed into a queue — often the ones an automated evaluator flagged as borderline, or a random sample for ongoing quality audits, or every run tied to a specific customer segment. A reviewer works through the queue, attaches labels or scores, and those human judgments become first-class feedback, comparable side-by-side with your automated evaluator scores.

This matters for two reasons beyond just "catching what automation misses." First, human labels are how you validate that your LLM-graded evaluators are actually trustworthy — if your automated judge disagrees with human reviewers on a meaningful fraction of cases, the judge's rubric needs work, not your app. Second, annotation queues are how you grow your dataset over time: a human-reviewed, human-corrected run is exactly the kind of high-quality example you want to fold back into your dataset for the next round of experiments.

Choosing the Right Evaluator for the Job

A common failure mode is reaching for an LLM-graded evaluator by default because it feels more "intelligent," when a five-line heuristic would have been faster, cheaper, and more reliable. Use this rough decision order:

If correctness can be checked mechanically — JSON schema validity, presence of a required field, a number falling in an expected range, a regex match — write a heuristic evaluator. It is instant, free, and has zero variance.
If correctness depends on semantic judgment but you have a clear rubric — factual accuracy against a reference, tone matching a style guide, whether an answer addresses the actual question asked — use an LLM-graded evaluator with an explicit, narrow rubric rather than a vague "rate this 1–10" prompt.
If correctness needs external verification — a fact that must be checked against a live database, a calculation that must actually be executed, a citation that must actually exist — write a custom code evaluator that calls out to the source of truth instead of trusting either heuristics or another model's judgment.
If correctness is genuinely subjective and safety-relevant — anything touching tone in sensitive conversations, edge cases in policy interpretation, borderline content decisions — route it through an annotation queue instead of trying to fully automate it away.

Most real evaluation suites end up as a blend of all four, attached to the same dataset, because different failure modes need different detection mechanisms. Relying on just one type is how teams end up either missing obvious mechanical bugs (because they only had an LLM judge) or missing genuine quality regressions (because they only had heuristics).

Making Evaluation Part of the Normal Workflow

None of this matters if evaluation is a one-off exercise you run before a big launch and then forget about. The teams that get real value from LangSmith evaluations treat it like CI for their prompts:

Every meaningful prompt, model, or retrieval change gets run as an experiment against the relevant dataset before it ships, not after.
Datasets are living artifacts — every production incident or user complaint becomes a candidate new example, so your test suite gets harder to pass over time instead of staying frozen at day one.
Evaluator definitions themselves get reviewed and versioned, because a bad evaluator that always scores 0.9 regardless of input is worse than no evaluator — it creates false confidence.
Experiment comparisons get attached to the actual decision of whether to ship, the same way a red test suite blocks a merge in traditional software.

The underlying shift is treating your prompts, chains, and agents as software artifacts with a real test suite, not as one-off text you tweak and eyeball. Datasets are your fixtures, evaluators are your assertions, experiments are your test runs, and annotation queues are your manual QA pass for the things automation cannot fully judge yet.

If you take one idea away from this: the hardest and most valuable evaluators you will build are rarely the exact-match kind — they are the ones that use LLM-as-a-Judge to grade genuinely subjective qualities like faithfulness, helpfulness, or tone, and getting those rubrics right is where most of the real engineering effort in LLM evaluation actually lives.