LangSmith Custom Evaluators: Beyond the Built-In Metrics

The first time you run an evaluation in LangSmith, the built-in evaluators feel like magic. You point a dataset at your chain, pick qa or cot_qa from the menu, and within a minute you have a table of pass and fail marks. Then you ship a feature, look at the scores, and slowly realize the truth that every serious LLM team eventually hits: the built-in metrics do not know what your product means by "correct." They know whether an answer roughly matches a reference. They do not know that your support bot must never invent a refund policy, that your SQL agent must produce a query that actually runs, or that your summariser must stay under 280 characters because it feeds a tweet. Those rules live in your head and in your codebase, not in a generic correctness prompt.

This is the gap custom evaluators fill. A custom evaluator is just a function you write that receives a run and its reference example, and returns a score. That is the whole idea. Once you internalise it, LangSmith stops being a dashboard you glance at and becomes a testing harness you trust the way you trust a unit test suite. In this article we will go well past the built-in menu. We will write heuristic evaluators, LLM-as-judge evaluators, structural validators, and evaluators that check tool calls, and wire them into datasets so they run automatically on every experiment. Every code block here is a real evaluator you can paste into a project and adapt.

Why the built-in evaluators run out of road

LangSmith ships a solid set of off-the-shelf evaluators. There is qa for reference-backed correctness, cot_qa for a chain-of-thought variant, context_qa for grounding, and a family of criteria evaluators for things like conciseness, relevance, and harmfulness. For a first pass on a question-answering system, these are genuinely useful and you should use them. The problem is not that they are bad, it is that they are general, and generality is exactly what you do not want when you are trying to catch the specific way your application breaks.

Consider a few failure modes that no generic evaluator will catch for you. Your agent returns valid JSON most of the time, and the built-in correctness score stays high because the content is right, but the malformed responses crash a downstream parser in production. A generic evaluator reads the semantic content and shrugs. Or your legal-document assistant paraphrases a clause in a way that is technically close to the reference but drops the word "not," inverting the meaning, and string similarity barely moves. Or your chatbot is supposed to escalate to a human whenever a user mentions a chargeback, and it usually does, but the built-in relevance score has no concept of "should have escalated."

Each of these is a rule that is obvious to you and invisible to a general metric. The built-in evaluators optimise for being reasonable across every possible use case, which means they are excellent at none of yours. Custom evaluators invert that trade: you give up generality and gain a metric that encodes your actual product contract. When you own the evaluator, you also own the definition of quality. You can version it, review it in a pull request, and change it deliberately when the product changes. A score that comes out of forty lines of Python you wrote is something you can reason about at three in the morning when the on-call pager goes off.

The anatomy of a LangSmith evaluator

Before we write anything ambitious, let us nail down the contract. In the current LangSmith SDK, an evaluator is a callable. The most flexible signature receives two arguments: run, the traced execution you are grading, and example, the dataset row containing the inputs and the reference output. From run.outputs you read what your system actually produced. From example.outputs you read what it was supposed to produce. Your job is to compare them and return a result.

The return value can take several shapes and LangSmith is forgiving about all of them. You can return a plain dictionary with a key and a score, a boolean that becomes a pass or fail, or a number that becomes a continuous score. For anything real, prefer the dictionary form, because it lets you attach a key (the metric name as it appears in the UI) and a comment (a human-readable explanation that shows up next to the score and is worth its weight in gold when you are debugging a regression).

Here is the smallest useful custom evaluator. It checks that the output is not empty, which sounds trivial until an upstream timeout starts returning blank strings and your correctness metric silently ignores them because an empty answer is not "wrong," it is just absent.

from langsmith.schemas import Run, Example


def not_empty(run: Run, example: Example) -> dict:
    """Fail any run whose output is blank or whitespace-only."""
    prediction = (run.outputs or {}).get("output", "")
    passed = bool(prediction) and bool(prediction.strip())
    return {
        "key": "not_empty",
        "score": 1 if passed else 0,
        "comment": "Output present" if passed else "Output was empty or whitespace",
    }

Notice the defensive (run.outputs or {}). Runs can fail, and a failed run may have None where you expect a dictionary. An evaluator that throws is worse than a lenient one, because a crashing evaluator produces no score at all and you lose the signal entirely. Treat your evaluators like production code that runs against hostile input, because that is exactly what they are. The rest of this article builds on this same skeleton: read the prediction, read the reference, decide, return a dictionary.

Heuristic evaluators: cheap, fast, and deterministic

The most underrated category of evaluator is the plain heuristic. No model call, no cost, no latency, no non-determinism. Just Python that encodes a rule you can state precisely. If you can describe the check in a sentence containing the words "must" or "never," you can probably write it as a heuristic, and you almost always should, because heuristics are the fastest and most reliable evaluators you will ever run.

Take a concrete product rule. Suppose you build summaries posted as tweets, so every output must be at most 280 characters and must not end mid-word because a truncated summary looks broken. Here is an evaluator that checks both and returns a graded score rather than a hard pass or fail, so you can see how close to the limit you are running.

from langsmith.schemas import Run, Example


def tweet_length(run: Run, example: Example) -> dict:
    text = (run.outputs or {}).get("output", "") or ""
    length = len(text)
    within_limit = length <= 280
    # A crude "ends cleanly" check: last char is punctuation or the text is short.
    ends_clean = length == 0 or text.rstrip()[-1] in ".!?…"

    if not within_limit:
        score, comment = 0.0, f"Too long: {length} chars (limit 280)"
    elif not ends_clean:
        score, comment = 0.5, f"Fits ({length} chars) but ends mid-thought"
    else:
        score, comment = 1.0, f"Good: {length} chars, clean ending"

    return {"key": "tweet_length", "score": score, "comment": comment}

The pattern generalises to a huge number of real checks. Does the output contain a forbidden phrase, like a competitor's name or a promise your legal team banned? Does it include a required disclaimer? Does a generated slug match ^[a-z0-9-]+$? Does a price field parse as a positive number? Every one of these is a few lines of string work.

A particularly valuable heuristic family checks structural validity for outputs meant to be machine-readable. If your chain is meant to emit JSON, you do not need a language model to tell you whether it emitted JSON. You need json.loads inside a try block. Here is an evaluator that validates the output parses as JSON and contains the keys your downstream code depends on, which catches the exact class of bug that generic correctness metrics wave through.

import json
from langsmith.schemas import Run, Example

REQUIRED_KEYS = {"intent", "confidence", "entities"}


def valid_json_schema(run: Run, example: Example) -> dict:
    raw = (run.outputs or {}).get("output", "") or ""
    try:
        parsed = json.loads(raw)
    except (json.JSONDecodeError, TypeError):
        return {"key": "valid_json", "score": 0, "comment": "Output is not valid JSON"}

    if not isinstance(parsed, dict):
        return {"key": "valid_json", "score": 0, "comment": "JSON is not an object"}

    missing = REQUIRED_KEYS - parsed.keys()
    if missing:
        return {
            "key": "valid_json",
            "score": 0.5,
            "comment": f"Valid JSON but missing keys: {sorted(missing)}",
        }

    return {"key": "valid_json", "score": 1, "comment": "Valid JSON with all required keys"}

Run this against a dataset and you will learn, quickly and cheaply, exactly how often your model breaks its output contract. That number is often higher than teams expect, and it is precisely the one that determines whether your integration is production-ready. No amount of semantic correctness matters if a parser downstream throws before it ever sees the content.

LLM-as-judge evaluators: grading what heuristics cannot

Heuristics are wonderful, but plenty of quality questions are irreducibly fuzzy. Is this answer polite? Does this summary preserve the key point of the source? Did the model refuse a genuinely harmful request while still helping with the benign part? You cannot regex your way to those judgments. This is where you reach for an LLM-as-judge evaluator, where a second model reads the output and scores it against a rubric you write.

The idea is simple and the execution has a few traps. You send the judge a carefully worded prompt containing the input, the output, optionally the reference, and a clear instruction to return a structured verdict. The single most important design decision is to force the judge to return structured output, not prose, because you need to parse a score out of it reliably. Ask for free-form text and you will spend your evenings writing brittle parsers for "I would rate this a solid 8, maybe 8.5." Ask for JSON with a fixed schema and you get a number you can trust.

Here is a self-contained LLM-as-judge evaluator that scores factual faithfulness of a summary against its source document. It uses the OpenAI SDK directly so the mechanics are visible, but the same shape works with any provider you prefer.

import json
from openai import OpenAI
from langsmith.schemas import Run, Example

client = OpenAI()

JUDGE_PROMPT = """You are grading whether a SUMMARY is faithful to its SOURCE.
Faithful means every claim in the summary is supported by the source, with no
invented facts, names, numbers, or conclusions.

SOURCE:
{source}

SUMMARY:
{summary}

Return ONLY a JSON object of the form:
{{"faithful": true or false, "reason": "<one short sentence>"}}"""


def faithfulness_judge(run: Run, example: Example) -> dict:
    source = (example.inputs or {}).get("document", "")
    summary = (run.outputs or {}).get("output", "") or ""

    prompt = JUDGE_PROMPT.format(source=source, summary=summary)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        response_format={"type": "json_object"},
        messages=[{"role": "user", "content": prompt}],
    )

    verdict = json.loads(response.choices[0].message.content)
    passed = bool(verdict.get("faithful"))
    return {
        "key": "faithfulness",
        "score": 1 if passed else 0,
        "comment": verdict.get("reason", ""),
    }

A few things make this evaluator trustworthy rather than theatrical. The judge runs at temperature=0, so the same output gets the same grade run after run, which is non-negotiable if you want to compare experiments over time. The prompt defines "faithful" precisely instead of leaving it to the model's taste, because a vague rubric produces a vague and drifting score. The response_format is pinned to a JSON object so parsing never guesses. And the reason field flows straight into the comment, so a failing row shows not just that it failed but why the judge thought so, which turns a mysterious red mark into an actionable bug report.

The honest caveat is that a judge is a model, and models are imperfect graders. The judge can be wrong, it can be biased toward longer answers, and it costs money and latency on every run. Treat its output as a strong signal, not gospel. The discipline that pays off is to spot-check the judge against a handful of human labels early, and only then trust it to scale your judgment across thousands of examples you will never read by hand.

Evaluating agents: scoring tool calls and trajectories

Once you move from single-turn chains to agents that call tools, a whole new category of evaluation opens up, and the built-in metrics have essentially nothing to offer here. The interesting question is no longer only "was the final answer right" but "did the agent take a sensible path to get there." Did it call the right tool, with well-formed arguments? Did it avoid a destructive tool it had no business touching? These are trajectory questions, and because your run is fully traced, the information you need is sitting right there in the run tree.

The practical move is to reach into the intermediate steps of the run and inspect the tool calls the agent made. Frameworks surface this slightly differently, but the shape is consistent: somewhere in run.outputs or the run's child runs you can find the sequence of actions. Here is an evaluator that checks whether an agent called a specific required tool, which is exactly the check you want for a rule like "any question about account balance must go through the get_balance tool and never be answered from the model's imagination."

from langsmith.schemas import Run, Example


def _collect_tool_names(run: Run) -> list[str]:
    """Walk intermediate steps and pull out every tool name the agent invoked."""
    steps = (run.outputs or {}).get("intermediate_steps", []) or []
    names = []
    for step in steps:
        # Steps are typically (action, observation) pairs.
        action = step[0] if isinstance(step, (list, tuple)) else step
        tool = getattr(action, "tool", None)
        if tool is None and isinstance(action, dict):
            tool = action.get("tool")
        if tool:
            names.append(tool)
    return names


def required_tool_used(run: Run, example: Example) -> dict:
    expected = (example.outputs or {}).get("expected_tool")
    if not expected:
        return {"key": "required_tool", "score": None, "comment": "No expected tool set"}

    used = _collect_tool_names(run)
    passed = expected in used
    return {
        "key": "required_tool",
        "score": 1 if passed else 0,
        "comment": f"Expected '{expected}'. Agent used: {used or 'none'}",
    }

Two design choices here matter beyond this example. When the example does not specify an expected tool, the evaluator returns a score of None rather than passing or failing, which tells LangSmith to skip scoring that row instead of polluting your aggregate with an irrelevant judgment. And the comment lists exactly which tools the agent used, so a failure is instantly diagnosable: you do not open the trace and hunt, the evaluator already told you the agent reached for search_web when it should have reached for get_balance.

You can extend this in every direction that matters for agents. Count the number of steps and penalise runs that loop pointlessly. Check that a destructive tool like delete_record was never called on a read-only query. Validate that the arguments passed to a tool match a schema. Each is the same move: read the trajectory out of the run, apply a rule, return a score. The trace is your source of truth, and a custom evaluator is how you turn that trace into a number you can track across every experiment.

Wiring evaluators into datasets and experiments

An evaluator that lives in a notebook and runs by hand is a fun exercise. An evaluator that runs automatically against a dataset every time you change your prompt is a safety net. The bridge between the two is the evaluate function, which takes the thing you want to test, the dataset to test it against, and the list of evaluators to apply. You define your system as a function that takes the example inputs and returns outputs, point evaluate at a named dataset, and pass every evaluator you have written as a list. LangSmith runs your system over each row, applies each evaluator, and writes the whole thing up as an experiment you can open in the UI and compare against previous runs.

from langsmith import evaluate


def my_summariser(inputs: dict) -> dict:
    document = inputs["document"]
    # ... your real chain or agent call goes here ...
    summary = call_your_chain(document)
    return {"output": summary}


results = evaluate(
    my_summariser,
    data="summariser-golden-set",
    evaluators=[
        not_empty,
        tweet_length,
        valid_json_schema,
        faithfulness_judge,
    ],
    experiment_prefix="summariser-v3",
    max_concurrency=4,
)

Every evaluator you stack onto that list becomes a column in the results table, and every experiment becomes a row in your history. The payoff arrives the moment you change something. You tweak a prompt to fix a tone complaint, rerun the exact same command, and the table shows not only whether tone improved but whether faithfulness quietly dropped, whether JSON validity held, whether anything ran long. That is regression testing for LLM behaviour, and it is the single habit that separates teams who ship LLM features with confidence from teams who ship them and pray.

A few operational notes make this smoother. Keep your evaluators in a dedicated module so they are importable, reviewable, and testable in isolation with ordinary unit tests, because an evaluator with a bug gives you false confidence, which is worse than none at all. Use experiment_prefix to name runs meaningfully. And tune max_concurrency to respect the rate limits of whatever your system and judge model talk to, so a large evaluation does not turn into a wall of 429 errors halfway through.

Combining evaluators into a quality contract

The real power shows up when you stop thinking about evaluators one at a time and start thinking about them as a contract. A single metric tells you one thing. A well-chosen suite tells you whether a change is safe to ship. The art is picking a set that covers the ways your system actually fails, weighted toward the failures that hurt most, without drowning in noise from checks nobody acts on.

A pragmatic contract for a production summariser layers the categories we have built. At the bottom, cheap deterministic heuristics act as gatekeepers: not empty, valid JSON if applicable, within length. These are fast, free, and catch the most common breakages, so they run on every row. In the middle sit structural and trajectory checks for anything agentic: the right tool was called, no forbidden tool fired, the step count stayed sane. At the top sit the expensive LLM-as-judge evaluators for the fuzzy qualities that genuinely need a model's judgment: faithfulness, tone, helpfulness. You run the cheap checks liberally and the expensive ones deliberately, and together they form a picture no single score could paint.

The discipline that keeps this contract honest is to treat a failing evaluator the way you treat a failing test. When faithfulness drops on a new prompt, you do not shrug and ship. You open the failing rows, read the judge's comments, and decide whether the prompt regressed or the evaluator is too strict. Either the code changes or the evaluator changes, but the red does not get ignored. That is what turns LangSmith from a dashboard you admire into a gate you respect. The built-ins tell you roughly how you are doing; a suite of custom evaluators tells you whether you are allowed to deploy.

Where to go from here

Custom evaluators are the point where LLM development starts to feel like engineering instead of alchemy. You stop guessing whether a change helped and start measuring it against rules you defined and can defend. Begin small. Pick the one failure mode that scares you most, whether that is malformed output, an invented fact, or a missing escalation, and write a single evaluator that catches it. Wire it into evaluate against a modest golden dataset of twenty or thirty rows and watch it turn red on the exact bad cases you already know about. That first green-to-red-to-green loop is the moment the whole practice clicks. From there, grow the suite as you learn how your system breaks in the wild: every production incident becomes a new evaluator and a new dataset row so the same bug can never sneak back in unseen, and every fuzzy quality complaint becomes a new judge prompt. Over a few months you accumulate a body of evaluators that encodes, in executable form, everything your team has learned about what "good" means for your product. That asset compounds, and it is far more durable than any single model or prompt you ship on top of it.

If you want to go deeper, from tracing and datasets to advanced evaluator design and running evaluations in CI, our LangSmith Tutorial course on teachyou.ai walks through the entire workflow hands-on, with real projects and the exact patterns shown here scaled up to production. It covers the parts this article only gestured at: managing datasets at scale, pairwise and summary evaluators, catching regressions automatically on every pull request, and building the kind of evaluation culture that lets a team ship LLM features without holding its breath. Start with one evaluator today, and build your way up to a harness you would stake a release on.