LangSmith Datasets: Building and Versioning Test Data

Every LLM team eventually hits the same wall. You tweak a prompt, the demo looks better, you ship it, and three days later a user reports that a case which used to work is now broken. You have no record of what "used to work" even means, because your test data lives in a scratch notebook, a Slack thread, and someone's memory. LangSmith datasets exist to fix exactly this problem. They give you a durable, versioned, queryable home for your test examples, and they plug directly into LangSmith's evaluation runner so every prompt change, model swap, or retrieval tweak can be measured against the same ground truth. In this guide we will build datasets from scratch with the Python SDK, grow them from production traces, organize them with splits and metadata, and, most importantly, version them so that an evaluation you run today can be reproduced exactly six months from now.

Why LangSmith Datasets Are the Backbone of LLM Testing

In traditional software testing, your test cases are code. They live in the repo, they are versioned by git, and a CI run against commit abc123 will behave the same way next year. LLM applications break this model in two ways. First, the "test cases" are usually natural-language inputs paired with reference outputs, and they are curated by a mix of engineers, product managers, and domain experts, many of whom do not work in your repo. Second, the interesting test cases come from production: the weird phrasings, the adversarial inputs, the questions your team never imagined. A static JSON file in the repo goes stale the moment real users show up.

A LangSmith dataset is a collection of examples, where each example has an inputs dictionary, an optional outputs dictionary that serves as the reference or ground truth, and an optional metadata dictionary for anything else you want to track. Datasets live on the LangSmith server, not in your codebase, which means they can be edited in the UI by non-engineers, appended to programmatically from traces, and shared across every experiment your team runs. Crucially, every mutation to a dataset is recorded, so the dataset behaves less like a mutable spreadsheet and more like a git repository for test data. That versioning behavior is what separates a real evaluation practice from vibes-based prompt engineering, and we will spend a large part of this article on it.

The three dataset types LangSmith supports are worth knowing before you create anything. The default kv (key-value) type is the most flexible: inputs and outputs are arbitrary dictionaries, which suits chains, agents, RAG pipelines, and anything with structured fields. The chat type stores lists of chat messages as inputs and a single message as output, which is convenient when you are fine-tuning or evaluating raw chat models. The llm type is the simplest, with a plain string in and a plain string out, matching classic completion-style models. When in doubt, use kv. It can represent everything the other two can, and evaluators do not care which type you picked as long as the keys are consistent.

Creating Your First Dataset with the Python SDK

Everything starts with the langsmith package and an API key. Install the SDK with pip install langsmith, set LANGSMITH_API_KEY in your environment, and instantiate a client. The client is a thin wrapper over the REST API, so anything you can do in the UI you can do in code, which is exactly what you want for repeatable pipelines.

from langsmith import Client

client = Client()

dataset = client.create_dataset(
    dataset_name="support-bot-golden",
    description="Golden Q&A pairs for the customer support assistant",
)

examples = [
    {
        "inputs": {"question": "How do I reset my password?"},
        "outputs": {"answer": "Go to Settings, open Security, and click Reset Password. A reset link is emailed to you."},
        "metadata": {"category": "account", "difficulty": "easy"},
    },
    {
        "inputs": {"question": "Can I get a refund after 30 days?"},
        "outputs": {"answer": "Refunds are available within 30 days of purchase. After that, we can offer account credit on a case-by-case basis."},
        "metadata": {"category": "billing", "difficulty": "medium"},
    },
    {
        "inputs": {"question": "Why was my card charged twice this month?"},
        "outputs": {"answer": "A duplicate charge usually means a plan change mid-cycle. Support can confirm and reverse an accidental duplicate."},
        "metadata": {"category": "billing", "difficulty": "hard"},
    },
]

client.create_examples(dataset_id=dataset.id, examples=examples)

A few practical notes on this snippet. The create_examples call accepts a list of example dictionaries and creates them in a single batch, which is far faster than looping over create_example for large sets. The metadata field is optional but you should treat it as mandatory from day one: category tags, difficulty ratings, source labels, and customer segments are what make a dataset queryable later. Also notice that the reference outputs here are short and factual rather than full model transcripts. Reference outputs do not need to be the exact string you expect the model to produce; they need to contain the information an evaluator, human or LLM, can use to judge correctness. Writing references as terse fact statements rather than polished prose keeps them stable across prompt styles.

One more thing that trips people up: dataset names are unique within a workspace, and create_dataset will fail if the name already exists. For idempotent scripts, check first with client.has_dataset(dataset_name="support-bot-golden") or read the existing one with client.read_dataset(dataset_name=...) before deciding to create.

Building Datasets from Production Traces

Hand-written examples get you started, but the highest-value test cases come from production. If your application is already traced with LangSmith, every real user interaction is sitting in your project as a run, complete with inputs, outputs, latency, errors, and any feedback scores you have attached. Turning the interesting ones into dataset examples is a query away.

from langsmith import Client

client = Client()
dataset = client.read_dataset(dataset_name="support-bot-golden")

runs = client.list_runs(
    project_name="support-bot-prod",
    run_type="chain",
    error=False,
    filter='and(gte(feedback_key("user_score"), 1), gt(latency, 5))',
)

for run in runs:
    client.create_example(
        inputs=run.inputs,
        outputs=run.outputs,
        dataset_id=dataset.id,
        source_run_id=run.id,
        metadata={"source": "production", "trace_project": "support-bot-prod"},
    )

The filter string uses LangSmith's query syntax to find runs that users rated positively but that took more than five seconds, a nice slice for building a performance-regression suite from cases you know should succeed. The source_run_id parameter is the underrated hero here: it links the example back to the originating trace, so when an evaluation fails on this example months later, you can jump straight to the real production interaction that spawned it and see the full context.

Two curation warnings from experience. First, do not bulk-import thousands of raw traces into a dataset. A dataset full of near-duplicate "how do I log in" questions tells you nothing new and slows every experiment. Sample deliberately: negative feedback runs, runs where the model said "I don't know," runs from new feature areas, and a random sample of everyday traffic. Second, production outputs are model outputs, not ground truth. A run that got a thumbs-up is probably correct, but "probably" is not a reference answer. The honest workflow is to import the trace's inputs and outputs, then have a human review and edit the outputs field in the LangSmith UI before that example counts as golden. LangSmith's annotation queues are built for exactly this review step: route candidate runs to a queue, let a reviewer approve or correct them, and add the approved ones to the dataset.

Bulk Import: CSV Files and Existing Test Suites

Most teams do not start from zero. There is usually a spreadsheet of QA pairs somewhere, or a JSON fixture file from an earlier testing effort. LangSmith's upload_csv helper turns a CSV into a dataset in one call, mapping columns to input and output keys.

dataset = client.upload_csv(
    csv_file="./golden_set.csv",
    name="support-bot-from-csv",
    description="Imported from the QA team's master spreadsheet",
    input_keys=["question"],
    output_keys=["answer"],
)

Every column named in input_keys becomes a field in the example's inputs, every column in output_keys becomes a field in outputs, and any remaining columns are ignored, so clean the sheet or split extra columns into metadata beforehand if you need them. For JSON or JSONL sources, skip the CSV round-trip entirely: load the file in Python, reshape it into the examples list format from the earlier snippet, and call create_examples. That path also lets you attach metadata per example, which upload_csv does not.

A structural decision worth making early: fewer, richer datasets beat many tiny ones. It is tempting to create billing-questions, account-questions, and shipping-questions as separate datasets, but you will then run three experiments per change and struggle to compare aggregate quality. The better pattern is one support-bot-golden dataset with a category metadata field and splits, which we will cover shortly, so a single experiment covers everything and you can still slice results by category in the UI. Reserve separate datasets for genuinely different tasks with different input schemas, such as a summarization dataset versus a classification dataset, because mixing incompatible schemas in one dataset makes your target function and evaluators ugly.

How Dataset Versioning Actually Works

Here is the mental model: a LangSmith dataset is append-and-edit storage with an automatic change log. Every time you add, update, or delete examples, LangSmith records the change, and the dataset's state at any past moment can be reconstructed. You never manually "commit" anything. Instead, you read the dataset "as of" a point in time, or "as of" a named tag that you have pinned to a point in time.

This design solves the reproducibility problem elegantly. Suppose you ran an experiment on June 1st, then your teammate added forty new hard examples on June 10th, and today your new prompt scores worse than the June 1st baseline. Is the prompt worse, or is the dataset harder? Without versioning you genuinely cannot tell, and teams burn days on this exact confusion. With versioning, you rerun today's prompt against the dataset as of June 1st and get an apples-to-apples answer.

from datetime import datetime, timezone
from langsmith import Client

client = Client()

# Read the dataset exactly as it existed on June 1st
june_examples = list(
    client.list_examples(
        dataset_name="support-bot-golden",
        as_of=datetime(2026, 6, 1, tzinfo=timezone.utc),
    )
)

# Pin a human-readable tag to the current state
client.update_dataset_tag(
    dataset_name="support-bot-golden",
    as_of=datetime.now(timezone.utc),
    tag="prod",
)

# Later, anyone can read the tagged version by name
prod_examples = list(
    client.list_examples(dataset_name="support-bot-golden", as_of="prod")
)

Tags are the feature to build your workflow around. A timestamp is precise but meaningless to humans; a tag like prod, v1.4, or pre-agent-migration communicates intent. The latest tag always points at the newest version automatically. A tag is a movable pointer, exactly like a git tag or branch head: when you have reviewed a batch of new examples and want them to count in CI, you move prod forward with another update_dataset_tag call. Until you do, CI keeps evaluating against the old pinned state, and in-progress curation cannot break your comparisons.

This gives you a clean two-track workflow. Track one is curation: anyone adds and edits examples continuously against latest, in the UI or via the SDK, without fear. Track two is evaluation: experiments, CI gates, and release comparisons always reference a tag. The moment of moving a tag becomes a deliberate, reviewable act, like merging to main.

Organizing Examples with Splits and Metadata

As a dataset grows past a few dozen examples, you will want to run different subsets in different contexts: a fast smoke-test subset on every pull request, the full set nightly, and a "hard cases" subset when you are tuning a specific weakness. LangSmith models this with splits. A split is a named grouping, and an example can belong to multiple splits at once, which makes them more flexible than folders.

from langsmith import Client

client = Client()

# Fetch billing examples and assign them to two splits
billing = list(
    client.list_examples(
        dataset_name="support-bot-golden",
        metadata={"category": "billing"},
    )
)

client.update_examples(
    example_ids=[ex.id for ex in billing],
    splits=[["billing", "regression"] for _ in billing],
)

# Evaluate only the regression split later
regression_examples = client.list_examples(
    dataset_name="support-bot-golden",
    splits=["regression"],
)

Splits and metadata overlap in capability, so here is a rule of thumb for choosing. Metadata describes facts about the example: where it came from, its category, its difficulty, the customer tier it involves. Splits describe how you intend to use the example: smoke, regression, hard-negatives, train versus test if you are also fine-tuning. Facts go in metadata because they never change; usage groupings go in splits because they evolve as your testing strategy evolves. The practical payoff is that list_examples filters on both, and the evaluate runner accepts whatever iterator of examples you hand it, so any slice you can query, you can evaluate.

Keep an eye on split hygiene. Because examples can be in several splits, it is easy to end up with a smoke split that quietly grew to 400 examples and now takes twenty minutes in CI. Decide on size budgets per split, for example thirty examples for smoke and everything for nightly, and audit them when they drift. The counts are visible on the dataset page in the LangSmith UI, so this is a thirty-second check during any curation session.

Running Evaluations Against a Pinned Version

Datasets exist to be evaluated against, so let us close the loop. The evaluate entry point takes a target function that maps an example's inputs to your application's outputs, a data source, and a list of evaluators. The critical detail for this article is the data argument: passing client.list_examples(..., as_of="prod") instead of a bare dataset name is what pins the experiment to a version.

from langsmith import Client, evaluate

client = Client()

def target(inputs: dict) -> dict:
    # Call your real application here: a chain, an agent, an API
    answer = my_support_bot.invoke(inputs["question"])
    return {"answer": answer}

def contains_reference_facts(outputs: dict, reference_outputs: dict) -> dict:
    hit = reference_outputs["answer"].split(".")[0].lower() in outputs["answer"].lower()
    return {"key": "contains_key_fact", "score": 1.0 if hit else 0.0}

results = evaluate(
    target,
    data=client.list_examples(dataset_name="support-bot-golden", as_of="prod"),
    evaluators=[contains_reference_facts],
    experiment_prefix="prompt-v5-gpt-4o-mini",
    metadata={"dataset_tag": "prod", "prompt_version": "v5"},
    max_concurrency=4,
)

Notice the metadata on the experiment itself. Recording which dataset tag and prompt version the experiment used means that when you look at a results table three months from now, the provenance is right there instead of archaeologically reconstructed from timestamps. The evaluator here is deliberately simple string matching; in real projects you will mix heuristic evaluators like this with LLM-as-judge evaluators for fuzzier qualities such as tone or completeness, but the dataset mechanics are identical regardless of evaluator sophistication.

For CI, the same call runs inside a pytest test or a GitHub Actions step: evaluate the smoke split as of the prod tag, assert that the aggregate score clears your threshold, and fail the build otherwise. Because both the split membership and the tag are pinned, a red build always means your change caused it, never that someone edited test data mid-flight. When someone does improve the dataset, moving the prod tag is its own pull-request-like event, ideally accompanied by rerunning the current production system against the new version to establish a fresh baseline before any new changes are judged against it.

Common Mistakes and How to Avoid Them

The first and most damaging mistake is evaluating against latest everywhere. It feels natural, and it silently destroys comparability the first time anyone edits an example between two experiments. The fix costs one line: tag versions and reference tags. Second is skipping reference outputs because "we use an LLM judge anyway." Reference-free judges have their place, but they drift with the judge model and cannot catch factual regressions; even a rough human-reviewed reference makes every evaluator sharper. Third is letting one person own the dataset. Test data curation is a team sport, and LangSmith's UI editing plus annotation queues exist so that support engineers and PMs, the people who actually know what a correct answer looks like, can contribute without touching Python.

Fourth, watch for schema drift. If early examples use {"question": ...} and later ones use {"query": ...}, your target function will crash on half the dataset. Enforce a schema in whatever script writes examples, and consider a tiny validation pass with list_examples in CI that asserts every example has the expected keys. Fifth, resist the urge to delete embarrassing examples. When the model fails an example that you decide is unfair or outdated, edit it or move it to a retired split with a metadata note explaining why, rather than deleting it. Deletions are versioned too, so nothing is truly lost, but an explicit retirement trail keeps the team honest about why the pass rate improved. And finally, remember that dataset size follows quality, not the reverse. Fifty carefully chosen, well-referenced, well-tagged examples will guide your development better than two thousand unreviewed trace dumps, and they will keep your evaluation bills and CI times sane while doing it.

Where to Go from Here

You now have the full lifecycle: create a dataset with create_dataset and create_examples, grow it from real traffic with list_runs and source_run_id, import legacy suites with upload_csv, slice it with metadata and splits, pin versions with update_dataset_tag, and run reproducible experiments by passing as_of into evaluate. The pattern to internalize is the two-track workflow, continuous curation on latest and deliberate evaluation on tags, because it scales from a solo project to a team of twenty without changing a line of code. Start small this week: create one dataset, seed it with twenty examples from your best and worst production traces, tag it prod, and wire one evaluator into CI. Every improvement after that compounds. If you want to go deeper, with guided projects covering tracing, evaluators, annotation queues, pairwise experiments, and production monitoring on top of the dataset skills from this article, check out the LangSmith Tutorial course on teachyou.ai, where Ira Menon and I walk through building a complete evaluation pipeline for a real application, from the first trace to a fully automated regression suite.