LangSmith for Fine-Tuning Data Collection

Every team that fine-tunes a model hits the same wall, and it is never the training job itself. The API call to kick off fine-tuning takes thirty seconds to write. The wall is the dataset: hundreds or thousands of high-quality prompt-completion pairs that actually represent what your users ask and what a good answer looks like. Most teams respond by hand-writing synthetic examples in a spreadsheet, which is slow, expensive, and subtly wrong, because invented examples never match real production traffic. Meanwhile, the best possible training data is already flowing through your application every single day: real user inputs, real model outputs, and real signals about which responses were good. If you have LangSmith tracing enabled, that data is already captured. The entire discipline of collecting langsmith fine-tuning data is about turning that passive stream of traces into a curated, versioned, exportable dataset — and LangSmith gives you every primitive you need to do it: run querying, feedback scores, annotation queues, datasets, and clean export paths to JSONL. This article walks through the full pipeline, from instrumenting your app to submitting a fine-tuning job trained entirely on your own production traffic.

Why Production Traces Beat Synthetic Data for Fine-Tuning

Fine-tuning works best when the training distribution matches the inference distribution. That sounds obvious, but it is the single most violated principle in practice. Teams write training examples the way they imagine users behave: clean grammar, complete sentences, one question at a time. Real users paste half a stack trace, misspell the product name, switch languages mid-sentence, and ask three things at once. A model fine-tuned on imaginary traffic performs beautifully on imaginary traffic and mediocre on the real thing.

Production traces solve this by construction. Every trace in LangSmith is a real input your system actually received, paired with the output your pipeline actually produced. When you fine-tune on curated traces, you are teaching the model the exact distribution it will face tomorrow, because it is the distribution it faced yesterday.

There is a second, more strategic reason to collect this data: model distillation. A common and very effective pattern is to run an expensive frontier model in production, capture its best outputs through tracing, then fine-tune a much smaller and cheaper model on those outputs. The small model learns to imitate the large model on your specific task, and you cut inference costs dramatically while keeping quality close to the original. None of that is possible without a disciplined trace-collection pipeline, which is exactly what LangSmith provides.

The third reason is compounding value. Traces accumulate automatically. Six months from now, when you decide to fine-tune for a new capability or migrate to a new base model, you will already have tens of thousands of candidate examples sitting in your project, timestamped, tagged, and scored. Teams that treat observability as a data collection strategy — not just a debugging tool — build a durable asset that gets more valuable every week.

Instrumenting Your Application So Traces Are Worth Keeping

Before you can curate data, you need traces rich enough to curate. Bare-minimum tracing captures inputs and outputs, but fine-tuning curation needs more: metadata to filter on, tags to segment by, and stable identifiers to join feedback against. The good news is that LangSmith tracing is nearly zero-effort if you use LangChain, and only slightly more effort with the raw SDK.

The environment setup is three variables:

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="your-api-key"
export LANGSMITH_PROJECT="support-bot-prod"

With those set, every LangChain invocation is traced automatically. For custom code, the @traceable decorator does the same job and lets you attach the metadata that will make filtering possible later:

from langsmith import traceable
from openai import OpenAI

client = OpenAI()

@traceable(
    run_type="llm",
    name="support_answer",
    tags=["support", "v2-prompt"],
    metadata={"model": "gpt-4o", "prompt_version": "2.3"},
)
def answer_ticket(question: str, context: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": f"Answer using this context:\n{context}"},
            {"role": "user", "content": question},
        ],
        temperature=0.2,
    )
    return response.choices[0].message.content

Two habits matter enormously here. First, record the prompt version in metadata on every run. When you later assemble a fine-tuning dataset, you almost always want examples from a single prompt version, because mixing outputs generated under different system prompts teaches the model inconsistent behavior. Second, tag runs by feature or intent — support, summarization, sql-generation — so you can build task-specific datasets instead of one muddy pile.

One more practical note: think about PII from day one. If your traffic contains emails, phone numbers, or account identifiers, configure input masking or run a scrubbing step before traces are logged. Cleaning PII out of a dataset after export is painful; preventing it from entering traces is cheap.

Capturing Feedback: The Signal That Separates Good Runs from Bad

A pile of traces is not a dataset. The difference between the two is a quality signal, and in LangSmith that signal is feedback. Feedback is a score attached to a specific run, and it can come from three sources: end users, application logic, and human reviewers.

End-user feedback is the classic thumbs-up widget. When a user clicks it, your frontend calls your backend, and your backend logs feedback against the run ID:

from langsmith import Client

ls_client = Client()

def record_user_feedback(run_id: str, thumbs_up: bool, comment: str = ""):
    ls_client.create_feedback(
        run_id=run_id,
        key="user_rating",
        score=1.0 if thumbs_up else 0.0,
        comment=comment,
    )

To make this work, your API responses need to carry the run ID back to the client. Generate the run ID yourself before invocation, or read it from the trace context, and return it alongside the model output. This tiny bit of plumbing is the highest-leverage engineering work in the whole pipeline, because it converts every user interaction into a labeled example.

Programmatic feedback is the second source, and it is underused. If your pipeline validates outputs — the JSON parsed, the SQL executed without error, the citation actually exists in the retrieved context — log that validation result as feedback too. A run where the generated SQL executed successfully and returned rows is a strong positive signal that cost you nothing to collect.

Implicit signals are the third source, and often the most honest. Did the user copy the response? Did they regenerate it? Did they rephrase the question immediately afterward (a strong negative)? Did the conversation end successfully? Each of these can be logged as a distinct feedback key. When curation time comes, a run with user_rating=1, json_valid=1, and no regeneration is close to a guaranteed keeper.

Querying and Filtering Runs with the LangSmith SDK

Once traces and feedback are flowing, curation becomes a query problem. The LangSmith SDK exposes list_runs with a filter syntax that lets you slice your production history precisely. The goal is to pull only the runs that meet your quality bar and match the task you are fine-tuning for.

Here is a realistic curation query: successful LLM runs from the production project, on the current prompt version, with positive user feedback, from the last 60 days:

from datetime import datetime, timedelta
from langsmith import Client

client = Client()

runs = client.list_runs(
    project_name="support-bot-prod",
    run_type="llm",
    start_time=datetime.now() - timedelta(days=60),
    filter=(
        'and('
        'eq(feedback_key, "user_rating"), '
        'eq(feedback_score, 1), '
        'has(tags, "support"), '
        'eq(metadata_key, "prompt_version"), '
        'eq(metadata_value, "2.3")'
        ')'
    ),
    is_root=False,
    error=False,
)

candidates = [
    r for r in runs
    if r.outputs and len(r.outputs.get("output", "")) > 50
]
print(f"{len(candidates)} candidate runs for fine-tuning")

A few filtering patterns come up constantly in real curation work. Filter out errored runs and runs with empty outputs first — they are noise. Deduplicate near-identical inputs, because production traffic is heavily repetitive and fifty copies of the same question will skew your training distribution toward it. Enforce a minimum output length to drop degenerate one-word responses, and consider a maximum too, since runaway outputs are usually failures. Finally, sample across time rather than taking the most recent runs only; traffic patterns shift, and a dataset drawn entirely from one week inherits that week's quirks.

For latency- or cost-motivated distillation projects, you can also filter on token counts and latency directly, selecting examples where the expensive model produced compact, fast, high-rated answers — exactly the behavior you want the small model to learn.

Annotation Queues: Human Review Before Data Ships

Automated filters get you from a hundred thousand runs to a few thousand candidates. Human review gets you from candidates to training data. LangSmith annotation queues are built for exactly this step: you push runs into a queue, reviewers work through them in a purpose-built UI, and each run comes out approved, corrected, or rejected.

The workflow is straightforward. Create a queue with a rubric — the written definition of what a good response looks like for this task — and add your candidate runs to it programmatically. Reviewers then see one run at a time: the input, the output, and structured scoring fields you define. They can attach feedback scores, leave comments, and critically, they can edit the output before approving it. That editing capability is what makes annotation queues more than a rating tool. A response that is 90 percent right becomes a perfect training example after a reviewer fixes the last 10 percent, which is far cheaper than writing an example from scratch.

A few practices make annotation dramatically more effective. Write the rubric down before anyone reviews anything, and include concrete examples of accept, fix, and reject decisions; without a shared rubric, two reviewers will disagree on a third of the items and your labels will be mush. Start with a small calibration batch — fifty runs reviewed by two people independently — and measure agreement before scaling up. Keep sessions short, because labeling quality decays fast after about an hour. And route only genuinely uncertain items to humans: runs with strong positive automated signals can flow into the dataset directly, runs with strong negative signals can be discarded directly, and human attention should concentrate on the ambiguous middle where it actually changes outcomes.

Rejected runs deserve a second look before you delete them. Runs where the model failed, paired with a reviewer-written correction, are the seed of a targeted improvement dataset — you are literally collecting the model's mistakes along with the right answers. Some of the most effective fine-tunes are built primarily from corrected failures rather than from successes.

Building and Versioning Datasets in LangSmith

Approved examples need a durable home, and in LangSmith that home is a dataset. A dataset is a named collection of examples, each with inputs and outputs, that supports versioning, splits, and export. Moving curated runs into a dataset is a few lines of code:

from langsmith import Client

client = Client()

dataset = client.create_dataset(
    dataset_name="support-finetune-v1",
    description="Curated support answers, prompt v2.3, human-approved",
)

for run in approved_runs:
    client.create_example(
        dataset_id=dataset.id,
        inputs=run.inputs,
        outputs=run.outputs,
        metadata={
            "source_run_id": str(run.id),
            "reviewed": True,
            "prompt_version": "2.3",
        },
    )

Storing the source run ID in example metadata is a habit worth keeping. It gives you full provenance: any example in any future training set can be traced back to the production interaction it came from, which matters for debugging strange model behavior and for compliance conversations alike.

Versioning is the feature that separates disciplined teams from chaotic ones. LangSmith datasets are versioned automatically as examples are added and modified, and you can pin a fine-tuning job to a specific dataset version. That means the statement "model support-ft-3 was trained on dataset support-finetune-v1 as of March 12" is verifiable, reproducible, and auditable. When a fine-tuned model behaves unexpectedly, the first question is always "what exactly was it trained on," and versioned datasets let you answer it in seconds instead of archaeology sessions.

Use dataset splits deliberately. Hold out a test split from the very first version and never train on it. This split becomes your fixed measuring stick: every fine-tuned model, and the base model, gets evaluated against the same held-out examples, so improvements are real rather than artifacts of a shifting benchmark.

Exporting to JSONL and Running the Fine-Tuning Job

With a curated, versioned dataset in place, the final transformation is mechanical: convert examples into the chat-format JSONL that fine-tuning APIs expect. Each training example becomes a JSON object with a messages array containing the system prompt, the user input, and the approved assistant output.

import json
from langsmith import Client

client = Client()

SYSTEM_PROMPT = "You are a concise, accurate support assistant for our product."

examples = client.list_examples(dataset_name="support-finetune-v1")

with open("train.jsonl", "w") as f:
    for ex in examples:
        record = {
            "messages": [
                {"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": ex.inputs["question"]},
                {"role": "assistant", "content": ex.outputs["output"]},
            ]
        }
        f.write(json.dumps(record) + "\n")

Then submit the file to your provider of choice:

from openai import OpenAI

oai = OpenAI()

training_file = oai.files.create(
    file=open("train.jsonl", "rb"),
    purpose="fine-tune",
)

job = oai.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-4o-mini-2024-07-18",
    suffix="support-v1",
)
print(job.id)

Two decisions in this step deserve real thought. The first is the system prompt. If you bake the full production system prompt into every training example, the fine-tuned model expects to see it at inference time too. Many teams intentionally train with a much shorter system prompt — or none — so the fine-tuned model internalizes the behavior instead of relying on prompt text, which is a large part of the cost savings. Whichever you choose, be consistent across every example and use the same convention at inference time.

The second is dataset size and balance. Meaningful behavior change typically starts at a few hundred well-curated examples, and quality beats quantity at every scale that matters; five hundred human-approved examples routinely outperform five thousand unfiltered ones. Check the balance of your dataset across intents and input types before training, because the model will faithfully learn whatever imbalance you feed it.

Closing the Loop: Evaluating the Fine-Tuned Model Before Rollout

A fine-tuning job that finishes is not a fine-tuning job that succeeded. Before the new model sees production traffic, it needs to prove itself on your held-out split, and LangSmith's evaluation tooling makes this a short script rather than a project. Define a target function that calls the fine-tuned model, point evaluate at the test split of your dataset, attach evaluators — exact-match or structural checks where the task allows, LLM-as-judge scoring for open-ended quality — and run the same evaluation against the base model and the previous fine-tune as baselines.

The comparison view in LangSmith then shows you, example by example, where the new model improved and where it regressed. Regressions are the interesting part. A fine-tune that improves average quality but breaks a specific intent category is not ready, and aggregate scores hide exactly this failure mode. Slice results by the tags and metadata you attached during collection, and inspect every category independently.

Once the model ships, the loop closes on itself. The fine-tuned model's production traffic is traced like everything else, users rate its responses, annotation queues fill with its ambiguous cases, and the next dataset version begins accumulating automatically. This is the real payoff of building the pipeline on LangSmith rather than on ad-hoc scripts: fine-tuning stops being a one-off project and becomes a flywheel. Each cycle of collect, curate, train, evaluate, deploy makes the next cycle cheaper, because the infrastructure — tracing, feedback, queues, datasets, evaluations — is already in place and already integrated.

Watch for data drift between cycles. If your product changed, your prompt changed, or your user base shifted, examples collected under the old regime may teach outdated behavior. Dataset versioning plus per-example metadata makes it easy to retire stale slices without discarding the whole corpus.

Common Mistakes That Ruin Fine-Tuning Datasets

Most failed fine-tunes trace back to the dataset, and the same handful of mistakes appear again and again.

• Training on unfiltered traces. Raw production output includes the model's bad days. Fine-tuning on everything teaches the model to reproduce its own failures with more confidence.
• Mixing prompt versions. Outputs generated under different system prompts encode contradictory behaviors. Filter to one version, or normalize deliberately.
• Skipping deduplication. Production traffic is repetitive; a dataset where one question appears two hundred times produces a model obsessed with that question.
• Leaking test data. If any training example also appears in your evaluation split, your metrics are fiction. Deduplicate across splits, not just within them.
• Ignoring negative signals. Regenerations, immediate rephrasings, and abandoned conversations mark bad examples that thumbs-down buttons miss, because most users never click anything.
• Treating annotation as an afterthought. Unreviewed data at scale is just automated noise. A smaller human-approved dataset nearly always wins.
• Forgetting provenance. Without source run IDs and dataset versions, you cannot explain why a model behaves the way it does, and every debugging session starts from zero.

Every one of these mistakes is cheap to prevent at collection time and expensive to fix after training. The pipeline described in this article — rich tracing, multi-source feedback, filtered queries, annotation queues, versioned datasets, held-out evaluation — exists precisely to make the cheap prevention automatic.

Fine-tuning is not a modeling problem; it is a data logistics problem, and LangSmith turns that logistics problem into a workflow you can run every month instead of a heroic one-off effort. Start by instrumenting your traces properly and wiring up one feedback signal this week — the dataset you will want in six months is being generated by your users right now, and the only question is whether you are capturing it. If you want to go deeper into every piece of this pipeline — tracing internals, feedback design, annotation workflows, dataset management, and evaluation — our LangSmith Tutorial course on teachyou.ai walks through the complete system hands-on, from your first trace to your first production fine-tune.