LangSmith Tutorial: Tracing and Debugging LLM Apps

Your chain works in the notebook. It answers the test question correctly, the demo goes fine, and then it ships. Two weeks later a user reports a wrong answer, or your latency dashboard shows p95 climbing past six seconds, and you have no idea which of the eleven steps in your RAG pipeline is responsible. This is the point where print statements and hope stop being a debugging strategy. LangSmith exists for exactly this problem: it gives you a recorded, inspectable trace of everything that happened inside a chain, agent, or tool call, so you can find the slow step, the bad retrieval, or the malformed prompt without guessing. This tutorial walks through enabling tracing, reading a trace, using it to find the actual bug, turning real traces into a reusable eval dataset, and comparing prompt versions side by side.

Why tracing matters more than logging

Regular application logging tells you that a request came in and a response went out. It does not tell you what the LLM saw, what it returned, how many tokens it burned, or which of your five chained calls actually produced the bad output. LLM applications fail in ways traditional logging was never built to catch: a retriever returns irrelevant chunks, a prompt template silently truncates context, a tool call returns an error that gets swallowed and passed to the next step anyway, or the model just hallucinates under a specific input shape you didn't test for.

Tracing solves this by capturing the full execution tree of a run — every LLM call, every tool invocation, every intermediate output, with inputs and outputs preserved at each node. Once you have that tree, debugging stops being archaeology and starts being a matter of clicking into the step that looks wrong. LangSmith is built specifically for this: it's the tracing and evaluation layer designed to sit alongside LangChain (and LangGraph), though it also works with un-instrumented LLM calls if you wrap them manually.

If you've only ever debugged by adding print(chain.invoke(...)) calls and re-running, the mental shift here is important: you stop re-running the app to see what happened, and instead inspect what already happened, recorded, with full fidelity, from the one call that actually failed in front of a real user.

Enabling tracing on a LangChain app

Getting a trace flowing is mostly environment configuration — you're not usually rewriting your chain. LangChain checks for a small set of environment variables at runtime and, when it finds them, wraps every traceable component (chat models, retrievers, tools, chains, agents) with instrumentation automatically.

The minimum setup looks like this:

export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=ls__your_api_key_here
export LANGCHAIN_PROJECT=my-rag-app

LANGCHAIN_TRACING_V2 turns tracing on. LANGCHAIN_API_KEY authenticates the run against your LangSmith account. LANGCHAIN_PROJECT groups traces into a named project in the dashboard — set one per app or per environment (my-rag-app-dev, my-rag-app-prod) so staging noise doesn't pollute production traces.

If you'd rather not rely on environment variables (useful in tests, notebooks, or multi-tenant services where you want to route traces to different projects at runtime), you can configure tracing programmatically:

import os
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langsmith import Client

# Explicit configuration instead of relying on env vars
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_PROJECT"] = "support-bot-eval"

client = Client()  # picks up LANGCHAIN_API_KEY from the environment

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a support assistant. Be concise and cite the doc used."),
    ("human", "{question}"),
])
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
chain = prompt | llm

# Every invoke() call below is automatically traced because
# LANGCHAIN_TRACING_V2 is set — no extra wrapping needed.
response = chain.invoke({"question": "How do I reset my API key?"})

Because tracing hooks into LangChain's callback system, nested chains, retrievers, and tool calls inside an agent all get captured without you touching their code. If part of your pipeline is a raw API call that bypasses LangChain entirely (a plain requests.post to an internal service, say), you can still bring it into the trace with the @traceable decorator from the langsmith SDK so it shows up as a labeled step instead of a black box:

from langsmith import traceable

@traceable(name="fetch_customer_context")
def fetch_customer_context(customer_id: str) -> dict:
    # any non-LangChain call — a DB lookup, an internal API, etc.
    return db.get_customer(customer_id)

Enable tracing in a dev project first, confirm the shape of the traces looks right, then flip a separate LANGCHAIN_PROJECT on for production. Don't trace straight into a shared "default" project — you'll regret it the first time you're trying to filter dev noise out of a production incident.

A few things worth knowing before you flip this on in production. Tracing adds a small amount of overhead per call — the SDK batches and sends run data asynchronously, so it shouldn't block your main request path, but it's worth load-testing once rather than assuming it's free. Tags and metadata are your friends: pass tags=["prod", "v2-prompt"] or metadata={"user_tier": "enterprise"} on individual invocations, and you'll be able to filter the trace list by them later instead of scrolling chronologically. And decide early whether to trace 100% of production traffic or sample it — for a low-volume internal tool, trace everything; for a high-volume consumer app, trace a percentage and always trace anything that gets a negative user feedback signal, so you don't pay for volume you'll never look at.

What a trace actually shows you

Once tracing is on, every invoke, stream, or batch call produces a run in your LangSmith project. A single run is a tree, not a flat log line, and that tree is the whole point.

For a typical RAG chain, a trace shows:

• The root run — the overall input (the user's question) and the overall output (the final answer), plus total latency and total token usage for the entire call.
• Nested child runs — one node per retriever call, per prompt formatting step, per LLM call, per tool invocation, in the order they actually executed, indented to reflect nesting (a tool call inside an agent step inside the parent chain).
• Per-step timing — how long each node took, so you can see whether the retriever, the reranker, or the generation call is what's actually slow.
• Per-step token counts and cost — prompt tokens and completion tokens for every LLM call, which matters both for debugging (is my prompt template bloating context unexpectedly?) and for cost tracking.
• Exact inputs and outputs at each node — the literal prompt sent to the model (after templating, with all variables filled in), the literal completion returned, the exact documents a retriever returned, and the exact arguments passed to a tool along with what it returned.
• Errors and exceptions, attached to the specific node that raised them, rather than bubbling up to a generic top-level failure.

The nested structure is what makes this useful for anything beyond a single LLM call. A two-step agent that calls a search tool and then summarizes results produces a trace with the search call as one child and the summarization call as another — so when the final answer is wrong, you can check whether the search step returned garbage or the summarization step ignored good results.

It's also worth understanding what gets recorded at the message level, not just the chain level. For chat models, LangSmith captures the full message array — system prompt, prior turns, the current user message, any tool-call messages and their results — exactly as it was sent to the provider, along with the model name and any generation parameters like temperature or max_tokens. That last detail matters more than it sounds: a common production bug is someone bumping a default parameter in one code path and not another, and the only way to catch it reliably is seeing the actual parameters attached to the actual call that misbehaved, rather than trusting that your config file reflects reality.

For agents specifically, the trace also captures the reasoning sequence where applicable — the "thought, action, observation" steps a ReAct-style loop produces, with each iteration as its own child run. This is often where the most confusing bugs live, because an agent can loop on the same failing tool call two or three times before giving up and returning a degraded answer, invisible from the outside — the user just sees a slow, mediocre response with no indication the agent spent most of its time retrying a call that was never going to succeed.

Debugging a slow response

Latency problems in LLM apps are rarely evenly distributed — usually one step is disproportionately expensive, and it's not always the one you'd guess.

The workflow is straightforward once you have traces: open the slow run, look at the waterfall of child run durations, and find the node that ate most of the wall-clock time. Common culprits that traces make obvious:

• A retriever making a synchronous network call to a vector store with no connection pooling, adding hundreds of milliseconds per call.
• A reranking step that calls a second model over every retrieved document instead of batching them.
• An LLM call with an oversized prompt — often caused by a prompt template that's silently including full conversation history instead of a trimmed window, which you can see directly because the trace shows the exact rendered prompt and its token count.
• Sequential tool calls that could have run in parallel — visible in the trace because each child run has its own start and end timestamp, so you can see two tool calls running one after another instead of concurrently.

Because the trace records the actual token counts per step, you can distinguish "this step is slow because the model is slow" from "this step is slow because we're sending it 6,000 tokens of unnecessary context." Those have very different fixes — one is a model/provider choice, the other is a prompt engineering bug you can fix in five minutes once you've actually seen the bloated prompt instead of assuming your template is doing what you wrote it to do.

A practical habit here: sort or filter your project's run list by latency and look at the top decile before you look at anything else. Most latency problems aren't evenly spread across every request — they cluster around a specific input shape (very long user messages, a particular document type that retrieves poorly and triggers a fallback path, a rare branch in an agent that calls an expensive tool). Once you've pulled up five or six of the slowest traces from the last day and they all share the same slow node, you've found your bottleneck without needing to add a single timer to your code. Compare that against the traces sitting at the median — if the slow node takes the same absolute time in both, the fix is architectural (cache it, parallelize it, swap the model); if it only blows up for the slow group, the fix is closer to the input (truncate long inputs, add a cheaper pre-filter before the expensive step runs).

Debugging an incorrect response

Wrong answers are harder to catch with timing alone, so the debugging path is a bit different: you inspect the actual content at each node rather than the duration.

Start at the final output and work backward. If the last LLM call produced a wrong answer, check its exact input first — the trace shows you the fully rendered prompt, not the template. It's common to discover the model was never actually given the information it needed: a variable substitution bug dropped a field, a context window truncation cut off the relevant part of a long document, or a retrieval step returned documents that don't actually answer the question.

For a RAG pipeline specifically, this usually means checking, in order:

1. The retriever's child run — what documents did it actually return, and do any of them contain the answer? If not, the bug is in retrieval (embedding model mismatch, wrong index, bad chunking), not generation.
2. The prompt-formatting step — did the retrieved documents actually make it into the prompt sent to the model, or did a template bug drop them?
3. The LLM call itself — given the exact prompt shown in the trace, is the model's answer actually reasonable? If the prompt looks right but the answer is still wrong, that's a model or prompt-wording problem, not a data problem.

This is the value proposition in one sentence: instead of re-running the chain locally and guessing where it goes wrong, you're looking at the recorded, exact record of the run that actually failed for a real user, with every intermediate value preserved.

Two other things are worth checking once you're inside a suspicious trace. First, tool-calling agents: if a tool call's arguments look wrong, work out whether the model was ever given enough information to construct the right arguments in the first place — a lot of "the agent called the wrong function" bugs are actually "the tool's description or schema was ambiguous," and the trace shows you the exact tool schema the model was working from. Second, multi-turn conversations: check whether the full conversation history made it into context or got silently truncated by a message-window limit, since a wrong answer late in a long conversation is very often the model losing access to something said five turns earlier rather than the model reasoning badly about what it can see.

If you're working with a team, it also helps to attach human feedback directly to runs as you find issues — a thumbs-down, a correctness score, a free-text note — rather than tracking bug reports in a separate spreadsheet. Feedback attached to a run in LangSmith becomes filterable later ("show me every run tagged incorrect this month"), which is exactly the query you need when it's time to build a dataset, covered next.

Turning real traces into an eval dataset

Once you've found a handful of bad or interesting traces, the next move is not to fix the prompt and move on — it's to capture those exact inputs as regression cases before you touch anything. This is where a lot of teams stop short: they fix the bug for the one example they saw, ship it, and have no way of knowing whether the fix broke a different case or whether the same failure mode recurs three weeks later.

LangSmith lets you take runs directly from a trace view and add them to a dataset — a persistent collection of input/output pairs (optionally with a reference or "golden" output) that you can re-run future chain versions against. The typical flow: filter traces (by error, by user feedback score, by a specific date range where you know something went wrong), select the ones worth keeping, and add them to a dataset, either with the actual output as the reference or with a corrected output you write in by hand.

Programmatically, pulling traces into a dataset looks like this:

from langsmith import Client

client = Client()

# Grab recent runs from a project, filtered to ones flagged
# with negative user feedback or a "wrong" tag
runs = client.list_runs(
    project_name="support-bot-eval",
    filter='and(eq(feedback_key, "correctness"), eq(feedback_score, 0))',
)

dataset = client.create_dataset(
    dataset_name="support-bot-regressions",
    description="Real production failures flagged as incorrect by users",
)

for run in runs:
    client.create_example(
        inputs=run.inputs,
        outputs=run.outputs,  # or a hand-corrected reference output
        dataset_id=dataset.id,
    )

This gives you a dataset built entirely out of cases that actually happened — not synthetic examples someone dreamed up, which is a meaningfully different (and better) source of eval coverage. Every time you change a prompt, swap a model, or restructure a chain, you re-run this dataset before shipping, and you find out immediately whether you fixed the original problem and whether you broke anything that used to work. This is also how you build institutional memory about your failure modes instead of re-discovering the same bug in production every few months.

A dataset built purely from failures is useful but incomplete — it's worth also sampling a handful of traces where the chain worked correctly and adding those alongside the failures. Otherwise you can end up in a state where every future prompt change is optimized purely to fix known failures, without any signal telling you if that same change broke something that used to work fine. A healthy regression dataset has both: known-bad cases you're actively trying to fix, and known-good cases you're actively trying not to break. Keep growing it over time as new failure modes surface — treat it the same way you'd treat a test suite in ordinary software, where every production bug ideally gets a corresponding regression case the moment it's found, not after the fact when someone remembers to add it.

Comparing two versions of a prompt or chain

Once you have a dataset, comparing chain versions is a matter of running both versions against the same inputs and diffing the outputs — which is exactly what LangSmith's evaluation runs are for. You define an experiment against a dataset, run version A, run version B, and get a side-by-side comparison view showing, per example, what each version returned, how long it took, and how many tokens it used.

from langsmith.evaluation import evaluate

def run_chain_v1(inputs: dict) -> dict:
    return {"answer": chain_v1.invoke(inputs["question"])}

def run_chain_v2(inputs: dict) -> dict:
    return {"answer": chain_v2.invoke(inputs["question"])}

# Run both versions against the same regression dataset
results_v1 = evaluate(
    run_chain_v1,
    data="support-bot-regressions",
    experiment_prefix="prompt-v1",
)

results_v2 = evaluate(
    run_chain_v2,
    data="support-bot-regressions",
    experiment_prefix="prompt-v2",
)

Both experiments show up against the same dataset in the dashboard, which is what makes the comparison meaningful — you're not eyeballing two separate runs from memory, you're looking at the same input producing two different outputs, next to each other, with the same evaluators applied to both. This is the difference between "I think the new prompt is better" and having a specific count of how many previously-failing examples now pass, and whether any previously-passing examples regressed.

This matters most when a prompt change looks like a strict improvement in the three examples you tested by hand, but quietly regresses a case you didn't think to check. A regression dataset built from real traces, combined with a side-by-side experiment comparison, is what catches that before a user does.

Attaching evaluators instead of eyeballing outputs

Running two prompt versions side by side is useful even with a human reading every row, but that stops scaling past a few dozen examples. The natural next step is attaching an evaluator function to the evaluate() call so each output gets scored automatically — exact match or a rule-based check for structured outputs, and a model-graded evaluator for anything open-ended like "is this answer helpful and grounded in the retrieved documents."

def correctness_evaluator(run, example) -> dict:
    predicted = run.outputs["answer"]
    reference = example.outputs["answer"]
    score = 1.0 if reference.lower() in predicted.lower() else 0.0
    return {"key": "correctness", "score": score}

results_v2 = evaluate(
    run_chain_v2,
    data="support-bot-regressions",
    evaluators=[correctness_evaluator],
    experiment_prefix="prompt-v2-scored",
)

A simple substring or exact-match evaluator works for narrow, structured tasks, but most real answers are too open-ended for string matching to mean much — two correct answers can be worded completely differently. That's the gap LLM-as-a-Judge fills: instead of a hand-written rule, you use a separate model call to grade each output against a rubric (or against the reference answer), and LangSmith treats that judge's score exactly like any other evaluator in the comparison view. Start there once your regression dataset has grown past the size where you're comfortable reading every row by hand — it's the natural next step after everything covered here, and worth its own dedicated walkthrough given how much the judge prompt itself affects the reliability of the scores it produces.