LangSmith for Multi-Agent Trace Analysis

Why Multi-Agent Systems Are So Hard to Debug

A single-agent LLM application is already tricky to debug. You have a prompt, a model call, maybe a few tool invocations, and a final answer. When something goes wrong, you can usually read the one conversation transcript and spot the problem.

Multi-agent systems are a different animal entirely. A supervisor agent delegates to a researcher, the researcher calls a search tool three times, hands its findings to a writer agent, the writer asks the supervisor for clarification, and the supervisor routes the question to a critic agent that was never supposed to be involved. The final output is wrong, and now you have to figure out which of the five agents, twelve LLM calls, and eight tool executions actually caused the failure. Print statements will not save you here. Logs scattered across agents give you fragments without causality. What you need is a single, hierarchical record of the entire execution — every agent, every handoff, every token — stitched together in the order it actually happened.

That is exactly what LangSmith multi-agent tracing gives you. In this guide, we will build up from the basics of how LangSmith structures a trace, instrument a realistic multi-agent system, and then walk through the analysis workflows that let you find routing bugs, runaway loops, and cost hotspots in minutes instead of days.

How LangSmith Structures a Trace: Runs, Trees, and Projects

Before you can analyze multi-agent traces, you need a mental model of what LangSmith actually records. There are three core concepts.

A run is the atomic unit. Every operation — an LLM call, a tool execution, a retriever query, a chain step, an agent turn — becomes a run. Each run captures its inputs, outputs, start and end timestamps, latency, token usage, errors, and any metadata or tags you attach.

A trace is a tree of runs. The top-level run is the root (usually your entry point — the user request hitting your graph or supervisor), and every operation triggered inside it becomes a child run. Children can have their own children, so a supervisor that calls a researcher agent that calls a search tool produces a three-level tree. This nesting is the killer feature for multi-agent work: the tree structure mirrors your delegation structure. When you open a trace in the LangSmith UI, you see the whole execution as a collapsible tree, and you can expand exactly the branch you care about.

A project is a named collection of traces. You typically use one project per application per environment — something like support-copilot-prod and support-copilot-dev — so production traffic and local experiments never get mixed together.

Runs also have a run type (llm, tool, chain, retriever, and so on), which drives how the UI renders them and how you can filter. LLM runs get the chat-style message view with token counts; tool runs show raw inputs and outputs. In a multi-agent trace, agent nodes typically show up as chain runs wrapping a sequence of LLM and tool runs.

The most important consequence of this model: you do not have to manually correlate logs across agents. As long as all agents execute within the same traced call stack, LangSmith assembles the tree for you. The hierarchy is the correlation.

Setting Up Tracing for a Multi-Agent Project

If your agents are built on LangChain or LangGraph, tracing is nearly zero-effort. Set a few environment variables and every component instruments itself automatically:

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="your-api-key"
export LANGSMITH_PROJECT="multi-agent-support-bot"

With those variables set, every graph invocation, model call, and tool execution in your process is captured and sent to the multi-agent-support-bot project. No code changes required.

If parts of your system are plain Python — custom orchestration code, agents built directly on a model SDK, glue functions between frameworks — you instrument them with the @traceable decorator from the LangSmith SDK. This matters a lot in real multi-agent systems, because the interesting bugs often live in the glue code between agents, not inside the agents themselves.

from langsmith import traceable
from langsmith.wrappers import wrap_openai
from openai import OpenAI

client = wrap_openai(OpenAI())

@traceable(run_type="chain", name="research_agent")
def research_agent(question: str) -> str:
    plan = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Plan a research approach."},
            {"role": "user", "content": question},
        ],
    )
    findings = run_search_tools(plan.choices[0].message.content)
    return findings

@traceable(run_type="tool", name="run_search_tools")
def run_search_tools(plan: str) -> str:
    # tool logic here
    ...

@traceable(name="supervisor")
def supervisor(user_request: str) -> str:
    findings = research_agent(user_request)
    return writer_agent(findings)

Because research_agent and run_search_tools are called inside supervisor, LangSmith nests them automatically. The wrapped OpenAI client means even raw SDK calls appear as proper LLM runs with token counts, inside whichever agent made them. You can freely mix decorated functions, LangChain runnables, and LangGraph graphs in one call stack — they all join the same trace tree.

One practical tip: give every agent an explicit, human-readable name. When you are scanning a trace tree with forty runs at 2 a.m., research_agent beats RunnableSequence every single time.

Tracing a LangGraph Supervisor Architecture

Let us make this concrete with the most common multi-agent pattern: a supervisor that routes work between specialist agents. Here is a compact LangGraph setup with a supervisor, a researcher, and a writer:

from langgraph.graph import StateGraph, MessagesState, START, END
from langchain_openai import ChatOpenAI
from typing import Literal

model = ChatOpenAI(model="gpt-4o")

def supervisor(state: MessagesState) -> dict:
    decision = model.invoke(
        [{"role": "system",
          "content": "Route to 'researcher', 'writer', or 'done'."}]
        + state["messages"]
    )
    return {"messages": [decision]}

def researcher(state: MessagesState) -> dict:
    result = model.invoke(
        [{"role": "system", "content": "You research facts."}]
        + state["messages"]
    )
    return {"messages": [result]}

def writer(state: MessagesState) -> dict:
    result = model.invoke(
        [{"role": "system", "content": "You write the final answer."}]
        + state["messages"]
    )
    return {"messages": [result]}

builder = StateGraph(MessagesState)
builder.add_node("supervisor", supervisor)
builder.add_node("researcher", researcher)
builder.add_node("writer", writer)
builder.add_edge(START, "supervisor")
builder.add_conditional_edges("supervisor", route_decision)
builder.add_edge("researcher", "supervisor")
builder.add_edge("writer", END)
graph = builder.compile()

graph.invoke(
    {"messages": [{"role": "user", "content": "Summarize our Q3 churn drivers."}]},
    config={"metadata": {"user_id": "u_482", "session": "s_9911"},
            "tags": ["prod", "churn-analysis"]},
)

With tracing enabled, this one invocation produces a trace whose root is the graph run. Under it you will see a supervisor run, then a researcher run, then supervisor again, then writer — each containing its own LLM child run with the exact messages sent and received. The trace tree literally reads like a transcript of the delegation: who was called, in what order, with what context, and what they returned.

Notice the metadata and tags passed in the config. These flow onto every run in the trace and become your filtering handles later. If you also use subgraphs — for example, the researcher is itself a multi-node graph — those appear as nested subtrees, so a deep hierarchy of agents stays navigable.

Reading a Multi-Agent Trace Tree Like a Pro

Opening a big trace for the first time can feel overwhelming, so here is a systematic way to read it.

Start at the root and check the shape. Before reading any content, look at the sequence of top-level children. Does the order of agent runs match the architecture you designed? A huge fraction of multi-agent bugs are visible purely in the shape: an agent that ran twice, an agent that never ran, a handoff that went to the wrong specialist. If your trace shows supervisor → writer but the question clearly needed research first, you have found a routing bug without reading a single prompt.

Follow the state, not just the messages. Each run shows its inputs and outputs. In a supervisor system, the input to each agent tells you exactly what context it received at handoff time. The classic failure is context starvation: the writer produced a vague answer because the researcher's findings never made it into the writer's input. Comparing the researcher's output with the writer's input makes that gap visually obvious.

Drill into the LLM runs last. Only once the shape and the handoffs look right should you open individual LLM runs to inspect prompts and completions. The chat view shows the fully rendered message list — system prompt, accumulated history, tool results — which is what the model actually saw, not what you think your template produces. Template bugs, duplicated system prompts, and history that grows with junk are all caught here.

Use latency and token columns as a heat map. Each node in the tree shows its duration and token usage. Scan for the outliers first. If one researcher invocation took a very long time relative to its siblings, expand it — you will often find a tool retrying, or a model call with a message history far larger than expected.

Check the error markers. Failed runs are flagged in the tree. In multi-agent systems, a tool error deep in one branch often gets swallowed: the agent catches it, apologizes in prose, and the supervisor happily continues with degraded output. The trace shows both the buried error and how it propagated — something aggregate logging almost never reveals.

Filtering, Searching, and Grouping Traces Across Agents

Trace-by-trace inspection is how you debug one bad output. To understand systemic behavior, you work at the project level, and that is where metadata, tags, and filters earn their keep.

The LangSmith UI lets you filter runs by name, run type, error status, latency, token counts, tags, metadata keys, and full-text content. Some genuinely useful multi-agent queries you can express:

• All traces where the researcher node errored in the last day — isolates one agent's failure rate from the rest of the system.
• Root runs where total latency exceeded your SLO — then open the slowest ones and see which agent dominated the time.
• Traces tagged prod whose metadata user_id matches a specific complaining customer — reproduce exactly what they experienced.
• All runs of a specific agent name so you can read that agent's behavior across many conversations in one list, rather than hunting through full traces.

You can also do this programmatically with the SDK, which is how you build custom multi-agent analytics:

from langsmith import Client

client = Client()

runs = client.list_runs(
    project_name="multi-agent-support-bot",
    filter='and(eq(name, "researcher"), eq(is_root, false))',
    start_time=yesterday,
)

for run in runs:
    total = run.total_tokens or 0
    print(run.trace_id, run.latency, total, run.error)

Pulling per-agent runs like this lets you answer questions the UI does not directly chart: which agent contributes the most tokens per conversation, how often the supervisor loops back to the same specialist, or how the researcher's latency distribution shifted after a prompt change. Because every run carries its trace_id, you can always jump from an anomalous agent run back to the full trace it belongs to.

A discipline worth adopting from day one: define a small, consistent metadata schema — user_id, session_id, agent_version, experiment — and attach it at the entry point so it propagates everywhere. Filtering is only as good as the fields you recorded.

Debugging the Classic Multi-Agent Failure Patterns

After enough time staring at multi-agent traces, you notice the same handful of failure patterns showing up in every system. Here is how each looks in LangSmith and what to do about it.

The infinite delegation loop. The supervisor sends work to an agent, is unhappy with the result, sends it again, and again, until a recursion limit kills the run. In the trace tree this is unmistakable: the same agent name repeating many times under the root, usually with near-identical inputs. The fix is usually in the supervisor prompt (no clear termination criterion) or in state handling (the specialist's output never gets marked as complete). The trace shows you which — compare the supervisor's input on iteration one and iteration five; if nothing changed, your state update is broken.

The silent tool failure. A tool run errors or returns empty, the calling agent smooths over it with confident-sounding text, and downstream agents build on fiction. Filter for tool runs with errors or empty outputs, then walk up the tree to see what the agent claimed afterward. The remedy is making tool failures loud in state so the supervisor can react, and the trace gives you the exact conversations to turn into regression test cases.

Context starvation at handoff. Agent B needed something Agent A produced, but the orchestration only passed a summary, or the wrong state key. Diff Agent A's output against Agent B's input in the trace. This bug is almost impossible to find from final outputs alone — the answer is just mysteriously mediocre — but it is one click of comparison in a trace tree.

Context flooding. The opposite problem: every agent receives the full accumulated history of every other agent, token counts balloon, and quality drops as models drown in irrelevant text. The token column exposes this instantly — if late-stage LLM runs show input sizes many times larger than early ones, your handoffs are passing too much.

Role bleed. Two agents with similar prompts start answering each other's questions — the critic starts rewriting instead of critiquing. Reading several traces of the two agents side by side (filter by agent name) makes the prompt overlap obvious in a way unit-testing each agent alone never will.

Analyzing Cost and Latency Per Agent

Multi-agent architectures multiply model calls, so cost analysis stops being optional. A single user request might fan out into ten or more LLM invocations, and without per-agent attribution you only know your total bill went up, not why.

Because every LLM run in a trace records token usage and model name, and every run carries its position in the tree, LangSmith can attribute cost hierarchically. At the trace level you see what one conversation cost end to end. Inside the tree, you see how that cost splits across agents. At the project level, dashboards aggregate token usage, cost, latency percentiles, and error rates over time, and you can slice by tags and metadata — which means slicing by agent version, experiment, or customer tier if you recorded them.

The recurring discoveries teams make once they look:

• One specialist agent accounts for the majority of spend because it re-reads the entire conversation history on every turn, even though it only needs the latest instruction.
• The supervisor, which everyone assumed was cheap "routing", makes a frontier-model call per hop and every conversation involves several hops. Downgrading the routing model to a small, fast one often cuts noticeable cost with zero quality impact — and you can verify the impact in the same dashboards.
• Latency is dominated not by any model but by a tool: one slow API call inside one agent stretches every conversation. The waterfall view of the trace, where child durations are laid out on a timeline, makes serial bottlenecks and missed parallelization opportunities visible at a glance.

A practical workflow: tag deployments with an agent_version metadata field, ship a prompt or model change to one agent, then compare cost and latency distributions between versions in the monitoring charts. You get an honest, production-data answer to "did that optimization actually help" instead of a guess.

From Traces to Evaluation: Closing the Loop

Trace analysis finds bugs; evaluation keeps them fixed. LangSmith connects the two directly, and this loop is where multi-agent quality work compounds.

Every interesting trace can be added to a dataset with a couple of clicks: the trace inputs become the test case, and a corrected output becomes the reference. When you find a routing failure, a starved handoff, or a hallucinated tool result while debugging, capture it. Over a few weeks you accumulate a regression suite made of real failures — far more valuable than synthetic test cases invented in a vacuum.

You can then run experiments against that dataset: execute your graph over every example and score the results with evaluators — LLM-as-judge for answer quality, custom code evaluators for structural checks. For multi-agent systems specifically, evaluate more than the final answer. Because traces preserve the full execution, you can write evaluators over trajectories: did the supervisor route to the researcher for questions that require facts, did the loop terminate within N hops, did the writer's input contain the researcher's findings? Trajectory-level assertions catch regressions that final-output scoring misses, because a multi-agent system can stumble into a right answer through a wrong path — until one day it doesn't.

Finally, feedback ties production reality back into the same system. Thumbs up/down from users, scores from online evaluators run on sampled traffic, or annotations from a human review queue all attach to runs as feedback. Filter for traces with negative feedback, read the trees, find the responsible agent, fix it, add the case to the dataset, and re-run the experiment. That cycle — trace, diagnose, capture, evaluate — is the operational heartbeat of a serious multi-agent product.

Practical Habits That Make Multi-Agent Tracing Pay Off

To wrap up the methodology, here is a short checklist that separates teams who get real value from langsmith multi-agent tracing from teams who just have pretty trees:

1. Name every agent and tool explicitly, and keep names stable — they are your primary filtering key and your eyes in the tree.
2. Attach user_id, session_id, and agent_version metadata at the entry point of every invocation, from day one.
3. Separate projects per environment so production analysis is never polluted by dev experiments.
4. Review the slowest and most expensive traces weekly, not just the failed ones — cost and latency pathologies hide in "successful" runs.
5. Turn every debugged failure into a dataset example before you ship the fix.
6. Write at least one trajectory evaluator per architectural invariant you rely on, such as "the loop always terminates" or "the writer always receives research output".
7. Use tags for experiments so A/B comparisons are one filter away.

None of these steps is difficult, but together they turn observability from a debugging tool you reach for in emergencies into a continuous improvement engine for your agent system.

Keep Going: Master LangSmith End to End

Multi-agent systems fail in ways single agents never do — silent handoff gaps, delegation loops, cost blowouts spread across a dozen model calls — and the only sane way to tame them is a trace tree that shows the whole execution as it actually happened. LangSmith gives you that tree essentially for free if you are on LangChain or LangGraph, and with a decorator and a wrapped client if you are not. From there, the workflow we covered — read the shape, follow the state, filter by agent, attribute cost, and feed every failure into datasets and evaluators — scales from your first two-agent prototype to a production system with a whole team of specialists.

If you want to go deeper — full observability setup, advanced filtering, online evaluators, annotation queues, prompt experiments, and building a complete evaluation pipeline around a real multi-agent application — check out the LangSmith Tutorial course on teachyou.ai. It walks through everything in this article hands-on, with production-grade projects you can adapt directly to your own agent stack.