LangSmith Latency Analysis: Finding Your Slowest Chain Steps

Your LLM app feels slow, and nobody can tell you why. The retriever team swears their vector search is fast. The prompt engineer insists the model call is "just how long GPT takes." The frontend developer points at the backend, and the backend developer points at the model provider. Meanwhile, users are staring at a spinner for eleven seconds and quietly closing the tab. This is the everyday reality of debugging latency in chained LLM applications: total response time is easy to measure, but the breakdown of where those seconds actually went is invisible unless you instrument for it. That is exactly the problem LangSmith latency analysis solves. Every run in LangSmith carries precise start and end timestamps for every step in your chain, rendered as a waterfall you can read like a profiler. In this guide, we will walk through how to instrument a chain, read its trace, hunt down the slowest steps across thousands of production runs, and fix the bottlenecks you find.

Why Chain Latency Is So Hard to See

A single user request to a modern LLM application is rarely a single operation. A typical retrieval-augmented generation pipeline might rewrite the user's query with one LLM call, embed the rewritten query, search a vector store, rerank the results, stuff the winners into a prompt template, call the main model, and then run an output parser or a guardrail check on the answer. That is seven or eight distinct steps, each with its own latency profile, and each capable of being the villain.

Traditional application performance monitoring tools were built for microservices and database queries. They can tell you that your /chat endpoint took nine seconds, but they have no concept of a chain, a retriever, or a token stream. Wrapping your whole pipeline in a timer gives you one number with no attribution. Sprinkling time.time() calls through your code gives you attribution but no history, no aggregation, and no way to answer questions like "did the p99 get worse after Tuesday's deploy?"

The problem compounds because LLM latency is not stable. The same prompt against the same model can take three seconds at 9 a.m. and nine seconds at peak load, because provider-side queueing and token generation speed fluctuate. Retrieval latency shifts as your index grows. Some steps are fast in the median but catastrophic in the tail. Averages hide all of this. To reason about chain latency honestly, you need three things: per-step timing on every single request, the ability to see the parent-child structure of those steps, and the ability to query the whole population of runs, not just the one you happened to reproduce locally. LangSmith gives you all three essentially for free once tracing is enabled.

How LangSmith Records Timing: Runs, Traces, and the Run Tree

The core data structure in LangSmith is the run. A run is a single unit of work — one LLM call, one retriever query, one tool invocation, one chain execution — with a recorded start time, end time, inputs, outputs, and metadata. Runs nest. When a chain invokes a retriever and then an LLM, the chain is the parent run and the retriever and LLM calls are child runs. The full tree of runs triggered by one top-level invocation is a trace.

This nesting is what makes latency analysis tractable. The duration of a parent run is the wall-clock span from its start to its end, which covers everything that happened inside it: child runs, your own Python or TypeScript glue code, serialization, network hops. That leads to a subtle but important insight: the parent's duration is not simply the sum of its children's durations. If your chain run took eight seconds but its children only account for six, two seconds are hiding in your own code between steps — JSON parsing, a synchronous database write, a blocking HTTP call you forgot about. Latency analysis in LangSmith is largely the art of reading these gaps.

Each run also carries a run type — llm, chain, tool, retriever, parser, and so on — which becomes crucial later when you want to ask questions like "show me every retriever call slower than two seconds" across an entire project. And for LLM runs specifically, LangSmith records streaming-aware events, including the timestamp of the first token, which unlocks the distinction between how long the model took to start answering and how long it took to finish. We will come back to why that distinction matters more than almost anything else.

Instrumenting Your Chain for Latency Analysis

If you are using LangChain or LangGraph, instrumentation is almost embarrassingly easy: set a few environment variables and every chain, model call, retriever query, and tool invocation gets traced automatically with full timing data.

import os

os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_API_KEY"] = "your-api-key"
os.environ["LANGSMITH_PROJECT"] = "rag-prod"

from langsmith import traceable
from openai import OpenAI

client = OpenAI()

@traceable(run_type="retriever", name="vector_search")
def retrieve_docs(query: str) -> list[str]:
    # your vector store lookup here
    return search_index(query, k=8)

@traceable(run_type="llm", name="answer_generation")
def generate_answer(query: str, docs: list[str]) -> str:
    context = "\n\n".join(docs)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": f"Answer using:\n{context}"},
            {"role": "user", "content": query},
        ],
    )
    return response.choices[0].message.content

@traceable(run_type="chain", name="rag_pipeline")
def rag_pipeline(query: str) -> str:
    docs = retrieve_docs(query)
    return generate_answer(query, docs)

print(rag_pipeline("How do I rotate API keys safely?"))

The @traceable decorator is the workhorse for code that is not built on LangChain abstractions. Every decorated function becomes a run with automatic start and end timestamps, and because rag_pipeline calls the other two functions, LangSmith stitches them into a single trace with correct parent-child relationships. No timers, no manual span management, no context propagation code.

Two habits will pay off enormously here. First, decorate at the granularity you want to debug. If your "retrieval" step internally does query rewriting, embedding, search, and reranking, decorate each sub-step separately — otherwise your waterfall will show one opaque three-second block and you will be back to guessing. Second, name your runs meaningfully. Six months from now, a waterfall full of steps called RunnableSequence tells you nothing; steps called query_rewrite, pinecone_search, and cohere_rerank tell you everything at a glance.

Reading the Trace Waterfall Like a Profiler

Open any trace in the LangSmith UI and you get two complementary views. The tree view on the left shows the nesting of runs with each step's duration printed next to its name. The waterfall view lays the same runs out on a horizontal time axis, where the left edge of each bar is when the step started, the width is how long it took, and vertical alignment shows you what ran concurrently.

Reading a waterfall is a skill, and it comes down to recognizing a handful of shapes:

• One dominant bar. A single step occupies most of the total width. This is the happy case for debugging: you have one clear bottleneck, usually the final LLM generation, and your optimization energy should go there and nowhere else.
• The staircase. Many medium bars, each starting exactly where the previous one ended. This is a fully sequential pipeline. The question to ask is which steps actually depend on each other's outputs. Anything independent — say, two retrievers hitting different indexes — is a candidate for running in parallel, which turns a staircase into stacked bars and cuts wall-clock time.
• The gap. Whitespace between the end of one child bar and the start of the next, inside a parent run. Nothing traced was running, but time passed. This is your own code: a slow loop, a synchronous I/O call, an untraced network request. Gaps are the bugs nobody looks for because no monitoring tool points at them — except a waterfall does, literally, as empty space.
• The repeated bar. The same step appearing five or six times in a row, typical of an agent looping through tool calls. Each iteration might be fast, but the loop multiplies everything. The fix is usually fewer iterations, not faster ones.

Percentages matter more than absolute numbers when you triage. A two-second retriever inside a twelve-second trace is a 17 percent problem; the same retriever inside a three-second trace is a 66 percent problem. Always ask what fraction of the total each step owns before deciding where to spend a week of engineering time.

Filtering for Slow Runs Across Your Whole Project

A single trace tells you why one request was slow. Production debugging requires the opposite direction: start from the population, find the pathological runs, then drill into their traces. LangSmith's run filtering is built for exactly this. In the project view, you can filter runs by latency threshold, run type, error status, tags, metadata, and time window — and combine them. The workflow that works in practice: filter to root runs with latency above your SLA, sort descending, and open the five worst traces. Patterns jump out fast. Either the same step dominates every slow trace (a systemic bottleneck) or the slow traces look nothing alike (tail flakiness, usually provider-side or network-level).

The same power is available programmatically through the SDK, which is how you build custom latency reports:

from langsmith import Client
from collections import defaultdict
from datetime import datetime, timedelta

client = Client()
step_times = defaultdict(list)

runs = client.list_runs(
    project_name="rag-prod",
    start_time=datetime.now() - timedelta(days=1),
    is_root=False,
)

for run in runs:
    if run.end_time and run.start_time:
        seconds = (run.end_time - run.start_time).total_seconds()
        step_times[run.name].append(seconds)

for name, times in sorted(step_times.items(),
                          key=lambda kv: -sum(kv[1])):
    times.sort()
    p50 = times[len(times) // 2]
    p95 = times[int(len(times) * 0.95)]
    print(f"{name:30s} n={len(times):5d} "
          f"p50={p50:6.2f}s p95={p95:6.2f}s "
          f"total={sum(times):8.1f}s")

This little script answers the question your team has been arguing about: ranked by total time consumed across all of yesterday's traffic, which named step is actually the most expensive? Sorting by aggregate time rather than average matters, because a step that runs eight times per trace at 400 milliseconds each costs you more than a step that runs once at two seconds. Tag your runs with deployment versions or experiment names via metadata, and you can slice these same numbers by release to see precisely which deploy made things worse.

Time to First Token: The Latency Metric Users Actually Feel

Total latency is the number engineers optimize; time to first token is the number users experience. If your application streams responses — and for chat-style products it absolutely should — the moment that matters psychologically is when the first words appear, not when the last ones do. A response that starts appearing in under a second and streams for six more feels responsive. A response that shows nothing for four seconds feels broken, even if it then delivers the full answer instantly.

LangSmith records first-token timing for streaming LLM runs and surfaces it alongside total run latency, both on individual traces and in project-level monitoring charts. This split immediately clarifies which of two very different problems you have. When time to first token is high, the delay lives before generation begins: everything upstream in your chain (retrieval, reranking, query rewriting), prompt assembly, provider-side queueing, and the model's prompt processing, which scales with input length. When time to first token is low but total latency is high, the model simply has a lot of tokens to produce, and your levers are output length — tighter instructions, capped max_tokens, more concise answer formats — or a model with faster generation throughput.

The upstream case is where chain analysis earns its keep. Every millisecond spent in retrieval and reranking happens before the user sees anything at all, which means a two-second reranker does not just add two seconds to total latency — it adds two seconds of dead silence at the exact moment the user is deciding whether your product is fast. Teams routinely discover in LangSmith that their "model latency problem" is actually 60 percent pre-generation chain work, and that trimming retrieval does more for perceived speed than any model swap would.

The Usual Suspects: Where Chains Lose Their Time

After you have read enough waterfalls, the same culprits show up again and again. Knowing them in advance makes triage faster.

• Sequential LLM calls that could be one call. Query rewriting, classification, then generation as three round trips means paying prompt-processing and network overhead three times. Ask whether the model could do two of those jobs in a single structured call.
• Oversized retrieval. Fetching twenty chunks and stuffing all of them into the prompt inflates prompt-processing time at the provider. The retriever bar looks fine; the cost shows up as a slower LLM bar. LangSmith's token counts per run help you correlate context size with generation latency directly.
• Reranking with a remote service. Cross-encoder reranking APIs add a full network round trip plus inference, sitting squarely in the pre-first-token window.
• Agent loops with chatty tools. An agent that makes five tool calls where two would do multiplies every per-call overhead by the loop count. The waterfall shows this as the repeated-bar pattern, and the fix is better tool descriptions and prompts that discourage redundant calls.
• Cold starts and connection setup. The first request after idle is slow because clients, connection pools, or serverless containers are initializing. In LangSmith this appears as a tail-latency population that shares a time-of-day or after-deploy pattern rather than a common slow step.
• Untraced glue code. The gaps again: synchronous logging to a slow sink, a blocking analytics call, await missing on something that should have been concurrent. If the gap is mysterious, wrap the suspect code in @traceable and redeploy — the gap will either get a name or move.
• Provider tail latency. Sometimes the LLM bar is just long and nothing you control caused it. Model providers have variable load, and identical calls can differ several-fold. You cannot fix this, but you can mitigate it with timeouts, retries with fallback models, and streaming so users see progress regardless.

Fixing the Bottlenecks You Find

Diagnosis is only useful if it changes the code. Each waterfall shape maps to a fairly standard set of remedies.

For the staircase, parallelize independent steps. In LangChain, RunnableParallel runs branches concurrently; in plain Python, asyncio.gather on your traceable async functions does the same. Two independent 1.5-second retrievers in sequence cost three seconds; in parallel they cost 1.5, and the waterfall shows the bars stacked instead of chained — a genuinely satisfying before-and-after to screenshot for your team.

For a dominant LLM bar, work the model levers in order of cheapness. Shorten the prompt: trim retrieved context to what reranking says is relevant, compress verbose system instructions, and drop few-shot examples that no longer earn their tokens. Cap output length where the product allows it. Then consider model routing — many pipelines use a heavyweight model for every request when a smaller, faster model handles the easy majority just as well; route by query complexity and reserve the big model for the hard tail. LangSmith experiments let you verify the quality trade-off with an eval suite instead of vibes, so latency wins do not silently become accuracy losses.

For slow retrieval, the options are index tuning, fewer candidates with better reranking, caching embeddings for repeated queries, and semantic caching of full answers for genuinely repetitive traffic. A cache hit turns an entire multi-second subtree into a millisecond lookup, and in LangSmith you will see the trace shape literally change.

For gaps, the fix is ordinary software engineering: make blocking I/O async, move non-critical work (logging, analytics, persistence) out of the request path onto a background task, and reuse clients instead of constructing them per request. None of this is exotic — the hard part was seeing it, and the waterfall already did that for you.

After every change, measure the same way you diagnosed: same filters, same percentiles, before and after. Latency work without measurement discipline has a way of optimizing steps that were never the problem.

Catching Latency Regressions Before Users Do

The final stage of maturity is making latency analysis continuous instead of episodic. LangSmith's monitoring dashboards chart latency percentiles over time per project, and you can break the numbers down by metadata — model name, deployment version, customer tier — to catch regressions that only affect a slice of traffic. A p50 that looks flat while p99 doubles is a classic pattern that dashboards catch and anecdotes miss.

Three practices turn this from a wall of charts into an early-warning system. First, tag every run with your release version via metadata at trace time; when the latency chart steps upward, the breakdown tells you within minutes which deploy did it, and the trace waterfalls tell you which step inside that deploy is responsible. Second, configure alerts on latency thresholds for your production project so a regression pages someone instead of waiting for a user complaint. Third, put latency into CI: run your evaluation dataset against every candidate prompt or pipeline change, and compare not just correctness scores but the latency distribution of the experiment runs. A prompt change that improves accuracy by one point while doubling median latency is usually a bad trade, and it is far cheaper to discover that in an experiment view than in production.

It also helps to write down a latency budget: a target for total p95 and a rough allocation per step — say, 300 milliseconds for retrieval, 200 for reranking, 800 to first token, four seconds total. Budgets turn "it feels slow" into "reranking is 2x over budget," which is an actionable engineering statement rather than a mood. Every number in that sentence comes straight out of LangSmith's per-step timing data.

From Mystery Slowness to Measured Speed

Latency in LLM applications stops being mysterious the moment every step is timed and every trace is a waterfall you can read. The workflow we covered is the whole game: instrument at the right granularity with tracing and @traceable, read waterfalls for the four telltale shapes, filter production runs to find where the slow tail lives, separate time to first token from total generation time, apply the standard fixes — parallelize, trim, cache, route — and then wire dashboards, alerts, and CI checks so the next regression announces itself instead of hiding in your p99. Teams that adopt this loop stop arguing about whose step is slow, because the answer is on the screen, with timestamps.

If you want to go deeper — tracing setup across Python and TypeScript, advanced filtering, evaluations, experiments, prompt management, and production monitoring, all with hands-on projects — check out the LangSmith Tutorial course on teachyou.ai. It walks you from your first trace to a fully observable production LLM application, including the exact latency-analysis workflows from this article, so you can find your slowest chain step and fix it with confidence.