LangSmith for Voice Agent Debugging

Voice agents feel like magic until they break in production, and then they feel like a crime scene with no witnesses. A user says something, the agent mishears it, a downstream tool gets the wrong argument, and the whole conversation derails. By the time your support team forwards you the complaint, all you have is a vague transcript and a customer who is never coming back. Text chatbots are hard enough to debug. Voice agents add speech-to-text errors, streaming latency, interruptions, tool calls, and text-to-speech artifacts on top of the usual large language model unpredictability. When something goes wrong across that pipeline, guessing is not a strategy. This is exactly the gap LangSmith fills. It gives you a recorded, searchable, replayable view of every step your voice agent takes, so you can stop guessing and start reading the actual trace. In this article we will walk through how to instrument a voice agent with LangSmith, what to look for in a trace, how to build evaluation datasets from real failures, and how to close the loop so your agent gets measurably better over time.

Why Voice Agents Are Harder to Debug Than Chatbots

A text chatbot has a fairly clean input and output. The user types a message, the model responds, and if something is wrong you can read exactly what the user sent. A voice agent has none of that clarity. The pipeline usually looks like this: audio comes in, a speech-to-text model transcribes it, the transcription flows into your agent logic, the agent may call one or more tools, the model generates a response, and a text-to-speech model turns that response back into audio. Every one of those stages can introduce an error, and those errors compound.

Consider a simple failure. A customer says "cancel my order," but the speech-to-text layer hears "council my order." Your agent has no matching intent, so it hallucinates a helpful sounding but completely wrong response. If you only look at the final audio output, you will conclude the model is bad at reasoning. If you look at the trace, you will see the model reasoned perfectly over garbage input. The bug is in transcription, not generation. Without step-level visibility, you will waste days fixing the wrong layer.

The other hard part is that voice is real time. Latency that would be invisible in a chat window becomes an awkward silence on a phone call. A tool that takes three seconds to respond is fine in text and unacceptable in voice. You need to know not just what happened but how long each step took, and you need that broken down per span rather than as a single end-to-end number. LangSmith records both the content and the timing of every step, which is why it works so well for this class of problem.

What LangSmith Actually Records

At its core LangSmith is a tracing and evaluation platform for large language model applications. When you instrument your code, it captures a tree of runs. The top level run represents the whole request. Nested inside are child runs for each meaningful operation: a speech-to-text call, a retrieval step, an LLM generation, a tool invocation, a text-to-speech call. Each run stores its inputs, its outputs, its start and end time, any errors, token counts where relevant, and any metadata or tags you attach.

The key mental model is the trace as a tree. In a voice agent, one turn of the conversation becomes one trace, and every sub-operation becomes a nested span you can expand and inspect. When you open a trace in the LangSmith UI, you see the full hierarchy. You can click into the speech-to-text span and see exactly what text it produced. You can click into the LLM span and see the exact prompt that was sent, including the system prompt, the conversation history, and the tool definitions. You can see which tool the model chose and what arguments it passed. This is the single most valuable thing LangSmith gives a voice agent developer: the ability to see the exact prompt and the exact transcription that caused a bad turn.

Metadata is what makes traces searchable at scale. You can tag every trace with a session identifier, a user identifier, the environment, the model name, and any business context you care about. Later, when a specific customer complains, you filter by their session identifier and pull up every turn of that exact conversation in seconds.

Setting Up LangSmith in a Voice Agent

Getting started is deliberately low friction. LangSmith can auto-instrument a lot of common frameworks through environment variables alone. You set your API key and project name, enable tracing, and any LangChain or LangGraph calls are captured automatically. For the parts of a voice pipeline that are not LangChain, such as your speech-to-text and text-to-speech calls, you wrap them with a decorator so they show up as spans in the same trace.

Here is a minimal Python setup. The environment variables turn tracing on globally, and the traceable decorator marks any function you want to appear in the trace tree.

import os
from langsmith import traceable
from langsmith.wrappers import wrap_openai
from openai import OpenAI

os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_API_KEY"] = "your-api-key"
os.environ["LANGSMITH_PROJECT"] = "voice-agent-prod"

# Wrapping the client auto-traces every chat and audio call
client = wrap_openai(OpenAI())


@traceable(run_type="tool", name="speech_to_text")
def transcribe(audio_bytes: bytes) -> str:
    result = client.audio.transcriptions.create(
        model="whisper-1",
        file=audio_bytes,
    )
    return result.text


@traceable(run_type="tool", name="text_to_speech")
def synthesize(text: str) -> bytes:
    response = client.audio.speech.create(
        model="tts-1",
        voice="alloy",
        input=text,
    )
    return response.content

The important detail is run_type. Marking a function as a tool, an LLM call, a retriever, or a chain changes how LangSmith renders it and what aggregate metrics it computes. Speech-to-text and text-to-speech are best modeled as tools because they are deterministic external calls with clear inputs and outputs.

Structuring the Trace for One Conversation Turn

A voice agent is a loop. Each turn should map cleanly to one trace so you can reason about it in isolation. The cleanest pattern is to wrap the entire turn handler in a single traceable function and let the nested calls attach themselves underneath it. That parent span becomes the anchor you attach session metadata to.

from langsmith import traceable


@traceable(run_type="chain", name="handle_turn")
def handle_turn(audio_bytes: bytes, session_id: str, history: list) -> bytes:
    # Step 1: transcription
    user_text = transcribe(audio_bytes)

    # Step 2: agent reasoning and tool use
    reply_text = run_agent(user_text, history)

    # Step 3: speech synthesis
    reply_audio = synthesize(reply_text)

    return reply_audio


# Attach metadata so the whole turn is filterable later
result = handle_turn(
    audio_bytes,
    session_id="call_8842",
    history=history,
    langsmith_extra={
        "metadata": {
            "session_id": "call_8842",
            "channel": "phone",
            "region": "in-south",
        },
        "tags": ["prod", "inbound"],
    },
)

Notice the langsmith_extra argument. It lets you pass metadata and tags without polluting your function signature. Because transcribe, run_agent, and synthesize are all decorated, they automatically nest inside handle_turn when called within it. The result is a single expandable trace per turn: transcription at the top, agent reasoning and any tool calls in the middle, synthesis at the bottom. When you later group traces by session_id, you can replay an entire phone call turn by turn in order.

Reading a Trace to Find the Real Bug

Instrumentation is worthless if you do not know what to look at. When you open a failed turn, work through the pipeline in the same order the data flowed and check each handoff.

Start at transcription. Read the text the speech-to-text span produced and compare it against what you know the user actually said. A surprising share of voice agent bugs die right here. Homophones, proper nouns, product names, numbers, and accented speech are the usual suspects. If the transcription is wrong, nothing downstream can be right, and you should be tuning your speech-to-text configuration, adding a domain vocabulary, or adding a normalization step rather than touching your prompt.

If the transcription is clean, move to the LLM span. LangSmith shows you the exact assembled prompt. This is where you catch context bugs. Common findings include the following.

• The system prompt was truncated or an older version was deployed.
• Conversation history was passed in the wrong order or with the wrong role labels.
• A tool definition was malformed, so the model never had the option it needed.
• Retrieved context was empty because a retrieval step silently failed.

Next check tool calls. If the model chose a tool, look at the arguments it generated and the value the tool returned. A frequent voice-specific failure is that the model extracts an argument from a mis-transcribed value. The model behaved correctly given its input, but the input was already corrupted upstream. Seeing the tool arguments next to the transcription makes this obvious in a way that reading logs never does.

Finally check timing. Every span shows its duration. If one turn felt slow on the call, expand the spans and find the one that ate the time. Often it is a single slow tool or an oversized prompt causing a long generation. LangSmith turns a vague "the agent felt laggy" complaint into a precise "the inventory lookup took 4.2 seconds" finding.

Building Evaluation Datasets From Real Failures

Finding one bug is good. Making sure it never comes back is better. This is where LangSmith moves from a debugger to an improvement engine. Every trace you inspect is a candidate example for a dataset. When you find a turn that went wrong, you can add it to a dataset directly, capturing the input and the corrected expected output. Over time you accumulate a collection of the exact hard cases your agent has faced in the wild.

A dataset in LangSmith is simply a set of examples, where each example has inputs and optionally a reference output. For a voice agent you typically build datasets at the text layer, using the transcription as input, so that your evaluations are fast and deterministic and do not require re-running audio. You can create examples programmatically as you triage failures.

from langsmith import Client

client = Client()

dataset = client.create_dataset(
    dataset_name="voice-agent-hard-turns",
    description="Real failed turns captured from production traces",
)

client.create_examples(
    dataset_id=dataset.id,
    examples=[
        {
            "inputs": {"user_text": "cancel my order 4471"},
            "outputs": {"intent": "cancel_order", "order_id": "4471"},
        },
        {
            "inputs": {"user_text": "actually change it to next tuesday"},
            "outputs": {"intent": "reschedule", "date": "next_tuesday"},
        },
    ],
)

The strategic point is that these are not made up test cases. They are the real utterances that broke your agent, promoted into a permanent regression suite. Every time a new failure class appears in production, you add an example, and your dataset becomes a living record of everything your agent must handle.

Running Evaluations and Scoring Agent Behavior

Once you have a dataset, you can run your agent against every example and score the results automatically. LangSmith calls the thing that produces a score an evaluator. An evaluator receives the run output and the reference output and returns a score, which can be a boolean, a number, or a category. You can write simple deterministic evaluators in plain code, and you can also use an LLM as a judge for fuzzy criteria like helpfulness or tone.

For a voice agent, the most useful evaluators tend to be concrete. Did the agent extract the correct intent. Did it pull the correct order number. Did it avoid inventing a policy that does not exist. Here is a straightforward correctness evaluator paired with an evaluation run.

from langsmith import evaluate


def intent_correct(run, example) -> dict:
    predicted = run.outputs.get("intent")
    expected = example.outputs.get("intent")
    return {
        "key": "intent_match",
        "score": int(predicted == expected),
    }


def order_id_correct(run, example) -> dict:
    return {
        "key": "order_id_match",
        "score": int(
            run.outputs.get("order_id") == example.outputs.get("order_id")
        ),
    }


def run_target(inputs: dict) -> dict:
    # This calls your real agent logic on the text input
    return run_agent_structured(inputs["user_text"])


results = evaluate(
    run_target,
    data="voice-agent-hard-turns",
    evaluators=[intent_correct, order_id_correct],
    experiment_prefix="intent-v3",
)

When this runs, LangSmith executes your agent on every example, applies each evaluator, and stores the results as an experiment. You get an aggregate score per evaluator and a per-example breakdown so you can see exactly which cases still fail. Because each run is prefixed, you can compare intent-v3 against intent-v2 side by side and see whether your prompt change actually helped or just moved the failures around. This is the difference between engineering and vibes. You change one thing, you re-run the suite, and the numbers tell you if you improved.

Comparing Prompt and Model Versions Over Time

The real payoff of all this instrumentation is confident iteration. Voice agents are tuned constantly. You tweak the system prompt to reduce a bad behavior, you swap in a new model to cut latency, you adjust a tool description to fix argument extraction. Every one of those changes can silently break something that used to work. Regression is the default outcome of tinkering with prompts, and without a scoreboard you will not notice until a customer does.

LangSmith experiments give you that scoreboard. Because every evaluation run is stored, you can line up experiments and see how each metric moved. If your latency went down but intent accuracy dropped two points, you can see that trade explicitly and decide whether it is worth it. If a new model version fixed three failure classes but introduced a new one, the per-example view shows you precisely which cases regressed so you can add a guard for them.

A healthy workflow looks like this in practice.

1. Ship a change to your voice agent.
2. Watch production traces for new failure patterns.
3. Promote any new failures into your dataset as examples.
4. Run the full evaluation suite before your next deploy.
5. Compare against the previous experiment and only ship if the numbers hold.

Over weeks this loop compounds. Your dataset grows to cover the long tail of weird real world utterances. Your evaluators encode exactly what correct behavior means for your product. Your experiments become an audit trail of every change and its measured effect. The agent stops being a fragile artifact you are afraid to touch and becomes a system you can improve with confidence.

Monitoring Voice Agents in Production

Debugging and evaluation cover development, but voice agents also need watching once they are live. LangSmith supports online monitoring, where production traces flow in continuously and you attach automatic evaluators or filters to flag problems as they happen. You can set up rules that surface any turn where a tool errored, where latency crossed a threshold, or where an LLM-as-judge evaluator scored the response as unhelpful.

The practical value is a triage queue built from reality. Instead of waiting for customer complaints, you review the turns your monitors flagged, confirm whether they are genuine failures, and feed the real ones straight into your dataset. Session metadata makes this fast. When you filter to a single problematic call, you replay it turn by turn, watch exactly where it went off the rails, and capture that turn as a new test case. The loop from production incident to permanent regression test can be measured in minutes rather than sprints.

A few habits make production monitoring pay off. Tag traces generously with session identifiers, channel, region, and model version so filtering is precise. Keep an eye on per-span latency, not just totals, because voice quality lives or dies on responsiveness. And treat every flagged failure as free training data. The utterances that break your agent in production are the most valuable examples you will ever get, because they are exactly the inputs your users actually produce.

Bringing It All Together

Debugging a voice agent without tracing is like fixing a car engine blindfolded. The pipeline is long, the failures compound across speech-to-text, reasoning, tool calls, and text-to-speech, and the final audio tells you almost nothing about which stage broke. LangSmith removes the blindfold. It records every step of every turn as a searchable, replayable tree, so you can read the exact transcription and the exact prompt that produced a bad result. It lets you promote real failures into datasets, score agent behavior with concrete evaluators, and compare versions so you know whether a change actually helped. And it keeps watching once you are in production, turning live incidents into permanent regression tests.

The mindset shift is the important part. Stop guessing at what your voice agent did and start reading what it actually did. Instrument the whole pipeline, structure each turn as one trace, inspect failures at the handoffs, and close the loop with datasets and evaluations. Do that consistently and your agent gets measurably better every week instead of mysteriously worse.

If you want to go deeper and build this workflow end to end, the LangSmith Tutorial course on teachyou.ai walks you through tracing, datasets, evaluators, LLM-as-judge scoring, and production monitoring with hands on projects, including voice and tool calling agents like the ones in this article. It is the fastest way to turn these ideas into a repeatable practice you can bring to your own agents.