Evaluating Voice and Audio AI Agents: Unique Challenges

Why your text-based eval harness will lie to you about voice agents

If you built your evaluation pipeline for a chat agent and you're now pointing it at a voice agent, stop. The transcript looks the same on paper—user says something, agent replies—but almost everything that actually matters happened in a channel your eval never touches. A voice agent that "passes" every test when you feed it clean text transcripts can still be unusable in production, because the real user never typed clean text. They mumbled. They got interrupted by a dog barking. They said "uh, yeah, no, I meant the other one" and paused for 1.8 seconds in the middle of the sentence.

I've spent the last year building and grading voice agents for IVR replacement, tutoring bots, and customer support lines, and the single biggest mistake teams make is treating a voice agent like a text agent with a microphone bolted on. It isn't. Voice adds an entire dimension of failure modes that text-only evals are structurally blind to: transcription errors that corrupt the input before the LLM ever sees it, latency that breaks conversational turn-taking, prosody and emotion that carry meaning independent of words, and barge-in behavior that has no analogue in a text chat window.

Think about what actually happens on a support call. The caller is driving, or in a noisy kitchen, or has a regional accent your ASR model was never fine-tuned on. They pause mid-sentence to think. They say a number twice because they're not sure the system heard it right. None of that exists in a curated text eval set, and none of it is optional to handle in production — it's the majority of real traffic, not the tail. A team that ships a voice agent after only ever testing it against clean, punctuated text prompts is essentially shipping untested code, dressed up in a green dashboard.

This article walks through what actually breaks in voice agents, how to build an evaluation harness that catches it, and where LLM-as-a-Judge fits into the picture—and where it categorically does not.

The pipeline you're actually evaluating

Before you can evaluate a voice agent, you need to be honest about what "the agent" is. In almost every production system, it's a pipeline, not a model:

1. Voice Activity Detection (VAD) decides when the user has started and stopped talking
2. Automatic Speech Recognition (ASR) turns audio into text
3. The LLM (or a cascade of LLM calls) reasons over that text and decides what to say
4. Text-to-Speech (TTS) turns the response back into audio
5. Orchestration logic handles interruptions, turn-taking, and streaming playback

Some newer systems collapse steps 1-4 into a single speech-to-speech model (think real-time multimodal models that never materialize an intermediate text transcript). That architecture removes some failure modes—no ASR transcription error to propagate—but it makes evaluation harder in a different way, because you can no longer cleanly attribute a bad response to "the ASR got it wrong" versus "the model misunderstood." You have to evaluate the audio-to-audio behavior as a black box.

Either way, the mistake is evaluating only step 3. If your test suite is a JSON file of {input_text, expected_output} pairs, you are testing the LLM's reasoning in isolation, and you are missing the majority of what breaks in a live call. A rigorous voice eval has to instrument the whole pipeline and score each stage separately, then score the end-to-end conversation on top of that.

Failure mode 1: ASR errors that the LLM can't recover from

The most common and most underestimated failure in voice agents is silent corruption from speech recognition. The LLM behaves perfectly given its input—the input was just wrong.

Word Error Rate (WER) is the standard ASR metric, and it's necessary but not sufficient. A 5% WER sounds great until you realize errors aren't uniformly distributed—they cluster on exactly the tokens that matter most: proper nouns, numbers, negations.

def compute_wer(reference: str, hypothesis: str) -> float:
    """Standard word error rate via edit distance on word sequences."""
    ref_words = reference.lower().split()
    hyp_words = hypothesis.lower().split()

    # Levenshtein distance over words instead of characters
    d = [[0] * (len(hyp_words) + 1) for _ in range(len(ref_words) + 1)]
    for i in range(len(ref_words) + 1):
        d[i][0] = i
    for j in range(len(hyp_words) + 1):
        d[0][j] = j

    for i in range(1, len(ref_words) + 1):
        for j in range(1, len(hyp_words) + 1):
            if ref_words[i - 1] == hyp_words[j - 1]:
                d[i][j] = d[i - 1][j - 1]
            else:
                d[i][j] = 1 + min(
                    d[i - 1][j],      # deletion
                    d[i][j - 1],      # insertion
                    d[i - 1][j - 1],  # substitution
                )

    return d[len(ref_words)][len(hyp_words)] / max(len(ref_words), 1)

WER alone won't tell you the story you need. "I want to cancel my flight to Boston" transcribed as "I want to cancel my flight to Austin" is a one-word error—maybe 8% WER on that sentence—but it's a catastrophic, business-critical failure. Meanwhile "um I want to, uh, cancel my flight" transcribed as "I want to cancel my flight" has a much higher WER but zero semantic impact.

So the real metric you want is semantic WER on the entities that drive downstream decisions: dates, amounts, names, yes/no answers, negations. Build a targeted test set of utterances that stress exactly these categories, and grade transcription accuracy on entity extraction, not raw word overlap:

def entity_level_accuracy(reference_entities: dict, transcribed_entities: dict) -> dict:
    """
    Compare extracted entities (dates, amounts, names) between ground-truth
    and ASR-transcribed text, rather than comparing raw word sequences.
    """
    results = {}
    for key, true_val in reference_entities.items():
        transcribed_val = transcribed_entities.get(key)
        results[key] = {
            "match": str(true_val).strip().lower() == str(transcribed_val).strip().lower(),
            "expected": true_val,
            "got": transcribed_val,
        }
    return results

Run this against a corpus that deliberately includes accents, background noise, code-switching (users mixing languages mid-sentence), and telephony-quality 8kHz audio if that's your deployment channel. A model evaluated only on clean 16kHz studio recordings will look great in the eval and fall apart on a real phone call.

Failure mode 2: Latency and the collapse of turn-taking

In text chat, a five-second response delay is mildly annoying. In voice, a five-second delay is a broken product—the user assumes the call dropped and either hangs up or starts talking again, which then collides with the agent's delayed response.

Voice agents need latency broken down by stage, because "time to first audio byte" and "time to full response" tell you different things:

• ASR latency: time from end-of-speech to final transcript
• Time to first token (TTFT): time from transcript to the LLM's first output token
• Time to first audio (TTFA): time from first token to the first audible chunk of TTS output—this is the number users actually feel
• Full turnaround: end-of-user-speech to end-of-agent-speech-starting

Track these as distributions, not averages. A p50 of 800ms with a p95 of 4 seconds is a much worse product than a p50 of 1.1 seconds with a p95 of 1.4 seconds, even though the average might look similar. Users remember the bad tail, and in a phone call every slow response compounds—there's no scrollback to reread while you wait.

import time
from dataclasses import dataclass, field

@dataclass
class TurnLatencyTrace:
    user_speech_end: float
    asr_final_transcript: float | None = None
    llm_first_token: float | None = None
    tts_first_audio_chunk: float | None = None
    agent_speech_start: float | None = None

    def breakdown(self) -> dict:
        return {
            "asr_latency_ms": self._delta(self.user_speech_end, self.asr_final_transcript),
            "ttft_ms": self._delta(self.asr_final_transcript, self.llm_first_token),
            "ttfa_ms": self._delta(self.llm_first_token, self.tts_first_audio_chunk),
            "total_turnaround_ms": self._delta(self.user_speech_end, self.agent_speech_start),
        }

    @staticmethod
    def _delta(start: float | None, end: float | None) -> float | None:
        if start is None or end is None:
            return None
        return round((end - start) * 1000, 1)

Set explicit SLOs per stage (for example, TTFA under 700ms at p95) and fail the build if a change regresses any stage, not just the total. It's common to "fix" total latency by starting TTS speculatively before the LLM has finished, which is a legitimate optimization—but your eval needs to catch the failure mode this introduces: speaking a sentence and then having to backtrack or contradict yourself when the full response diverges from the speculative prefix.

Failure mode 3: Barge-in and interruption handling

Barge-in—the user interrupting the agent mid-sentence—has no equivalent in text evaluation, and it is one of the most common ways voice agents feel broken. Humans interrupt constantly in real conversation: to correct a misunderstanding, to say "wait, no," to just agree and move things along. An agent that can't handle this feels robotic even if every individual sentence it says is perfect.

Evaluate barge-in along three axes:

• Detection accuracy: does the system correctly recognize that the user started talking over the agent, versus false-triggering on background noise or the agent's own audio leaking into the mic (echo)?
• Stop latency: how long between the user starting to speak and the agent's audio actually cutting off?
• Context recovery: after an interruption, does the agent correctly incorporate what the user just said, or does it ignore the interruption and resume its previous sentence, or worse, does it get confused about what was already said versus what's now being asked?

Build test scenarios that specifically inject interruptions at controlled points—early in a sentence, mid-sentence, right at the end—and score whether the agent's next turn is coherent given the interruption:

def score_barge_in_recovery(
    agent_utterance_before_interrupt: str,
    user_interrupt_text: str,
    agent_response_after_interrupt: str,
) -> dict:
    """
    A minimal rubric-based check. In practice this final judgment
    is where LLM-as-a-Judge earns its keep — see closing section.
    """
    checks = {
        "acknowledges_interruption": None,   # did the agent drop its old sentence cleanly?
        "addresses_new_input": None,         # did it actually respond to what the user said?
        "no_repetition": None,               # did it avoid re-stating what it already said?
        "no_contradiction": None,            # is the new response consistent with prior turns?
    }
    return checks  # filled in by a judge model against a rubric, not string matching

If you don't test barge-in explicitly, you won't find this bug until real users hit it, because almost none of your scripted test conversations naturally include mid-sentence interruptions—those get authored by people typing turns into a spreadsheet, and typed test data is turn-taking by construction.

Failure mode 4: Prosody, emotion, and paralinguistic signal

Text carries none of this, and it's easy to forget it exists once your eval pipeline has flattened everything to a transcript. But a huge amount of communicative content in speech lives outside the words: tone, pace, pauses, emphasis, and emotional affect. A user saying "that's fine" in a flat, resigned tone is communicating something close to the opposite of "that's fine" said brightly. If your agent's downstream logic (or the LLM reasoning over the ASR transcript) only ever sees the word "fine," it will misread frustrated customers as satisfied ones.

This cuts two ways in evaluation:

On the input side, you need to test whether your system detects sentiment/frustration signals from prosody, not just lexical content. This usually requires either a dedicated paralinguistic classifier or a multimodal model that takes raw audio (not just the transcript) as input to the reasoning step. If your architecture throws away the audio after ASR and only ever passes text downstream, you have structurally capped how much emotional signal your agent can ever act on—that's an architecture decision worth revisiting, not just an eval gap.

On the output side, you need to evaluate whether the TTS voice's prosody matches the content. An agent apologizing for a billing error in a chipper, upbeat voice reads as tone-deaf even if the words are exactly right. This is genuinely hard to score automatically. In practice, teams either use a specialized prosody/emotion classifier on the output audio, or fall back to human evaluation on a sample, because general-purpose LLM judges operating on transcripts alone cannot hear tone at all—they only ever see what was said, never how.

A concrete, cheap technique: log the emotion/sentiment tag your TTS engine's own style parameter was set to for each turn, alongside the semantic content of what's being said (apology, bad news, confirmation, celebration), and run a straightforward mismatch check as a regression guard:

EXPECTED_TONE_BY_INTENT = {
    "apology": {"empathetic", "calm", "sincere"},
    "bad_news": {"empathetic", "calm"},
    "confirmation": {"neutral", "friendly"},
    "celebration": {"upbeat", "friendly"},
}

def check_tone_mismatch(detected_intent: str, tts_style_tag: str) -> bool:
    """Returns True if there's a likely tone mismatch worth flagging for review."""
    expected = EXPECTED_TONE_BY_INTENT.get(detected_intent, set())
    return bool(expected) and tts_style_tag not in expected

This won't catch everything, but it's a cheap tripwire that catches the worst offenders (a cheerful voice delivering bad news) before they reach a human evaluator's sample.

Failure mode 5: Multi-turn drift over long audio conversations

Voice conversations tend to run longer and looser than text chats—people ramble, circle back, restate things, and change their mind mid-thought in ways that are much rarer in typed exchanges where there's an implicit cost to typing more. This means context management failures show up more in voice: the agent loses track of what was already confirmed, re-asks a question the user already answered, or fails to update its understanding when the user corrects an earlier statement.

Evaluate this with long synthetic conversations (15-30 turns) that include natural human noise: false starts, self-corrections, mid-conversation topic changes, and callbacks to something mentioned five turns earlier. Score specifically for:

• Redundant questioning: did the agent ask for information the user already gave?
• Correction uptake: when the user says "actually, make that Tuesday, not Monday," does every subsequent turn use Tuesday?
• Stale confirmation: does the agent confirm details that were later changed?

def find_redundant_questions(agent_turns: list[str], slots_filled_by_turn: list[set]) -> list[int]:
    """
    Flags turns where the agent asks about a slot that was already filled
    in a previous turn — a common symptom of context loss in long voice calls.
    """
    already_filled = set()
    flagged_turns = []
    for i, (turn_text, slots) in enumerate(zip(agent_turns, slots_filled_by_turn)):
        for slot in slots:
            if slot in already_filled and _turn_asks_about(turn_text, slot):
                flagged_turns.append(i)
        already_filled |= slots
    return flagged_turns

def _turn_asks_about(turn_text: str, slot: str) -> bool:
    # placeholder for a real slot-question classifier
    return slot.replace("_", " ") in turn_text.lower() and "?" in turn_text

Building the composite eval: what to actually measure per call

Pull the above together into a single scorecard per test conversation rather than a pile of disconnected metrics. A useful structure:

• Transcription layer: entity-level accuracy, WER on critical spans
• Latency layer: TTFA p50/p95, full turnaround p50/p95, per-stage breakdown
• Interaction layer: barge-in detection rate, stop latency, recovery coherence
• Task layer: did the conversation actually accomplish the user's goal (booking made, issue resolved, correct information given)?
• Experience layer: tone appropriateness, redundancy, naturalness

Run this scorecard against three tiers of test data, in increasing order of realism and decreasing order of convenience:

• Synthetic clean audio: TTS-generated test utterances, cheap and fast, good for regression testing specific behaviors
• Synthetic noisy/accented audio: augmented with background noise, telephony codecs, and varied accents—catches the ASR robustness gaps that clean audio hides
• Real user recordings (with consent and privacy handling): the ground truth for whether any of this actually matters, and the only tier that will surface failure modes you didn't think to test for

Teams that skip straight to "it passed on synthetic clean audio, ship it" are the ones who get blindsided by a spike in call abandonment three weeks after launch, once real accents and real background noise start hitting a pipeline that was only ever validated in a quiet room.

It's also worth building a lightweight regression gate around this scorecard rather than a one-off report. Every time a team swaps an ASR provider, changes a TTS voice, or bumps the underlying LLM version, all five layers can shift at once — a "better" LLM might reason more verbosely, which pushes TTFA past your SLO even though the model upgrade was otherwise a clear win. Wire the scorecard into CI so a change that improves task success but blows the latency budget doesn't sail through unnoticed. Voice products live and die on the combination of correctness and responsiveness together, not either one in isolation, and a gate that only checks one of the two will let real regressions through.

Where LLM-as-a-Judge fits, and where it doesn't

Once you've collected the audio-layer metrics above—WER, latency percentiles, barge-in detection rates—you still need to judge the semantic and behavioral quality of the conversation itself: was the response appropriate, was the tone right given the transcript, did the agent recover gracefully from the interruption, was the correction properly incorporated five turns later? These are open-ended judgment calls that don't reduce to a regex or an exact-match check, and this is exactly the gap LLM-as-a-Judge is built for.

The key adaptation for voice is that your judge needs the right inputs. A judge model scoring only the final transcript will miss everything about latency and prosody, so feed it structured context alongside the transcript: the turn latencies, whether a barge-in occurred and how it was handled, the tone tag on the TTS output, and the slots filled in prior turns. Give the judge a concrete rubric ("does this response acknowledge the user's interruption without repeating already-stated information? score 1-5") rather than an open-ended "rate this conversation," and validate the judge itself against a small set of human-labeled conversations before trusting it at scale. Voice adds more surface area for the judge to reason over, but the discipline is the same one that makes LLM-as-a-Judge reliable anywhere else: clear rubrics, grounded inputs, and a human-labeled calibration set to keep the judge honest.