Ragas

Ragas for Voice-Based RAG Assistants

Ira Menon · Jun 3, 2026 · 15 min read

Why voice changes everything about RAG evaluation

Text-based RAG evaluation is a solved-enough problem. You have a clean query string, a clean set of retrieved chunks, and a clean generated answer. Ragas was built for exactly that world, and it does a great job scoring faithfulness, context precision, and answer relevancy when everything upstream is text from the start.

Voice-based RAG assistants break that assumption in the first step. Before your retriever ever sees a query, the user's speech has already passed through a speech-to-text (STT) model that may have mangled a product name, dropped a negation, or merged two sentences into one run-on question. The "query" your RAG pipeline retrieves against isn't what the user said — it's what the STT model thinks the user said. Then, after your LLM generates an answer, it goes through a text-to-speech (TTS) engine that flattens formatting, strips out markdown, and sometimes truncates long responses to fit conversational pacing.

This means a voice RAG assistant has two extra failure surfaces that a text chatbot doesn't: transcription drift on the way in, and speech synthesis drift on the way out. If you evaluate only the middle layer — retrieval and generation — with vanilla Ragas metrics, you'll get scores that look great in your test harness and still ship an assistant that mishears "Ragas" as "regas" and gives users a wrong answer with total confidence.

This article walks through how to actually adapt Ragas for voice pipelines: what metrics still apply unchanged, what needs new instrumentation, and how to build an evaluation harness that catches voice-specific failures before your users do.

This gap matters more than it might seem at first glance. Voice interfaces are being deployed for customer support lines, in-car assistants, healthcare intake bots, and internal enterprise tools where employees ask questions hands-free. In every one of these contexts, the cost of a wrong answer is higher than in a text chat window, because there's no scrollback for the user to double check, no way to hover over a citation, and often no screen at all. If your evaluation strategy is "run Ragas on the text and call it done," you are evaluating a pipeline that doesn't match what actually ships.

The anatomy of a voice RAG pipeline

Before touching metrics, it helps to lay out the pipeline stages explicitly, because each stage needs its own evaluation strategy.

Audio capture — raw user audio, often streamed in chunks
STT transcription — audio converted to text query (Whisper, Deepgram, AssemblyAI, etc.)
Query rewriting — optional step to clean up disfluencies ("um", "so yeah", repeated words) before retrieval
Retrieval — the RAG step: embedding the query, searching the vector store, reranking
Generation — the LLM produces an answer grounded in retrieved context
Response shaping — trimming the answer for spoken delivery (shorter sentences, no bullet lists, no markdown)
TTS synthesis — text converted back to audio

Ragas, out of the box, is built to evaluate the retrieval and generation stages. It has no native concept of an STT confidence score or a TTS naturalness score. Your job is to wrap Ragas so it slots into stages 4 and 5, while building complementary checks around stages 2, 3, and 6.

A useful mental model: Ragas evaluates whether the answer is correct given the transcript. You need separate checks for whether the transcript is correct given the audio, and whether the spoken output is faithful to the generated text.

Setting up a Ragas evaluation dataset for voice transcripts

The first practical step is building an evaluation dataset that reflects real voice input, not clean text you typed into a test file. If you evaluate your RAG pipeline against hand-written queries like "What is the refund policy for annual plans?" you will never surface the failure mode where a user says "what's the refund policy on the yearly plan" and the STT renders it as "what's the refund policy on the very plan."

A good voice RAG eval set has three parallel fields per example:

The reference transcript (what was actually said, verified by a human or a high-accuracy model)
The STT output (what your pipeline's transcription model actually produced)
The ground truth answer (what the correct response should contain)

Here's how you structure that as a Ragas-compatible dataset:

from datasets import Dataset

voice_eval_samples = [
    {
        "user_input": "what's the refund policy on the yearly plan",  # STT output, noisy
        "reference_input": "what's the refund policy on the yearly plan",  # human verified
        "retrieved_contexts": [
            "Annual subscriptions are refundable within 14 days of purchase...",
            "Monthly subscriptions can be cancelled anytime with no refund for the current cycle...",
        ],
        "response": "Annual plans are refundable within 14 days of purchase, prorated based on unused months.",
        "reference": "Annual subscriptions get a 14-day refund window from purchase date.",
    },
    # ... more samples spanning noisy STT, accents, background noise, interruptions
]

dataset = Dataset.from_list(voice_eval_samples)

Notice the extra reference_input field. This isn't a standard Ragas column, but you'll use it to compute a transcription-drift score before anything even touches retrieval. Keep it in your dataset so every downstream metric can be sliced by "clean STT" versus "noisy STT" samples.

Measuring transcription drift before you measure retrieval quality

If you only look at Ragas scores on the final answer, a bad STT transcription and a bad retrieval decision look identical: the answer is wrong. You need to separate these two failure causes, or you'll spend a week tuning your reranker when the actual bug is that Whisper keeps transcribing "Ragas" as "Regus."

A simple, effective approach is to compute a semantic similarity score between the STT output and the verified reference transcript, independent of Ragas. You can reuse Ragas's underlying embedding infrastructure to do this consistently with the rest of your pipeline:

from ragas.embeddings import embedding_factory
import numpy as np

embeddings = embedding_factory()

def transcription_drift_score(stt_output: str, reference_input: str) -> float:
    vec_a = embeddings.embed_query(stt_output)
    vec_b = embeddings.embed_query(reference_input)
    cosine_sim = np.dot(vec_a, vec_b) / (
        np.linalg.norm(vec_a) * np.linalg.norm(vec_b)
    )
    return float(cosine_sim)

for sample in voice_eval_samples:
    drift = transcription_drift_score(sample["user_input"], sample["reference_input"])
    sample["transcription_drift"] = drift

Once you have this score attached to every sample, you can segment your Ragas results by drift bucket. A pattern I see constantly in voice RAG projects: context precision looks fine on average, but drops sharply for any sample with transcription_drift < 0.85. That's your signal that the retriever is doing its job correctly — it's being handed a corrupted query and doing the best it can with it. The fix belongs in STT tuning or query rewriting, not in your embedding model or chunking strategy.

Adapting core Ragas metrics for spoken queries

With drift measured separately, you can now run the standard Ragas metrics on the retrieval and generation stages, but with a few adjustments to account for how voice queries differ from typed ones.

Context Precision and Context Recall still work well, but you should expect naturally lower baseline scores on voice datasets even with a perfect STT model, because spoken queries are less precise than typed ones. People speak in fragments, use pronouns without clear antecedents ("what about that one"), and rely on conversational context that a single-turn retrieval can't see. Don't chase the same context recall numbers you'd expect from a text-only support bot — set your target thresholds based on a voice-specific baseline, not your existing text benchmarks.

Faithfulness becomes more important in voice pipelines, not less, because there's no way for a user to "re-read" a spoken answer to catch an inconsistency. If the assistant says something ungrounded, the user has already moved on before they could catch it. I'd treat faithfness as a hard gate metric for voice RAG rather than a soft quality signal.

Answer Relevancy needs recalibration too. Ragas's answer relevancy metric works by generating synthetic questions from the answer and comparing them to the original query. If your original query field is the noisy STT output, you're effectively testing "is the answer relevant to the mis-transcription" rather than "is the answer relevant to what the user meant." Where possible, compute answer relevancy against the reference_input (verified transcript) instead of the raw STT output, so your metric reflects user intent rather than pipeline noise.

Here's a full evaluation call putting these pieces together:

from ragas import evaluate
from ragas.metrics import (
    Faithfulness,
    ContextPrecision,
    ContextRecall,
    AnswerRelevancy,
)
from ragas.llms import llm_factory

llm = llm_factory()

# swap in the verified transcript for relevancy scoring
dataset = dataset.map(
    lambda row: {"user_input": row["reference_input"]},
    desc="using verified transcript for relevancy scoring",
)

results = evaluate(
    dataset=dataset,
    metrics=[
        Faithfulness(llm=llm),
        ContextPrecision(llm=llm),
        ContextRecall(llm=llm),
        AnswerRelevancy(llm=llm),
    ],
)

print(results.to_pandas()[
    ["faithfulness", "context_precision", "context_recall", "answer_relevancy"]
].describe())

Run this twice — once with user_input set to the raw STT output, once with it set to the verified reference — and diff the two result sets. The delta between them is a direct, quantified measure of how much your STT layer is costing you in downstream RAG quality.

Evaluating the response-shaping step before TTS

Most teams don't think of "response shaping" as a step worth evaluating, but it's where voice assistants quietly break faithfulness. Your LLM might generate a perfectly grounded, faithful answer with a bulleted list and inline citations. Then a post-processing step strips markdown, collapses the list into a single run-on sentence for TTS, and in doing so drops a caveat that was in its own bullet point ("except for enterprise plans, which follow a different refund window").

This is a silent faithfulness regression that never shows up if you only run Ragas against the pre-shaping generation output. You need a second faithfulness pass against the *shaped* text that will actually be spoken.

from ragas.metrics import Faithfulness
from ragas.dataset_schema import SingleTurnSample
import asyncio

def shape_for_speech(answer: str) -> str:
    # illustrative simplification of a real shaping step
    shaped = answer.replace("\n- ", ". ").replace("**", "")
    return shaped

async def check_shaped_faithfulness(sample, llm):
    shaped_response = shape_for_speech(sample["response"])
    shaped_sample = SingleTurnSample(
        user_input=sample["user_input"],
        response=shaped_response,
        retrieved_contexts=sample["retrieved_contexts"],
    )
    scorer = Faithfulness(llm=llm)
    return await scorer.single_turn_ascore(shaped_sample)

# compare pre-shaping and post-shaping faithfulness for the same sample

If the shaped-text faithfulness score is meaningfully lower than the pre-shaping score, your response-shaping logic is the bug — not your retriever, not your generation prompt. This is a distinction that's easy to miss because both stages produce "the answer," and teams often only evaluate whichever version is easiest to grab from logs.

Handling multi-turn voice conversations

Voice assistants are rarely single-shot. A real session looks like: "What's your return policy?" → "Does that apply to the annual plan too?" → "What about if I bought it as a gift?" Each turn depends on context from the previous ones, and a lot of that context lives in pronouns and ellipsis that only make sense with conversation history.

Ragas supports multi-turn evaluation through conversational samples, and this is where voice RAG evaluation diverges most sharply from single-turn text RAG. You need to evaluate not just whether turn 3's answer is faithful to its own retrieved context, but whether the query rewriting step correctly resolved "does that apply" into something retrievable in the first place.

from ragas.dataset_schema import MultiTurnSample
from ragas.messages import HumanMessage, AIMessage

conversation = MultiTurnSample(
    user_input=[
        HumanMessage(content="what's your return policy"),
        AIMessage(content="Our standard return window is 30 days from delivery."),
        HumanMessage(content="does that apply to the annual plan too"),
        AIMessage(content="For annual subscription plans, refunds follow a 14-day window instead of the standard 30-day policy."),
    ],
    reference="Annual plans have a 14-day refund window, different from the 30-day standard return policy.",
)

# multi-turn metrics evaluate whether context carried correctly across turns

When building your voice eval set, deliberately include multi-turn sequences with referring expressions ("that", "it", "the other one") because these are exactly where voice assistants fail differently from chat assistants — users interrupt themselves, restate mid-sentence, and expect the system to track intent across audio segments that a text-based chat UI would never produce.

Building a CI-friendly regression suite

Once you've assembled these pieces — drift scoring, dual faithfulness passes, multi-turn samples — the next step is wiring this into something that runs automatically whenever you touch the STT model, the retriever, the prompt, or the shaping logic. Treat it the same way you'd treat a test suite for application code.

Freeze a fixture set of 50-150 real (anonymized) voice sessions covering common intents, edge-case accents, and known problem phrases specific to your domain vocabulary
Run the drift score, dual faithfulness pass, and standard Ragas metrics on every fixture as part of your pipeline's CI
Set hard thresholds per metric (for example, faithfulness must not regress below a locked baseline) and fail the build if a change pushes scores below that line
Store historical scores per pipeline version so you can see whether a "small" prompt tweak quietly degraded context recall three releases ago

import json
from pathlib import Path

def check_regression(results, baseline_path="baseline_scores.json", tolerance=0.03):
    baseline = json.loads(Path(baseline_path).read_text())
    current = results.to_pandas().mean(numeric_only=True).to_dict()

    failures = []
    for metric, baseline_score in baseline.items():
        current_score = current.get(metric, 0)
        if current_score < baseline_score - tolerance:
            failures.append(
                f"{metric}: {current_score:.3f} vs baseline {baseline_score:.3f}"
            )

    if failures:
        raise AssertionError("Regression detected:\n" + "\n".join(failures))
    print("No regressions detected.")

Running this suite on every pull request that touches the pipeline turns Ragas from a one-off notebook exercise into an actual quality gate, which is the only way voice RAG systems stay reliable as the domain vocabulary and the underlying models keep changing underneath you.

It's worth being deliberate about what "regression" means for a voice pipeline versus a text one. A text RAG system might tolerate a temporary dip in context recall if the tradeoff is faster retrieval latency. A voice system usually can't make that tradeoff the same way, because latency and quality both affect the user's experience directly and simultaneously — a slow, correct answer and a fast, wrong answer are both failures, just different kinds. When you set thresholds for your CI gate, pair every quality metric with a latency budget for that same stage, and fail the build if either one is violated. This keeps teams from "fixing" a faithfulness regression by adding a slower verification pass that quietly breaks the assistant's responsiveness instead.

It's also worth versioning your fixture set alongside your code. Voice assistants tend to accumulate new intents and new domain vocabulary over time — new product names, new policies, new slang your users adopt. If your 150-sample fixture set is frozen from six months ago, it will miss regressions in whatever your users are asking about today. Schedule a recurring review, monthly or quarterly depending on how fast your product changes, where you pull a fresh sample of real (anonymized) sessions into the fixture set and re-baseline.

Common pitfalls teams hit when evaluating voice RAG

A few patterns show up repeatedly when teams first try to bolt Ragas onto a voice pipeline:

Evaluating against typed test queries instead of real transcripts. This gives you clean numbers that don't predict production behavior at all. Always source your eval set from actual STT output, warts included.
Conflating STT errors with retrieval errors. Without a separate drift score, every low Ragas score looks like a RAG problem, and teams end up re-tuning chunking strategies that were never broken.
Evaluating the pre-shaping answer only. The text your LLM generates and the text your TTS engine speaks are not always the same string. Evaluate both.
Ignoring latency as a quality dimension. Ragas doesn't measure latency, but in a voice context, a factually perfect answer that takes eight seconds to start speaking will feel broken to the user. Pair your Ragas scores with a separate latency budget check — time-to-first-audio-byte matters as much as faithfulness for the perceived quality of a voice assistant.
Skipping multi-turn cases. Single-turn eval sets systematically miss the referring-expression failures that are unique to how people actually talk to voice assistants.

None of these pitfalls require exotic tooling to fix — they mostly require being deliberate about which stage of the pipeline you're actually measuring at each step, instead of treating "the final answer" as the only thing worth scoring.

Putting it all together

A mature evaluation setup for a voice-based RAG assistant ends up looking like four layers stacked on top of each other: a transcription drift score that isolates STT quality, standard Ragas retrieval metrics (context precision and recall) computed against the verified transcript, a dual faithfulness check across both the raw generation and the speech-shaped output, and multi-turn conversational samples that exercise reference resolution across turns. Wire all four into a CI gate with locked baselines, and you have a system that catches regressions before they reach a live caller instead of after a support ticket comes in.

None of this requires abandoning Ragas or building a custom eval framework from scratch — it requires treating voice as an additional set of pipeline stages that need their own instrumentation, layered around the retrieval and generation evaluation that Ragas already does well. The teams that get burned are the ones who assume "we already have Ragas set up for our chatbot" means the same harness transfers cleanly to voice. It transfers partially. The gaps are exactly the stages unique to audio, and that's where most real-world voice RAG failures actually live.

If you want to go deeper on building this kind of evaluation harness step by step — including hands-on labs on drift scoring, multi-turn sample construction, and wiring Ragas into a CI pipeline — check out the Ragas Tutorial course on teachyou.ai, where we build a full voice RAG evaluation suite from an empty repo to a working regression gate.