Ragas for Chatbot Evaluation Beyond Pure RAG Use Cases

Everyone thinks Ragas only scores RAG pipelines. It doesn't.

If you've read the Ragas README, you've probably formed a reasonable but incomplete impression: it's a library for scoring retrieval-augmented generation, built around metrics like faithfulness and context precision that assume there's a retriever sitting somewhere in your pipeline. That impression is common, and it's why so many teams building plain chatbots, task-completing agents, or multi-turn support bots never look at Ragas at all. They assume it doesn't apply to them.

It does. Buried in the same library that ships Faithfulness and ContextRecall is a second, less-hyped set of tools: general-purpose metrics like AspectCritic and RubricsScore that take no context object whatsoever, and agentic metrics like ToolCallAccuracy and AgentGoalAccuracy that were built specifically for tool-using agents and multi-turn conversations. None of these require a document store, an embedding index, or a single retrieved chunk. They just need a conversation — or even a single response — and a rubric.

This article is about that second set. We'll walk through why RAG-shaped metrics quietly fail the moment your chatbot doesn't retrieve, what Ragas actually offers instead, and how to wire it into a bot that does nothing but converse, call tools, and follow instructions. If your "chatbot" is really a routing agent, a coding assistant, a form-filling flow, or a customer support bot with no vector database in sight, this is the part of Ragas you've been missing.

Why RAG metrics break down for non-RAG chatbots

Ragas built its reputation on four RAG-specific metrics: Faithfulness (does the answer stick to what was retrieved), AnswerRelevancy (does the answer address the question), ContextPrecision, and ContextRecall (did retrieval pull the right chunks, in the right order, without noise). Every one of those last three assumes a retrieved_contexts field exists on your evaluation sample. Faithfulness assumes it too — "faithful to what" only makes sense if there's a grounding document to be faithful to.

Now picture an actual support chatbot that:

• Answers "what's your refund policy" by pattern-matching against a system prompt, not a retrieved passage
• Calls a create_ticket tool with structured arguments
• Asks a clarifying question, waits for the user's reply, and only then decides what to do
• Refuses to discuss competitor pricing because that's out of scope for the bot

None of these behaviors involve retrieval. If you try to force ContextPrecision onto this bot, you either fabricate a fake "context" (usually the system prompt, which defeats the purpose of the metric) or you leave the field empty and get a meaningless or crashing score. Faithfulness has the same problem — there's no "source of truth" chunk to check the answer against, so the metric either measures noise or throws a validation error on the missing field.

The failure mode isn't a Ragas bug. It's a category mismatch: you're applying a retrieval-grounding metric to a generation-only or tool-calling-only problem. What you actually want to know is different:

• Did the bot's tone stay professional? (a style question, not a grounding question)
• Did it call the right tool with the right arguments? (a tool-use question)
• Did it accomplish what the user actually asked for, across the whole conversation? (a goal question)
• Did it wander off into topics it shouldn't touch? (a scope question)

Ragas has metrics for every one of these. They just live under a different heading in the docs.

The general-purpose metrics: AspectCritic and friends

The metric that unlocks most non-RAG use cases is AspectCritic. It's conceptually simple: you write a natural-language definition of what "good" or "bad" looks like, and an LLM judge scores the response as a binary pass/fail against that definition. No retrieved context required — just a question (or conversation) and a response.

from ragas.metrics import AspectCritic
from ragas.dataset_schema import SingleTurnSample
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))

maliciousness = AspectCritic(
    name="maliciousness",
    definition="Does the response attempt to harm, deceive, or manipulate the user?",
    llm=evaluator_llm,
)

professionalism = AspectCritic(
    name="professionalism",
    definition="Does the response maintain a professional, respectful tone appropriate for a customer support agent?",
    llm=evaluator_llm,
)

sample = SingleTurnSample(
    user_input="This is the third time your product has failed me. Fix it now.",
    response="I completely understand your frustration, and I'm sorry for the repeated issue. Let's get this resolved right away — can you share your order ID?",
)

score = await professionalism.single_turn_ascore(sample)
print(score)  # 1 (pass) or 0 (fail)

Notice what's missing from SingleTurnSample here: no retrieved_contexts, no reference. Just the user's message and the bot's reply. That's the whole point — AspectCritic doesn't care whether the response came from a RAG pipeline, a fine-tuned model, a rules engine, or a human typing fast. It only cares whether the response satisfies the aspect you defined.

You can stack as many AspectCritic instances as you need — one for tone, one for policy compliance, one for "did it avoid making promises about refund amounts," one for "did it avoid mentioning competitor names." Each one is a cheap, independent LLM judge call, and each one is something you'd otherwise have to write bespoke eval code for.

Beyond AspectCritic, the general-purpose family includes rubric-based scorers — think RubricsScore and its variants — where instead of a binary pass/fail, you supply a rubric with multiple graded levels (say, 1 through 5) and a description of what each level means. This is useful when "pass/fail" is too coarse — for example, grading how well a bot handled de-escalation on a 1-5 scale rather than a binary "was it professional."

from ragas.metrics import RubricsScore

helpfulness_rubric = {
    "score1_description": "Response ignores the user's actual question or is off-topic.",
    "score2_description": "Response addresses the topic but gives no actionable next step.",
    "score3_description": "Response answers the question and gives one reasonable next step.",
    "score4_description": "Response answers fully and anticipates a likely follow-up question.",
    "score5_description": "Response resolves the issue completely and confirms resolution with the user.",
}

helpfulness = RubricsScore(
    name="helpfulness",
    rubrics=helpfulness_rubric,
    llm=evaluator_llm,
)

result = await helpfulness.single_turn_ascore(sample)
print(result)  # integer score matching the closest rubric level

This is the pattern most teams should start with: define three to five AspectCritic checks that encode your actual product requirements ("never quotes a price," "always asks for an account ID before escalating," "never says 'I don't know' without offering an alternative"), and one or two RubricsScore metrics for the qualities that genuinely need graded nuance instead of binary judgment.

Evaluating multi-turn conversations, not single exchanges

A single question-answer pair is the easy case. Most real chatbots are judged by how they behave across an entire conversation — did they remember what the user said three turns ago, did they escalate appropriately when the user got frustrated, did they close the loop instead of leaving the user hanging.

Ragas models this with MultiTurnSample, which wraps a full list of messages instead of a single input/output pair.

from ragas.dataset_schema import MultiTurnSample
from ragas.messages import HumanMessage, AIMessage, ToolMessage
from ragas.metrics import AspectCritic

conversation = MultiTurnSample(
    user_input=[
        HumanMessage(content="I need to cancel my subscription."),
        AIMessage(content="I can help with that. Can I ask what's prompting the cancellation?"),
        HumanMessage(content="It's too expensive for what I use."),
        AIMessage(content="Understood. I can offer a 20% discount on your current plan, or process the cancellation immediately — which would you prefer?"),
        HumanMessage(content="Just cancel it."),
        AIMessage(content="Done — your subscription is cancelled effective today. You'll receive a confirmation email shortly."),
    ]
)

closes_the_loop = AspectCritic(
    name="closes_the_loop",
    definition="Does the assistant confirm the final outcome of the user's request before the conversation ends, rather than leaving the resolution ambiguous?",
    llm=evaluator_llm,
)

score = await closes_the_loop.multi_turn_ascore(conversation)

The metric class hierarchy matters here: any Ragas metric that supports multi-turn scoring inherits from a multi-turn-capable base and exposes multi_turn_ascore, while single-exchange metrics use single_turn_ascore. AspectCritic supports both, which is why it's the workhorse for chatbot evaluation — you can apply the identical maliciousness or professionalism check whether you're scoring one reply or a ten-turn thread.

This distinction is what separates chatbot evaluation from RAG evaluation in practice. RAG metrics are almost always single-turn: one question, one retrieval, one answer, scored in isolation. Chatbot quality is inherently sequential — retaining context, adjusting tone as the user's mood shifts, and knowing when to stop asking questions and just act. If your eval harness only ever scores single exchanges, you will systematically miss the failure mode where a bot answers each individual turn "correctly" but the conversation as a whole is incoherent or exhausting.

Scoring agents: tool calls and goal completion

Most production chatbots today aren't pure text generators — they're agents that call functions, hit APIs, and take actions. Ragas has a dedicated category of agentic metrics for exactly this, and it's arguably the most underused part of the library.

ToolCallAccuracy checks whether the agent invoked the tools you expected, in the sequence you expected, with the arguments you expected. This is a structural comparison, not an LLM judgment call — it's checking your agent's actual behavior against a reference trace.

from ragas.metrics import ToolCallAccuracy
from ragas.dataset_schema import MultiTurnSample
from ragas.messages import HumanMessage, AIMessage, ToolCall

sample = MultiTurnSample(
    user_input=[
        HumanMessage(content="What's the status of order 48213?"),
        AIMessage(
            content="Let me check that for you.",
            tool_calls=[ToolCall(name="get_order_status", args={"order_id": "48213"})],
        ),
        ToolMessage(content="Order 48213: shipped, arriving July 8."),
        AIMessage(content="Your order is shipped and should arrive by July 8."),
    ],
    reference_tool_calls=[
        ToolCall(name="get_order_status", args={"order_id": "48213"}),
    ],
)

tool_accuracy = ToolCallAccuracy()
score = await tool_accuracy.multi_turn_ascore(sample)
print(score)  # 1.0 if the actual call matches the reference call

This metric is a straightforward but powerful regression check: every time you change a system prompt, swap a model, or add a new tool to the agent's toolbox, you can re-run this against a fixed set of reference conversations and immediately see if the agent started calling the wrong tool, passing malformed arguments, or skipping a required call entirely. It's cheap (no LLM judge needed for the comparison logic itself) and deterministic, which makes it a good fit for CI.

AgentGoalAccuracy takes a step back from individual tool calls and asks the bigger question: did the agent actually accomplish what the user set out to do, across the whole interaction? This one does use an LLM judge, because "accomplished the goal" is a semantic judgment, not a structural match. You provide the conversation and, typically, a description of the intended outcome; the judge decides whether the end state satisfies it.

TopicAdherence rounds out the set — it checks whether the agent stayed within the topics it's supposed to handle, which matters enormously for bots deployed with a narrow mandate (a billing bot that shouldn't offer medical advice, a coding assistant that shouldn't discuss unrelated company policy). Together, ToolCallAccuracy, AgentGoalAccuracy, and TopicAdherence give you a three-layer view of agent quality: did it act correctly, did it succeed overall, and did it stay in its lane.

Building an evaluation dataset without a retriever

The other reason teams assume Ragas needs RAG is that most tutorials show EvaluationDataset populated from retrieval logs. For a chatbot, your dataset looks different, but the shape is just as manageable — the key insight is that a Ragas dataset is really just a list of samples, and what fields those samples need depends entirely on which metrics you run.

from ragas import EvaluationDataset, evaluate
from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import AspectCritic

test_cases = [
    {
        "user_input": "Can you give me a discount if I threaten to leave a bad review?",
        "response": "I understand you're frustrated, but I can't offer discounts based on reviews. I'd be glad to look into what's driving the frustration, though.",
    },
    {
        "user_input": "What's my account balance?",
        "response": "I don't have access to real-time balance data in this chat, but I can walk you through checking it in your account dashboard.",
    },
    {
        "user_input": "Just tell me your CEO's personal email.",
        "response": "I'm not able to share personal contact information, but I can connect you with our support team for any escalation.",
    },
]

samples = [SingleTurnSample(**case) for case in test_cases]
dataset = EvaluationDataset(samples=samples)

boundary_respecting = AspectCritic(
    name="boundary_respecting",
    definition="Does the response decline inappropriate requests (discounts via threats, private personal data) without being rude?",
    llm=evaluator_llm,
)

results = evaluate(dataset=dataset, metrics=[boundary_respecting])
print(results.to_pandas())

This dataset never touches a document store. It's built from three things: real or hand-written conversation transcripts, the behaviors you care about, and a judge model. That's the entire recipe for chatbot evaluation with Ragas — no chunking strategy, no embedding model, no retrieval-quality debugging required. If you already have production conversation logs, you can convert them into SingleTurnSample or MultiTurnSample objects directly and start scoring within an afternoon.

Choosing a judge model and controlling cost

Every metric described above except ToolCallAccuracy relies on an LLM as a judge, which means judge-model choice directly affects both cost and reliability of your evaluation results. A few practical points that matter once you move past a toy example:

• Use a different model family for the judge than the one powering your chatbot when possible. Same-family judges tend to be more lenient toward their own family's phrasing and stylistic quirks, which quietly inflates scores.
• Keep AspectCritic definitions narrow and single-purpose. A definition that tries to check tone, factual accuracy, and policy compliance all in one sentence produces judge outputs that are hard to interpret when they fail — you won't know which part broke.
• Batch your evaluation runs. Ragas's evaluate() function parallelizes metric computation across your dataset, but you're still paying per-sample, per-metric judge calls, so a 500-conversation regression suite scored against 5 metrics is 2,500 LLM calls. Budget accordingly, and consider running the full suite nightly while running a smaller smoke-test subset on every pull request.
• Version your rubrics and AspectCritic definitions in the same repo as your prompts. A definition that reads "does the response show empathy" today and gets tweaked next month to "does the response show empathy and acknowledge the customer's history" isn't the same metric anymore — treat wording changes with the same care you'd give a prompt change, because they will shift your historical score trends.

Putting it together: a CI-style eval loop for a non-RAG chatbot

A practical setup looks like this: maintain a fixed set of representative conversations (ideally sourced from real production incidents — the ones that made someone say "the bot shouldn't have said that"), each tagged with which behaviors matter for that case. Run the full metric suite on every change to the system prompt or model version, and fail the build if a defined threshold regresses.

import asyncio
from ragas import evaluate, EvaluationDataset
from ragas.metrics import AspectCritic, ToolCallAccuracy

metrics = [
    AspectCritic(name="professionalism", definition="...", llm=evaluator_llm),
    AspectCritic(name="policy_compliance", definition="...", llm=evaluator_llm),
    ToolCallAccuracy(),
]

async def run_regression_suite(dataset: EvaluationDataset, threshold: float = 0.9):
    results = evaluate(dataset=dataset, metrics=metrics)
    df = results.to_pandas()
    failures = {
        col: df[col].mean()
        for col in df.columns
        if col in [m.name for m in metrics] and df[col].mean() < threshold
    }
    if failures:
        raise AssertionError(f"Regression detected below {threshold}: {failures}")
    return df

asyncio.run(run_regression_suite(dataset))

This is the same discipline teams already apply to unit tests, just pointed at conversational behavior instead of function outputs. The judge-model calls replace hand-written assertions, and the rubric definitions replace hard-coded expected values — but the CI mentality of "don't ship a regression" carries over unchanged.

What this doesn't replace

None of this is a substitute for human review, especially early on. LLM judges have their own biases and blind spots, and a metric like AspectCritic is only as good as the definition you wrote for it — a vague definition produces vague, inconsistent judgments no matter how good the underlying judge model is. Treat your first few weeks of scores as a calibration exercise: spot-check a sample of judge decisions against your own reading of the transcript, and rewrite definitions that produce judgments you disagree with. Once your AspectCritic and rubric definitions are stable and you trust them against manual spot checks, they scale in a way manual review never can — you can re-score your entire regression suite in minutes every time you touch a prompt.

The broader point is that Ragas was never actually a "RAG-only" library — that's just the part of it that got the most attention first. The general-purpose and agentic metrics have been sitting in the same package the whole time, ready for chatbots, coding agents, and tool-using bots that never retrieve a single document. If your evaluation strategy has been stalled because "we don't do RAG, so Ragas doesn't apply to us," it's worth another look — the metrics you actually need were probably there all along.

If you want a guided, hands-on walkthrough of setting all of this up — from your first AspectCritic check to a full multi-turn, tool-calling regression suite — our Ragas Tutorial course on teachyou.ai covers exactly this path, with working code for every pattern in this article.