DeepEval

DeepEval Conversation Testing: Evaluating Multi-Turn Chats

Ira Menon · Jun 14, 2026 · 11 min read

Why single-turn evals miss what actually breaks chatbots

Most LLM evaluation tutorials show you a single prompt, a single response, and a metric that scores it. That works fine for a summarizer or a classifier. It falls apart the moment you're testing a support bot, a sales assistant, or an agent that's supposed to remember what the user said three messages ago.

Here's the failure mode every team building a conversational product eventually hits: the bot answers turn one perfectly, turn two perfectly, and then on turn five it forgets the user already told it their account is on the Pro plan, or it drifts out of its assigned persona, or it answers a question the user never actually finished asking. None of that shows up if you're grading turns in isolation. You need to grade the *conversation*.

DeepEval, the open-source LLM evaluation framework, ships a purpose-built API for exactly this: ConversationalTestCase and Turn. Instead of one input and one output, you hand the framework an ordered list of turns — who said what, what context was retrieved, what tools were called — and a family of conversational metrics score the transcript as a whole. This article walks through the real API, shows working code for building test cases, running conversational metrics, and wiring the whole thing into pytest, and points out where teams usually get the setup wrong.

The core object: Turn and ConversationalTestCase

Everything in DeepEval's conversational testing starts with two classes from deepeval.test_case: Turn and ConversationalTestCase.

A Turn represents one message in the exchange. It has a role (either "user" or "assistant"), a content string, and optional fields for retrieval_context (what your RAG pipeline pulled in for that turn) and tools_called (what functions or APIs the assistant invoked). A ConversationalTestCase wraps a list of these turns plus conversation-level metadata like scenario, expected_outcome, and chatbot_role.

from deepeval.test_case import ConversationalTestCase, Turn

turns = [
    Turn(role="user", content="Hi, I need help resetting my password."),
    Turn(
        role="assistant",
        content="Sure, I can help with that. Can you confirm the email on your account?",
    ),
    Turn(role="user", content="It's jordan@example.com"),
    Turn(
        role="assistant",
        content=(
            "Thanks Jordan. I've sent a password reset link to "
            "jordan@example.com. It expires in 30 minutes."
        ),
        tools_called=None,
        retrieval_context=None,
    ),
]

test_case = ConversationalTestCase(
    turns=turns,
    chatbot_role="A friendly, concise customer support agent for a SaaS product.",
    scenario="A user has forgotten their password and wants to regain account access.",
    expected_outcome="The user receives a working password reset link and understands next steps.",
)

A few things worth calling out here because they trip people up:

turns cannot be empty. ConversationalTestCase validates this and raises a TypeError if you pass an empty list.
role is strictly "user" or "assistant" — there's no "system" role in a Turn. If you need to encode a system prompt or persona, that's what chatbot_role is for at the test-case level.
scenario and expected_outcome aren't decorative. Several conversational metrics (goal accuracy, conversation completeness) read these fields directly to judge whether the conversation actually resolved what the user came in for.
You can also pass dicts instead of Turn instances — ConversationalTestCase will coerce a list of plain dicts into Turn objects for you, which is handy if you're loading transcripts from a JSON log.

test_case = ConversationalTestCase(
    turns=[
        {"role": "user", "content": "Can you cancel my subscription?"},
        {"role": "assistant", "content": "I can help. Which plan are you on?"},
    ],
    scenario="User wants to cancel a subscription.",
)

Capturing retrieval context and tool calls per turn

If your chatbot is RAG-backed or agentic, individual turns carry more than just text. DeepEval lets you attach retrieval_context and tools_called to each Turn, which matters because a conversation can go off the rails at turn four even though turns one through three were fine — you want the metric to be able to point at exactly which turn failed and why.

from deepeval.test_case import Turn, ToolCall, ConversationalTestCase

turns = [
    Turn(role="user", content="What's my current data usage this month?"),
    Turn(
        role="assistant",
        content="Let me check that for you.",
        tools_called=[
            ToolCall(
                name="get_usage_data",
                input_parameters={"account_id": "acct_9182"},
                output={"usage_gb": 42.7, "cap_gb": 50},
            )
        ],
    ),
    Turn(
        role="assistant",
        content="You've used 42.7 GB out of your 50 GB cap this month.",
        retrieval_context=[
            "Billing FAQ: Usage resets on the 1st of each billing cycle.",
        ],
    ),
    Turn(role="user", content="Great, and when does that reset?"),
    Turn(
        role="assistant",
        content="Your usage resets on the 1st of your billing cycle.",
        retrieval_context=[
            "Billing FAQ: Usage resets on the 1st of each billing cycle.",
        ],
    ),
]

conversation = ConversationalTestCase(
    turns=turns,
    chatbot_role="A billing support assistant with access to account tools.",
)

ToolCall takes a name, optional input_parameters, and optional output — this is the same ToolCall class used for single-turn tool-use evaluation, reused here so multi-turn tool sequences can be scored with tool-correctness-style metrics if you need that on top of conversational ones.

Conversation-level metrics: what actually gets scored

This is where conversational testing earns its keep. DeepEval ships several metrics under deepeval.metrics that operate on the whole ConversationalTestCase rather than a single exchange:

`ConversationCompletenessMetric` — checks whether the assistant satisfied the user's intentions across the conversation, using a sliding window over recent turns (window_size, default 3) to judge whether each user request eventually got resolved.
`KnowledgeRetentionMetric` — flags turns where the assistant asks for information the user already provided earlier in the conversation, or otherwise contradicts something it should have retained.
`RoleAdherenceMetric` — checks whether the assistant stayed in character relative to the chatbot_role you defined on the test case.
`ConversationalGEval` — a conversational version of G-Eval, letting you define a custom LLM-graded rubric (e.g. "did the assistant de-escalate an angry customer") over the full turn sequence.

All of these subclass BaseConversationalMetric and share a similar constructor shape: threshold, model, include_reason, async_mode, strict_mode, and verbose_mode. Here's completeness and retention side by side:

from deepeval.metrics import (
    ConversationCompletenessMetric,
    KnowledgeRetentionMetric,
    RoleAdherenceMetric,
)
from deepeval.test_case import ConversationalTestCase, Turn

test_case = ConversationalTestCase(
    turns=[
        Turn(role="user", content="I want to upgrade to the Pro plan."),
        Turn(role="assistant", content="Sure, I can do that. What's your account email?"),
        Turn(role="user", content="dana@example.com"),
        Turn(role="assistant", content="What's your account email again?"),
        Turn(role="user", content="I just gave it to you. It's dana@example.com."),
        Turn(role="assistant", content="Got it, upgrading dana@example.com to Pro now."),
    ],
    chatbot_role="A calm, competent billing assistant.",
    scenario="User wants to upgrade their subscription plan.",
    expected_outcome="The user's plan is successfully upgraded to Pro.",
)

completeness = ConversationCompletenessMetric(threshold=0.7, window_size=3)
retention = KnowledgeRetentionMetric(threshold=0.8)
role_adherence = RoleAdherenceMetric(threshold=0.7)

for metric in (completeness, retention, role_adherence):
    metric.measure(test_case)
    print(metric.__class__.__name__, metric.score, metric.reason)

Run that against the transcript above and KnowledgeRetentionMetric should catch the obvious bug: the assistant asked for an email it was already given one turn earlier. That's the entire point of conversational testing — a single-turn metric grading "What's your account email again?" in isolation has no way to know that's a retention failure, because in isolation it's a perfectly reasonable question.

Writing a custom rubric with ConversationalGEval

Built-in metrics won't cover everything you care about. Maybe you need to check that a support bot never promises a refund it isn't authorized to promise, or that a sales assistant always attempts to book a demo before the conversation ends. ConversationalGEval lets you write that as a plain-language rubric instead of hand-rolling a metric class.

from deepeval.metrics import ConversationalGEval
from deepeval.test_case import TurnParams

de_escalation_metric = ConversationalGEval(
    name="De-escalation",
    criteria=(
        "Determine whether the assistant successfully de-escalates the "
        "user's frustration over the course of the conversation, without "
        "making promises the assistant isn't authorized to make."
    ),
    evaluation_params=[TurnParams.ROLE, TurnParams.CONTENT],
    threshold=0.6,
)

de_escalation_metric.measure(test_case)
print(de_escalation_metric.score)
print(de_escalation_metric.reason)

evaluation_params tells the metric which fields of each Turn to actually show the judge LLM — typically TurnParams.ROLE and TurnParams.CONTENT, but you can also include TurnParams.RETRIEVAL_CONTEXT or TurnParams.TOOLS_CALLED if your rubric depends on what was retrieved or invoked, not just what was said. This is the same pattern as single-turn GEval, just scoped to the conversation.

Running conversational tests with evaluate() and pytest

Once you have test cases and metrics, DeepEval's evaluate() function accepts lists of ConversationalTestCase exactly the way it accepts LLMTestCase lists — you don't need a different entry point for conversational evals.

from deepeval import evaluate
from deepeval.metrics import ConversationCompletenessMetric, RoleAdherenceMetric
from deepeval.test_case import ConversationalTestCase, Turn


def build_conversation(user_messages, chatbot):
    """Replay a scripted user script through your actual chatbot and
    collect the resulting turns."""
    turns = []
    history = []
    for message in user_messages:
        turns.append(Turn(role="user", content=message))
        history.append({"role": "user", "content": message})
        reply = chatbot.respond(history)
        turns.append(Turn(role="assistant", content=reply))
        history.append({"role": "assistant", "content": reply})
    return turns


test_cases = [
    ConversationalTestCase(
        turns=build_conversation(
            ["I want to cancel my order #4471", "Yes, please cancel it."],
            chatbot=my_chatbot,
        ),
        chatbot_role="An order-support assistant that can cancel orders.",
        scenario="User wants to cancel a recent order.",
        expected_outcome="Order #4471 is cancelled and the user is notified.",
    ),
]

evaluate(
    test_cases=test_cases,
    metrics=[
        ConversationCompletenessMetric(threshold=0.7),
        RoleAdherenceMetric(threshold=0.7),
    ],
)

For CI, the more common pattern is assert_test inside a pytest file, which raises an assertion error (failing the test) if any metric falls below its threshold:

# test_support_bot_conversations.py
import pytest
from deepeval import assert_test
from deepeval.metrics import (
    ConversationCompletenessMetric,
    KnowledgeRetentionMetric,
)
from deepeval.test_case import ConversationalTestCase, Turn


def make_case(turns, **kwargs):
    return ConversationalTestCase(turns=turns, **kwargs)


test_cases = [
    make_case(
        turns=[
            Turn(role="user", content="I need a refund for order #1029."),
            Turn(role="assistant", content="Let me check the order status for you."),
            Turn(role="assistant", content="Order #1029 qualifies for a refund. Processing it now."),
        ],
        chatbot_role="A refund-processing support agent.",
        scenario="User requests a refund for a specific order.",
        expected_outcome="The refund is processed and confirmed to the user.",
    ),
]


@pytest.mark.parametrize("test_case", test_cases)
def test_conversation_quality(test_case):
    completeness = ConversationCompletenessMetric(threshold=0.7)
    retention = KnowledgeRetentionMetric(threshold=0.8)
    assert_test(test_case, [completeness, retention])

Run it the same way you'd run any DeepEval suite:

deepeval test run test_support_bot_conversations.py

The deepeval test run wrapper around pytest gives you the usual DeepEval console output — per-metric scores, reasons, and pass/fail — but now scoped to entire conversations instead of single exchanges. It also plays nicely with pytest-xdist for parallel runs if your conversation set gets large, since DeepEval's test runner is built on top of pytest itself.

Simulating conversations instead of hand-writing them

Hand-writing every turn works for a handful of regression cases, but it doesn't scale to the hundreds of conversational paths a real chatbot needs to handle. Two practical approaches fill that gap:

Replay real logs. If you have production or staging chat logs, parse them into Turn objects (the dict-coercion behavior mentioned earlier makes this straightforward — map your log schema to {"role": ..., "content": ...} and let ConversationalTestCase do the rest) and run your conversational metrics against the historical transcripts to catch regressions before a new prompt or model version ships.
Simulate with an LLM user. Write a small harness where one LLM plays the "user" persona (grumpy customer, confused first-time user, power user asking edge-case questions) and your chatbot plays the assistant, looping for N turns, then feed the resulting transcript into a ConversationalTestCase. This is the same idea as red-teaming but aimed at conversational quality rather than safety — you're generating adversarial *conversations*, not just adversarial prompts.

Either way, the important discipline is keeping scenario and expected_outcome populated on every generated test case. Metrics like ConversationCompletenessMetric are only as good as the ground truth you give them about what "done" looks like — without an expected_outcome, the metric has to infer success criteria from the conversation alone, which is strictly weaker.

Common mistakes when testing multi-turn conversations

A few patterns show up repeatedly in teams adopting conversational testing for the first time:

Treating turns as independent and grading each with `LLMTestCase`. This throws away the entire value proposition — you'll never catch retention failures, persona drift, or unresolved threads this way. If the thing under test is a conversation, the test case should be a conversation.
Skipping `chatbot_role`. RoleAdherenceMetric and ConversationalGEval rubrics about persona both depend on this field. Leaving it blank means role adherence has nothing to check against.
Using a `window_size` that's too small or too large for `ConversationCompletenessMetric`. Too small and it won't catch a request that took five turns to resolve; too large and short exchanges get diluted. Tune it to the typical length of your real conversations, not the default blindly.
Not capturing `tools_called` per turn on agentic bots. If your assistant calls a function to look something up, that call is part of the reasoning trail. Omitting it makes it harder for a metric — or a human reading the failure reason — to tell whether a wrong answer came from bad reasoning or a bad tool result.
Forgetting `strict_mode`. By default, threshold is a soft cutoff based on a continuous score. Setting strict_mode=True collapses that into a binary pass/fail at a score of 1, which is useful for release gates where "mostly compliant" isn't good enough, but overkill for exploratory metric tuning.

Bringing it together

Multi-turn conversation testing forces you to think about your chatbot the way your users actually experience it — as a thread, not a sequence of disconnected Q&A pairs. DeepEval's Turn and ConversationalTestCase classes give you a structured way to capture that thread, including retrieval context and tool calls per turn, while metrics like ConversationCompletenessMetric, KnowledgeRetentionMetric, RoleAdherenceMetric, and ConversationalGEval let you score it against both built-in criteria and your own rubrics. Wired into evaluate() or assert_test, the same conversational test cases become part of your regular pytest suite and CI pipeline.

If you want to go deeper — building conversation simulators, tuning window sizes for completeness scoring, combining conversational and single-turn metrics in one pipeline, and setting up CI gates around chatbot releases — that's exactly what we cover hands-on in the DeepEval Tutorial course on teachyou.ai, with real chatbot projects instead of toy examples.