DeepEval

DeepEval for Chatbots: End-to-End Conversation Quality Checks

Pramod Dutta · Jun 13, 2026 · 13 min read

Why chatbot testing breaks normal QA habits

You ship a chatbot, it passes your smoke tests, and two weeks later a user posts a screenshot where the bot contradicts itself between turn three and turn seven. Nobody wrote a test for that because nobody could have predicted that exact exchange. This is the core problem with conversational AI: the surface area isn't a fixed set of screens with fixed buttons, it's an open-ended sequence of turns where each response depends on everything that came before it.

Traditional software testing assumes determinism. Click button, expect result. Chatbots break that assumption twice over — once because the underlying model is non-deterministic even at low temperature, and again because a "correct" response in isolation can still be wrong in context. A bot that gives a technically accurate answer to turn five but ignores a constraint the user stated in turn two has failed, even though no single message looks broken.

DeepEval was built for exactly this gap. It's an open-source LLM evaluation framework that treats conversations as first-class test subjects, not just isolated prompt-response pairs. Instead of asking "is this one output good," it asks "did this entire conversation hold together, stay grounded, and satisfy the user's actual goal." That reframing is what makes it useful for chatbots specifically, as opposed to single-shot generation testing.

In this article we'll walk through how to actually set up conversational evaluation with DeepEval — from single-turn metrics you probably already know, up through multi-turn conversational metrics, custom conversation-level assertions, and wiring it all into a CI pipeline so regressions get caught before your users find them.

The difference between single-turn and conversational evaluation

Most teams starting out with LLM evaluation begin with single-turn metrics: given a prompt and a response, is the response relevant, faithful to the retrieved context, and free of hallucination. DeepEval's AnswerRelevancyMetric, FaithfulnessMetric, and similar single-turn metrics handle this well, and if your chatbot were a Q&A endpoint that resets after every message, that would be enough.

But real chatbots carry state. A support bot that helps a user reset a password needs to remember which account they mentioned three turns ago. A tutoring bot needs to track what the student already got wrong so it doesn't repeat the same explanation. A sales assistant needs to not contradict a price quote it gave earlier in the same session. None of these failure modes show up if you evaluate each turn independently — you need to evaluate the conversation as a unit.

DeepEval models this with a ConversationalTestCase, which wraps an ordered list of LLMTestCase turns and lets you run metrics that look across the whole thing. Here's the shape of it:

from deepeval.test_case import LLMTestCase, ConversationalTestCase

turn_1 = LLMTestCase(
    input="Hi, I'm locked out of my account, email is priya@example.com",
    actual_output="Thanks Priya, I can help. Can you confirm the last 4 digits of your registered phone number?"
)

turn_2 = LLMTestCase(
    input="It's 4821",
    actual_output="Got it. I've sent a reset link to priya@example.com. Check your inbox."
)

turn_3 = LLMTestCase(
    input="What was my email again, I forgot what I typed",
    actual_output="You gave me priya@example.com — that's where the reset link went."
)

convo_test_case = ConversationalTestCase(turns=[turn_1, turn_2, turn_3])

Notice turn 3 is deliberately testing memory — did the bot actually retain the email from turn 1, or does it hallucinate a different one because it lost track of context. This is the class of bug that single-turn testing structurally cannot catch, and it's the most common category of real-world chatbot complaint.

Setting up DeepEval for a conversational project

Installation is the same as any DeepEval project. Start with a virtual environment and the package itself.

python -m venv .venv
source .venv/bin/activate
pip install deepeval
deepeval login

The deepeval login step connects your local runs to Confident AI (DeepEval's companion dashboard) if you want hosted result tracking and regression history across runs. It's optional for local-only testing, but for a chatbot with dozens of conversation fixtures, you'll want the dashboard once your suite grows past a handful of files.

Next, set your model provider. DeepEval defaults to using an LLM as the judge for most metrics (this is the "LLM-as-a-judge" pattern), so you need an evaluation model configured separately from whatever model powers your chatbot.

export OPENAI_API_KEY="sk-..."
deepeval set-local-model  # optional, if using a local judge model via Ollama etc.

Project layout matters here more than in single-turn testing, because conversation fixtures get long and you want them reusable across multiple metrics. A reasonable structure:

chatbot-eval/
  conversations/
    password_reset_happy_path.py
    password_reset_wrong_otp.py
    tutoring_multi_topic_drift.py
  metrics/
    conversation_metrics.py
  test_chatbot_conversations.py

Keeping conversation fixtures in their own files makes it easy to add new ones as support tickets reveal new failure patterns — every time production surfaces a bad multi-turn exchange, it becomes a permanent regression fixture.

Core conversational metrics in DeepEval

DeepEval ships several metrics designed specifically for ConversationalTestCase objects. The two most load-bearing ones for chatbot QA are ConversationalGEval and ConversationCompletenessMetric, alongside KnowledgeRetentionMetric and RoleAdherenceMetric for more specialized checks.

ConversationalGEval lets you define a custom rubric in plain language and have an LLM judge score the whole conversation against it. This is the workhorse for chatbot QA because most of what you care about — tone, consistency, staying in character, not contradicting earlier turns — doesn't map cleanly to a single canned metric.

from deepeval.metrics import ConversationalGEval
from deepeval.test_case import LLMTestCaseParams

consistency_metric = ConversationalGEval(
    name="ConversationConsistency",
    criteria=(
        "Determine whether the assistant's responses remain consistent "
        "with facts, promises, or constraints established earlier in the "
        "conversation. Penalize any contradiction, forgotten detail, or "
        "restated information that conflicts with a prior turn."
    ),
    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
    threshold=0.7,
)

consistency_metric.measure(convo_test_case)
print(consistency_metric.score, consistency_metric.reason)

KnowledgeRetentionMetric checks specifically whether the bot forgets information the user already provided, which is the single most common complaint users file against support bots ("I already told you my order number!"). RoleAdherenceMetric checks whether the bot stays within its defined persona or system prompt boundaries across the whole session — useful for bots that are supposed to refuse off-topic requests but sometimes drift into answering them anyway after enough turns.

from deepeval.metrics import KnowledgeRetentionMetric, RoleAdherenceMetric

retention_metric = KnowledgeRetentionMetric(threshold=0.8)
role_metric = RoleAdherenceMetric(
    chatbot_role="a password-reset support agent who only handles account access issues",
    threshold=0.8,
)

retention_metric.measure(convo_test_case)
role_metric.measure(convo_test_case)

Run all three together against the same fixture and you get a much fuller picture than any single metric alone — one might catch the forgotten email, another might catch the bot suddenly offering unrelated billing advice it wasn't scoped to give.

Writing conversation fixtures that actually catch bugs

The value of this whole approach lives or dies on fixture quality. A fixture that only tests the happy path will pass every time and tell you nothing. The fixtures worth writing are the ones that stress the exact places conversations tend to break:

Topic switches mid-conversation — user asks about billing, then abruptly pivots to a technical issue, then comes back to billing. Does the bot lose the billing context?
Corrections — user gives wrong info, then corrects themselves two turns later ("actually my order number is 5521, not 5512"). Does the bot pick up the correction or keep using the stale value?
Long-context recall — a ten-plus turn conversation where a detail from turn one needs to resurface in turn ten.
Adversarial persona drift — repeated attempts to get the bot to act outside its defined role, checking whether it holds the line across escalating pressure.
Ambiguous follow-ups — "same as before" or "the other one" style references that require the bot to resolve pronouns against conversation history.

Here's a fixture built around the correction case, which in practice is one of the highest-frequency real bugs:

from deepeval.test_case import LLMTestCase, ConversationalTestCase

def build_correction_fixture():
    turns = [
        LLMTestCase(
            input="My order number is 5512, it hasn't arrived",
            actual_output="I'm sorry to hear that. Let me look into order 5512."
        ),
        LLMTestCase(
            input="Sorry, typo — it's actually 5521",
            actual_output="No problem, updating to order 5521. Checking status now."
        ),
        LLMTestCase(
            input="Any update?",
            actual_output="Order 5521 shows as shipped and arriving in 2 days."
        ),
    ]
    return ConversationalTestCase(turns=turns)

If your production bot regresses and starts answering turn three with "order 5512 shows as shipped," this fixture catches it immediately, whereas single-turn testing of turn three alone would have no way to know 5512 was ever wrong.

Wiring conversational tests into pytest

DeepEval integrates directly with pytest, which means these conversation fixtures slot into whatever test runner you already use. The pattern is to parametrize over your fixture functions and assert against your chosen metrics.

import pytest
from deepeval import assert_test
from deepeval.metrics import ConversationalGEval, KnowledgeRetentionMetric
from deepeval.test_case import LLMTestCaseParams
from conversations.password_reset_wrong_otp import build_correction_fixture

consistency_metric = ConversationalGEval(
    name="ConversationConsistency",
    criteria="Check the assistant never uses stale or corrected information after a user correction.",
    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
    threshold=0.7,
)

retention_metric = KnowledgeRetentionMetric(threshold=0.8)

@pytest.mark.parametrize("build_fixture", [build_correction_fixture])
def test_conversation_quality(build_fixture):
    convo = build_fixture()
    assert_test(convo, [consistency_metric, retention_metric])

Run it the same way you'd run any pytest suite:

deepeval test run test_chatbot_conversations.py

The deepeval test run wrapper (instead of plain pytest) gives you DeepEval's richer console output — per-metric scores, reasons for failure, and a summary table — which is far more useful for debugging a failing conversation than a bare pytest assertion error.

Simulating real conversations instead of hand-writing every fixture

Hand-writing fixtures works for known failure patterns, but you also want coverage for conversation shapes you haven't thought of yet. DeepEval's conversation simulator addresses this by generating synthetic multi-turn conversations against your actual chatbot, driven by a description of a user persona and goal.

from deepeval.conversation_simulator import ConversationSimulator

simulator = ConversationSimulator(
    user_intentions={
        "reset a forgotten password": 5,
        "dispute a duplicate charge": 5,
        "escalate to a human agent after being unsatisfied": 3,
    },
    user_profile_items=[
        "first_name", "last_name", "account_email", "typical_frustration_level"
    ],
)

def chatbot_callback(user_input: str, conversation_history: list) -> str:
    # replace with your actual chatbot invocation
    return my_chatbot.respond(user_input, conversation_history)

conversational_test_cases = simulator.simulate(
    model_callback=chatbot_callback,
    stopping_criteria="The user's goal has been resolved or explicitly escalated.",
)

This generates a batch of realistic multi-turn conversations across the intents you specify, each ending naturally per your stopping criteria rather than at a fixed turn count. Running your metric suite against these simulated conversations surfaces edge cases that a small hand-written fixture set will always miss — particularly around users who are impatient, contradict themselves, or ask several things at once.

Treat simulated conversations as a discovery tool, not a replacement for your curated regression fixtures. When a simulated run fails a metric in an interesting way, promote that specific transcript into your permanent fixture file so it's guaranteed to be checked on every future run.

Catching hallucination and grounding failures across turns

For chatbots backed by retrieval (RAG-based support bots, internal knowledge assistants), grounding failures compound across turns in ways that are easy to miss. A bot might correctly cite its source in turn one, then in turn four — while discussing something adjacent — start blending in details that were never in any retrieved document.

You can layer DeepEval's FaithfulnessMetric per-turn inside a conversation while still evaluating the conversation holistically, by running it against each LLMTestCase that carries retrieval_context:

from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

faithfulness_metric = FaithfulnessMetric(threshold=0.8)

turn_with_context = LLMTestCase(
    input="What's your refund policy for digital courses?",
    actual_output="Digital courses are refundable within 14 days of purchase if less than 20% has been completed.",
    retrieval_context=[
        "Refund policy: Digital course purchases are eligible for a full refund "
        "within 14 days of purchase, provided the learner has completed less "
        "than 20% of the course content."
    ],
)

faithfulness_metric.measure(turn_with_context)
assert faithfulness_metric.score >= 0.8, faithfulness_metric.reason

Run this per-turn check across every turn in a conversation that involves retrieval, in addition to the conversation-level metrics. A bot can pass conversational consistency (it isn't contradicting itself) while still failing faithfulness (it's consistently wrong, or consistently inventing a detail that sounds plausible but isn't in any source document). You want both checks, because they catch different failure classes.

Automating this in CI so regressions can't ship silently

None of this is useful if it only runs manually before a demo. The real payoff is catching a regression the moment a prompt template, a retrieval config, or a model version change breaks a conversation flow that used to work. A minimal GitHub Actions setup:

name: chatbot-conversation-eval

on:
  pull_request:
    paths:
      - "prompts/**"
      - "chatbot/**"
      - "chatbot-eval/**"

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - name: Install dependencies
        run: pip install deepeval
      - name: Run conversational eval suite
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: deepeval test run chatbot-eval/test_chatbot_conversations.py

Scope the trigger to paths that actually affect chatbot behavior — prompt templates, retrieval config, model version pins — so you're not burning judge-model API calls on every unrelated commit. For teams running this daily, it's also worth setting a threshold budget: not every metric needs to gate the merge. Treat consistency and knowledge retention as hard blockers, and treat softer stylistic metrics as warnings that get logged to the dashboard without failing the build. This keeps CI fast and keeps the signal-to-noise ratio high enough that engineers actually trust the red X when it shows up.

Common pitfalls when testing chatbots this way

A few things trip up teams the first time they build a conversational eval suite:

Judge model drift — if you change the LLM used as your evaluation judge, your historical scores are no longer directly comparable. Pin your judge model version and treat changing it as a breaking change to your test suite, not a routine update.
Fixtures that are too short — a three-turn fixture rarely stresses memory retention. Real regressions tend to show up around turn six to ten, once enough context has accumulated that the model has more to lose track of.
Only testing happy paths — the fixtures that matter most are the ones modeling corrections, interruptions, and topic switches, not the clean linear conversations that always looked fine anyway.
Ignoring cost — conversational metrics that use an LLM judge cost real money per run, and a full conversation costs more to judge than a single turn. Budget for this before your suite scales to hundreds of fixtures, and consider caching or a cheaper judge model for local development runs versus a stronger judge for CI.
Treating the simulator output as ground truth — synthetic conversations are a discovery tool for edge cases, not validated golden data. Always review a simulated transcript before promoting it into your permanent regression suite.

Closing thoughts

Chatbots fail in ways that single-turn testing is structurally blind to — forgotten details, stale corrections, persona drift, and grounding that degrades over the course of a session. DeepEval's ConversationalTestCase and its family of conversational metrics give you a concrete, code-first way to catch these failures before they reach production, and its pytest integration means this slots into a workflow your team already runs on every pull request.

Start small: pick your three most common real support tickets, turn each into a conversation fixture, and wire in ConversationalGEval plus KnowledgeRetentionMetric. Expand from there using the conversation simulator to surface the edge cases you haven't thought of yet. If you want a structured, hands-on path through all of this — building fixtures, writing custom rubrics, and setting up the CI pipeline from scratch — check out the DeepEval Tutorial course on teachyou.ai, where we build a full conversational evaluation suite for a real chatbot end to end.