LLM Evaluationmulti-turn evaluationconversational AILLM-as-judgeagent testing

Evaluating Multi-Turn Conversations

Pramod Dutta · Jun 29, 2026 · 15 min read

Multi-turn evaluation measures whether an LLM behaves correctly across a whole conversation, not just on one isolated prompt. It scores things a single-turn test cannot see: does the assistant remember what the user said four messages ago, does it stay on task when the user changes direction, does it recover after it misunderstands. If you ship a chatbot, agent, or support copilot and only test single prompts, you are grading the wrong thing.

This article is for engineers who already have an LLM app in production or near it. It covers why single-turn scores mislead, what dimensions to score in a conversation, how to build a multi-turn evaluation harness with an LLM judge, how to simulate users so you can generate conversations at scale, and the failure modes that only show up over many turns. Everything here comes with runnable code and real commands you can adapt.

Why single-turn evaluation misses the real failures

Most eval tutorials score a model on a fixed dataset of (input, expected_output) pairs. That works for classification, extraction, or one-shot generation. It falls apart for anything conversational, because the interesting failures are relational: they depend on turns that came before.

Consider a support bot. Turn 1 the user says "I'm on the Pro plan." Turn 5 they ask "can I add three more seats?" A single-turn test of turn 5 has no way to know the answer depends on the Pro plan context established earlier. The model might give a correct-sounding answer that is wrong for this user. A single-turn harness scores it as fine.

Here is the concrete gap. Single-turn evaluation checks a function f(prompt) -> response. A real conversation is a stateful process where each response is conditioned on the entire history. Multi-turn evaluation checks the trajectory, not the point.

The failures that only appear across turns:

Context loss: the model forgets a constraint the user gave earlier and contradicts it.
Instruction drift: a system rule ("never quote a price") holds for three turns then leaks on turn six.
Bad recovery: the user corrects the model ("no, I meant the annual plan") and the model doubles down instead of updating.
Repetition and loops: the agent re-asks for information the user already provided.
Goal abandonment: over a long chat the assistant slowly stops pursuing what the user actually wanted.

None of these are visible in a (prompt, response) pair. This is why multi-turn evaluation is a different discipline, not a bigger dataset.

What to actually measure in a multi-turn evaluation

Before you write any harness, decide what "good" means for a conversation. Vague goals ("be helpful") produce vague scores. Break the conversation into concrete, scoreable dimensions. These are the ones that hold up across most conversational apps.

Task completion: did the conversation reach the user's actual goal by the end? This is the headline metric. Score the whole transcript, not a turn.
Relevancy per turn: is each assistant message a sensible response to the latest user message given the history?
Knowledge retention: does the assistant correctly use facts the user stated in earlier turns?
Role and policy adherence: does the assistant stay inside its system prompt rules across every turn, including the last one?
Faithfulness: when the assistant makes claims, are they grounded in the provided context or retrieved documents, or did it hallucinate?
Conversation completeness: were all of the user's sub-requests addressed, or did some get dropped?
Recovery: after a misunderstanding or user correction, did the assistant get back on track?

You will not score all seven for every app. A RAG support bot cares most about faithfulness, retention, and task completion. A tutoring bot cares about relevancy and recovery. Pick three or four that map to how your users actually get hurt, and score those consistently.

One more distinction that matters: turn-level metrics versus conversation-level metrics. Relevancy and faithfulness are naturally per-turn: you can score message by message and then aggregate. Task completion and conversation completeness are conversation-level: they only make sense given the full transcript. Your harness needs to handle both.

Building a multi-turn evaluation harness from scratch

Let me show the core loop in plain Python before naming any framework, so you understand what the tools do under the hood. The pattern has three parts: a data model for a conversation, a way to produce assistant responses, and a judge that scores the transcript.

First, a minimal conversation type. A conversation is an ordered list of turns, each with a role and content.

from dataclasses import dataclass, field

@dataclass
class Turn:
    role: str          # "user" or "assistant"
    content: str

@dataclass
class Conversation:
    turns: list = field(default_factory=list)
    context: str = ""  # retrieved docs or scenario setup, optional

    def as_messages(self):
        return [{"role": t.role, "content": t.content} for t in self.turns]

Next, run your app to produce a full conversation from a fixed script of user turns. This is the simplest form: you hardcode what the user says, and let your model fill in the assistant turns one at a time, always passing the growing history.

import anthropic

client = anthropic.Anthropic()

def run_scripted_conversation(system_prompt, user_turns, model="claude-sonnet-4-5"):
    convo = Conversation()
    for user_msg in user_turns:
        convo.turns.append(Turn("user", user_msg))
        resp = client.messages.create(
            model=model,
            max_tokens=1024,
            system=system_prompt,
            messages=convo.as_messages(),
        )
        assistant_text = resp.content[0].text
        convo.turns.append(Turn("assistant", assistant_text))
    return convo

The key detail: every call passes the entire convo.as_messages() history. That is what makes it multi-turn. If you accidentally pass only the latest user message, you have rebuilt a single-turn test and lost everything this article is about.

Now the judge. An LLM-as-judge reads the full transcript and scores one dimension. Keep each judge focused on a single dimension with an explicit rubric. A judge that scores five things at once is unreliable.

JUDGE_PROMPT = """You are evaluating a conversation between a user and an AI assistant.

Score ONE dimension: KNOWLEDGE RETENTION.
Definition: The assistant correctly uses facts the user stated in earlier
turns and never contradicts them.

Return a JSON object: {{"score": 1-5, "reason": "...", "evidence_turn": N}}
- 5: uses all earlier facts correctly, no contradictions.
- 3: minor slip, uses most facts but misses one.
- 1: contradicts or ignores facts the user clearly stated.

Conversation transcript:
{transcript}
"""

import json

def judge_retention(convo, model="claude-sonnet-4-5"):
    transcript = "\n".join(f"{t.role.upper()}: {t.content}" for t in convo.turns)
    resp = client.messages.create(
        model=model,
        max_tokens=512,
        messages=[{"role": "user",
                   "content": JUDGE_PROMPT.format(transcript=transcript)}],
    )
    raw = resp.content[0].text
    start, end = raw.find("{"), raw.rfind("}") + 1
    return json.loads(raw[start:end])

Wire these together and you have a working harness: define user scripts, run them through your app, score each transcript on the dimensions you picked, aggregate. That is the whole shape of multi-turn evaluation. Everything a framework adds is convenience on top of this loop.

A few things to get right in the judge that beginners miss:

Ask for structured JSON with a score and a reason. The reason forces the judge to justify itself and gives you something to read when a score looks wrong.
Make the judge cite an evidence_turn. A score you cannot trace to a specific turn is not actionable.
Use a strong model as the judge, ideally a different or larger model than the one under test. A model grading its own output is more forgiving of its own mistakes.
Pin the judge model and prompt version. If the judge changes, your scores are no longer comparable across runs.

Simulating users so you can scale multi-turn evaluation

Scripted conversations are fine for a handful of golden test cases, but they do not scale and they do not surprise you. Real users are messy: they change their mind, they are terse, they get frustrated, they ask off-topic questions. To evaluate robustly you need many conversations that look like that. The answer is a simulated user: a second LLM that plays the human.

The simulated user has a goal (a scenario) and a persona, and it generates the next user turn given the conversation so far. You run it against your assistant in a loop until the goal is met or a turn limit hits.

USER_SIM_PROMPT = """You are role-playing a user talking to a support assistant.

Your goal: {goal}
Your persona: {persona}

Continue the conversation with ONE natural user message. Stay in character.
If your goal is fully met, reply with exactly: [DONE]

Conversation so far:
{transcript}
"""

def simulate_user_turn(goal, persona, convo, model="claude-sonnet-4-5"):
    transcript = "\n".join(f"{t.role.upper()}: {t.content}" for t in convo.turns) or "(start)"
    resp = client.messages.create(
        model=model,
        max_tokens=256,
        messages=[{"role": "user", "content": USER_SIM_PROMPT.format(
            goal=goal, persona=persona, transcript=transcript)}],
    )
    return resp.content[0].text.strip()

Now the full simulation loop: the simulated user and your assistant take turns until the user signals done or you hit a cap.

def run_simulated_conversation(system_prompt, goal, persona,
                               max_turns=8, model="claude-sonnet-4-5"):
    convo = Conversation()
    for _ in range(max_turns):
        user_msg = simulate_user_turn(goal, persona, convo, model)
        if "[DONE]" in user_msg:
            break
        convo.turns.append(Turn("user", user_msg))
        resp = client.messages.create(
            model=model, max_tokens=1024,
            system=system_prompt, messages=convo.as_messages(),
        )
        convo.turns.append(Turn("assistant", resp.content[0].text))
    return convo

With this you can generate hundreds of conversations by varying goals and personas. Build a small matrix: five goals crossed with four personas ("terse", "confused", "changes mind halfway", "adversarial") gives you twenty distinct multi-turn scenarios, each scored on your chosen dimensions. That is a real multi-turn evaluation suite, not a demo.

Personas worth including because they expose specific failures:

The mind-changer: states one thing, then reverses it, to test retention and recovery.
The under-specifier: gives too little info, to test whether the assistant asks instead of guessing.
The topic-jumper: switches subjects mid-chat, to test focus and whether the assistant loses the original thread.
The adversary: tries to get the assistant to break a policy, to test rule adherence across turns.

Two cautions with simulated users. First, the simulator can drift out of character or solve the task for the assistant. Keep its prompt tight and give it a clear [DONE] exit. Second, a simulated user is not a replacement for real transcripts; it is a way to get coverage cheaply. Always keep a set of real, human conversations as your ground-truth check that the simulator is realistic.

Using an evaluation framework instead of hand-rolling

The loops above are the mechanism. In practice you will want a framework so you are not maintaining judge plumbing yourself. Several open-source options handle multi-turn evaluation directly. DeepEval has conversational metrics that take a list of turns and score things like turn relevancy, knowledge retention, role adherence, and conversation completeness, plus a conversation simulator that plays the user for you. Ragas focuses on RAG and has metrics suited to grounded, retrieval-backed chats. LangSmith and Langfuse handle tracing and let you attach evaluators to real production conversations, which matters when you want to score live traffic and not just a test set.

The shape is the same across all of them: you assemble a conversation object from ordered turns, you pick metrics, you run, you read aggregated scores with per-turn reasons. Here is the idea in DeepEval-style pseudocode so you can map it back to the hand-rolled version.

from deepeval.test_case import ConversationalTestCase, Turn
from deepeval.metrics import ConversationalGEval

test_case = ConversationalTestCase(turns=[
    Turn(role="user", content="I'm on the Pro plan."),
    Turn(role="assistant", content="Got it, you're on Pro."),
    Turn(role="user", content="Can I add three more seats?"),
    Turn(role="assistant", content="Yes, Pro supports adding seats..."),
])

retention = ConversationalGEval(
    name="Knowledge Retention",
    criteria="The assistant never contradicts facts the user stated earlier.",
)
retention.measure(test_case)
print(retention.score, retention.reason)

Notice this is exactly the judge from earlier, wrapped. ConversationalGEval is an LLM-as-judge with a criteria string instead of a hand-written prompt. Understanding the hand-rolled version is what lets you debug the framework when a score looks off, tune the criteria, and know which metric is turn-level versus conversation-level.

Whichever tool you use, run it from CI. A typical command with DeepEval looks like deepeval test run test_conversations.py, where the test file builds your conversational test cases and asserts scores clear a threshold. Gate merges on the aggregate so a regression in multi-turn behavior blocks the deploy.

Wiring multi-turn evaluation into CI

An eval you run by hand once a month catches nothing. The value shows up when every change to your prompt, model, or retrieval runs the suite automatically. Structure it like a test suite.

Keep a versioned set of scenarios (goals plus personas) in the repo, in JSON or YAML. Treat them like fixtures.
For each scenario, run the conversation (scripted or simulated) and score it on your chosen dimensions.
Assert that aggregate scores meet a threshold, for example mean task completion at or above 4 out of 5, and no single dimension below 3.
Fail the build if a threshold is missed, and print the failing transcripts with the judge reasons so the diff is obvious.

Watch cost and flakiness. A simulated conversation with a judge can be a dozen model calls per scenario; a hundred scenarios is real spend and real minutes. Run the full suite on a schedule and merge gates, and a fast smaller subset on every pull request. Because judges are probabilistic, set thresholds with margin and consider averaging a metric over two or three runs, or lowering judge temperature, so a borderline score does not flap red and green on reruns.

Common mistakes in multi-turn evaluation

The failures I see teams hit when they first build this:

Testing only the last turn. If you score just the final assistant message you have not done multi-turn evaluation, you have done single-turn on the end of a conversation. Score the trajectory.
One mega-judge. A single judge scoring five dimensions at once is noisy and its reasons are useless. One judge per dimension.
Judge and model are the same instance. A model grading its own conversation inflates scores. Use a separate, strong judge.
No real transcripts. If your entire suite is simulated, you are grading your simulator's idea of a user. Anchor with real conversations.
Unversioned judges. You improve the judge prompt, scores shift, and now you cannot tell if the model got worse or the ruler changed. Version and pin the judge.
Ignoring conversation length. Many failures only appear on turn eight or twelve. If every test conversation is three turns, you will never see context loss. Include long conversations on purpose.

FAQ

What is the difference between single-turn and multi-turn evaluation? Single-turn evaluation scores one prompt and one response in isolation. Multi-turn evaluation scores a full conversation, so it can measure things that depend on history: memory of earlier turns, staying on task, recovering from a misunderstanding, and whether the whole conversation reached the user's goal. If your product is a chatbot or agent, the failures that hurt users are multi-turn, so single-turn scores give false confidence.

Do I need a framework or can I hand-roll multi-turn evaluation? You can hand-roll it. The core is a conversation data model, a loop that produces assistant turns while passing the full history, and one LLM judge per dimension. That is enough to start. Frameworks like DeepEval, Ragas, LangSmith, or Langfuse save you the plumbing, give you tested metrics, and handle tracing and aggregation. Understand the hand-rolled loop first so you can debug and tune whatever framework you adopt.

How do I evaluate a conversation without a fixed expected answer? Use an LLM-as-judge with a rubric instead of exact-match comparison. The judge reads the transcript and scores a dimension (task completion, retention, relevancy) against explicit criteria, returning a score and a reason. For grounded RAG chats you can also check faithfulness against the retrieved context. Reference-free judging is the norm for open-ended conversation, since there is rarely one correct transcript.

How many conversations do I need to test? More than a handful and fewer than you fear. Build a matrix of goals crossed with personas: five goals and four personas gives twenty scenarios that cover common paths and awkward ones. Add real production transcripts as ground truth. Scale up the simulated set for coverage, but keep the real set as the check that your simulator is realistic. Quality and diversity of scenarios matter more than raw count.

Can the simulated user replace real user testing? No. A simulated user is a way to get broad coverage cheaply and to run the suite in CI without waiting for human traffic. It is only as realistic as its prompt, and it can drift or accidentally solve the task for your assistant. Always keep a set of real human conversations as ground truth and periodically check that simulated conversations look like the real ones.

Which metrics matter most for a support or RAG chatbot? For a grounded support bot, prioritize task completion (did the conversation solve the user's problem), faithfulness (are claims grounded in retrieved docs, not hallucinated), and knowledge retention (does it use facts the user gave earlier). Add role and policy adherence if the bot has hard rules like never quoting prices. Pick three or four dimensions that map to how your users actually get hurt and score those every run.

How do I stop LLM-as-judge scores from being flaky? Lower the judge temperature, pin the judge model and prompt version, and use a strong model as the judge rather than the one under test. Ask for a score plus a reason plus an evidence turn so you can audit disagreements. For borderline metrics, average over two or three runs and set thresholds with margin so a single noisy score does not flip your CI gate red and green on reruns.