Evaluating Multi-Turn Conversations: Beyond Single-Response Metrics

Why Your Eval Pipeline Is Lying to You

You shipped a customer support agent. Every individual response passed your eval suite — helpfulness scores above 4/5, no hallucinations flagged, tone checks green across the board. Then a user opens a support ticket complaining that the bot "kept forgetting what I told it" and "contradicted itself twice in the same conversation." You go back and check the transcript. Every single turn, evaluated in isolation, looks fine. The problem only exists in the relationship between turns.

This is the blind spot in most LLM evaluation setups today. We inherited our eval habits from single-turn benchmarks — MMLU, HellaSwag, single-prompt helpfulness scoring — because that's how the first generation of chat models was tested. But production systems are rarely single-turn. They're agents that hold context across five, ten, fifty turns. They're tutoring bots that need to remember a student said they're a beginner three messages ago. They're coding assistants that need to keep track of a file they edited two turns back. When you evaluate each response as if it exists in a vacuum, you're measuring a different product than the one your users experience.

This article is about the gap between "is this response good" and "is this conversation good," why that gap matters more than most teams realize, and how to build an eval harness that actually catches multi-turn failures before your users do.

The Failure Modes Single-Turn Metrics Can't See

Before building anything, it helps to have a concrete taxonomy of what actually breaks across turns. In practice, most multi-turn failures fall into five buckets.

Context amnesia. The model forgets information the user provided earlier — a name, a constraint, a preference — and asks for it again or acts as if it was never given. This is the most common failure and the easiest to miss, because turn 7 in isolation looks like a perfectly reasonable response. It's only wrong relative to turn 2.

Self-contradiction. The model states something in turn 3 and the opposite in turn 8. In a coding assistant, this might look like recommending useEffect for a data fetch early on, then in a later turn recommending against it for the same use case without acknowledging the change of position. Contradiction detection requires comparing claims across the whole transcript, not scoring any one message.

Goal drift. The user came in wanting to debug a failing test. Five turns later, the conversation is about refactoring an unrelated module, and nobody explicitly decided to switch tasks — the model just followed tangents. Each individual turn is topically coherent with the one before it, but the conversation as a whole has lost its anchor.

Compounding errors. A small factual slip in turn 2 becomes the foundation for turn 4's reasoning, which becomes the foundation for turn 6's final answer. Any single turn looks like a reasonable next step given what came before — the error is invisible unless you trace it back to its source.

Failure to escalate or terminate. The model should recognize when a conversation needs to hand off to a human, or when the user's request has already been satisfied and continuing is unhelpful (over-eager follow-up questions, restating things already said). This is a property of the trajectory, not any single message.

None of these show up in a rubric that asks "was this response helpful, accurate, and safe?" applied turn by turn. You need metrics that look at the conversation as a unit.

Reframing the Unit of Evaluation

The core mental shift is this: the unit of evaluation should be the conversation, not the turn. That sounds obvious once stated, but it has real implications for how you structure your eval pipeline.

A single-turn eval pipeline usually looks like this: take a prompt, generate a response, score the response against some criteria, aggregate scores across a test set. A multi-turn-aware pipeline needs an extra layer: take a full transcript (or a growing prefix of one), score properties that depend on the *history*, and only then aggregate.

Concretely, this means your eval function's signature changes. Instead of:

def score_response(prompt: str, response: str) -> float:
    ...

You need something closer to:

def score_turn_in_context(
    conversation_history: list[dict],
    current_turn: dict,
) -> dict:
    """
    conversation_history: all prior turns (user + assistant),
        including any tool calls and their results
    current_turn: the turn under evaluation

    Returns a dict of named scores, e.g.:
    {
        "consistency_with_history": 0.9,
        "context_retention": 1.0,
        "goal_alignment": 0.8,
    }
    """
    ...

The second signature forces you to actually pass the history into the judge, which is the single most important change most teams need to make. It sounds trivial, but a surprising number of "multi-turn eval" setups I've seen in the wild still score each assistant message with only the immediately preceding user message as context, discarding everything before that. That's not a multi-turn eval — it's a single-turn eval running in a loop.

Building Trajectory-Level Metrics

Once you're passing full history into your evaluator, you can define metrics that only make sense at the trajectory level.

Context retention rate. Seed the conversation with specific facts (a name, a budget, a stated constraint) early on, then at various later turns, ask the model something that requires recalling that fact — either directly, by asking again in a different form, or indirectly, by checking whether a later recommendation is consistent with the constraint. Score the fraction of retention checks the model passes.

Consistency score. For every pair of assistant turns, check whether they make claims that logically conflict. This is naturally suited to an LLM judge, because contradiction detection between natural-language statements is not something you can do reliably with string matching or embeddings alone — you need something that understands semantics and negation.

Task completion trajectory. Define the user's actual goal for the conversation up front (this usually requires human annotation of your eval set, or a strong model summarizing the goal from turn 1). Then, at the end of the conversation, check whether that goal was met, partially met, or abandoned. This catches goal drift because it evaluates the destination against the stated origin, not against local progress.

Recovery ability. Deliberately inject a wrong turn (either from the user, simulating a correction, or by having the model make a mistake naturally) and measure whether the model corrects course gracefully in subsequent turns rather than doubling down. This is close to a robustness metric, but scoped to dialogue rather than single-prompt robustness.

Turns-to-resolution. For task-oriented conversations, count how many turns it takes to reach a satisfactory answer. This is a proxy for efficiency, but also indirectly measures whether the model is asking useful clarifying questions versus going in circles.

Here's a simplified harness that runs a scripted multi-turn test case and scores it on several of these axes using an LLM judge:

from dataclasses import dataclass, field
from typing import Literal

Role = Literal["user", "assistant"]

@dataclass
class Turn:
    role: Role
    content: str

@dataclass
class ConversationCase:
    turns: list[Turn]
    seeded_facts: dict[str, str]  # e.g. {"budget": "$500", "name": "Priya"}
    stated_goal: str

def build_judge_prompt(case: ConversationCase, transcript: list[Turn]) -> str:
    history_text = "\n".join(f"{t.role.upper()}: {t.content}" for t in transcript)
    facts_text = "\n".join(f"- {k}: {v}" for k, v in case.seeded_facts.items())

    return f"""
You are auditing a multi-turn AI conversation for consistency and context retention.

STATED USER GOAL:
{case.stated_goal}

FACTS ESTABLISHED EARLIER IN THE CONVERSATION:
{facts_text}

FULL TRANSCRIPT:
{history_text}

Score the transcript from 1-5 on each of the following, and justify each score
with a specific quote from the transcript:

1. context_retention: Did the assistant correctly use the established facts
   whenever they were relevant, without asking for them again or contradicting them?
2. internal_consistency: Are there any two assistant turns that make conflicting
   claims or recommendations?
3. goal_alignment: Did the conversation stay anchored to the stated user goal,
   or did it drift into unrelated territory without the user asking for that?

Return strict JSON:
{{"context_retention": <1-5>, "context_retention_reason": "...",
  "internal_consistency": <1-5>, "internal_consistency_reason": "...",
  "goal_alignment": <1-5>, "goal_alignment_reason": "..."}}
"""

def evaluate_conversation(case: ConversationCase, judge_call) -> dict:
    prompt = build_judge_prompt(case, case.turns)
    raw = judge_call(prompt)
    return raw  # parse to JSON in production, with retries on malformed output

Notice the judge prompt explicitly asks for a quote to justify each score. This is not decoration — it's a guardrail against the judge hallucinating a plausible-sounding score without actually re-reading the transcript. Requiring citations forces the grounding step.

Designing Test Conversations, Not Just Test Prompts

Most eval sets are collections of independent prompts. A multi-turn eval set needs to be a collection of *scripted conversation arcs*, each one designed to probe a specific failure mode.

A useful pattern is to write test conversations as fixtures with an explicit "trap" built in — a turn where the correct behavior is well-defined and failure is unambiguous. For example:

1. Turn 1: user states a hard constraint ("I'm allergic to shellfish, please keep that in mind for any food recommendations").
2. Turns 2-5: unrelated conversation about restaurant ambience, budget, location.
3. Turn 6: user asks for a menu recommendation.

The correct behavior at turn 6 is unambiguous — no shellfish should appear in the recommendation. This is a clean context-retention trap: it's testable programmatically (you can literally grep the response for shellfish-adjacent terms as a first-pass check) and it isolates one specific failure mode without conflating it with others.

Similarly, for consistency traps:

1. Turn 1: user asks the model to recommend a database for a project with specific requirements.
2. Turn 4: after more requirements are added, the user asks the model to summarize its recommendation so far.

If the summary at turn 4 doesn't match what was actually recommended at turn 1 (assuming the added requirements didn't invalidate it), that's a consistency failure you can catch.

The key design principle: each test conversation should isolate one hypothesis about what might break, the same way a good unit test isolates one behavior. A 15-turn conversation that vaguely "tests general conversational ability" gives you a score but no diagnosis. A 6-turn conversation built specifically to test whether a stated dietary constraint survives five unrelated turns gives you an actionable signal — you know exactly what broke and where.

Handling Long-Context Degradation

A specific and increasingly important sub-problem is what happens as conversations get long enough to approach the context window, or long enough that the relevant fact is now buried under thousands of tokens of intervening dialogue. This deserves separate treatment because the failure mode is architectural, not just a prompting issue.

Two things worth measuring separately here:

Position sensitivity. Take the same seeded fact and place it at different positions in a conversation of the same total length — early, middle, late — then test retention at the end. Many models show a "lost in the middle" pattern where facts stated in the middle of a long context are recalled less reliably than facts at the very start or very end. If you only ever test with facts seeded at turn 1, you'll systematically overestimate retention for your production traffic, where important facts get established at all sorts of positions.

Retrieval versus generation errors. When a model fails to use an earlier fact, distinguish whether it never had access to it (truncated context, summarization dropped it, RAG retrieval missed it) from whether it had access but failed to use it. These require completely different fixes — one is an infrastructure problem, the other is a prompting or model-capability problem. Your eval harness should log what was actually in the context window at generation time, not just what was in the full conversation, so you can tell these apart.

If your system uses conversation summarization to manage context length (a common pattern once conversations exceed a manageable token budget), you should evaluate the summarization step itself as a first-class component, not just the final response. A summary that silently drops the shellfish allergy is the root cause of a downstream retention failure that will otherwise look like a model reasoning problem.

Aggregating Scores Without Losing the Signal

Once you have per-conversation scores across several axes, the temptation is to average everything into a single number and track that over time. Resist this more than you would with single-turn evals, because multi-turn failures are often rare-but-severe rather than uniformly distributed.

A model that retains context perfectly in 95% of conversations but completely fails on long conversations with facts stated in the middle is a genuinely dangerous production model — that 5% might correspond to your highest-value, longest-engagement users. A single averaged "conversation quality" score of 0.95 hides this completely. Prefer reporting distributions and worst-case slices over single averages:

• Break down scores by conversation length bucket (2-3 turns, 4-8 turns, 9+ turns) — retention and consistency typically degrade with length, and a flat average obscures the cliff.
• Track the worst-scoring axis per conversation, not just the average axis score, so a conversation that's great on goal alignment but terrible on consistency doesn't get smoothed into "fine."
• Keep a standing regression set of conversations that previously failed, and re-run them on every model or prompt change — multi-turn failures are exactly the kind of thing that silently regresses when you tweak a system prompt to fix an unrelated single-turn issue.

This is also where human review still matters. Automated multi-turn evals are good at flagging candidates for review — long conversations, low-consistency scores, conversations where the judge itself expressed low confidence — but the final call on genuinely ambiguous cases (was that actually a contradiction, or a legitimate change of recommendation given new information?) benefits from a human spot-check, at least until you've built enough calibration data to trust the automated judge on that specific failure mode.

Putting It Into Practice

If you're starting from a single-turn eval pipeline and want to add multi-turn coverage without a rewrite, a reasonable rollout looks like this:

1. Inventory your production conversations by length and identify the length distribution your users actually hit — there's no point building elaborate 20-turn test fixtures if 90% of real conversations are 3 turns.
2. Write 15-20 scripted conversation fixtures, each targeting one specific failure mode (context retention, consistency, goal drift), with an explicit trap turn and an unambiguous correct answer.
3. Extend your judge prompt to accept full transcript history, not just the current turn, and require quoted justification for each score.
4. Run the fixture set against your current production model as a baseline, and treat any conversation that scores badly as a bug report, not just a data point.
5. Add the fixture set to CI or your regular eval cadence so a system prompt change or model swap can't silently regress multi-turn behavior.

None of this replaces single-turn evaluation — you still need to know if any individual response is factually wrong, unsafe, or off-tone. But single-turn metrics answer "is this response acceptable," and that is a necessary but insufficient question for anything that ships as a conversation rather than a one-shot completion. The moment your product holds state across turns, your eval suite has to hold state across turns too.

The judge prompt pattern shown here — feeding a full transcript into a capable model, asking for structured scores with quoted evidence — is itself a specific case of a broader technique: using LLM-as-a-Judge to automate evaluation that would otherwise require exhaustive human annotation. It scales well precisely because "did this response contradict an earlier one" is a semantic judgment, not a pattern match, and that's exactly the kind of judgment a well-prompted judge model is good at making — as long as you give it the whole conversation, not just the last line.