DeepEval

DeepEval for Summarization Tasks: A Worked Example

Ira Menon · Jun 11, 2026 · 11 min read

Why summarization is harder to evaluate than it looks

A summary can be fluent, well-formatted, and completely wrong. It can drop the one number that mattered, invert a conclusion, or quietly add a claim the source document never made. None of that shows up if you're eyeballing outputs or checking ROUGE overlap against a reference summary, because ROUGE only measures word overlap, not truthfulness or completeness.

This is the exact problem SummarizationMetric in DeepEval was built to solve. Instead of comparing your generated summary against a "gold" reference summary (which you often don't have for production traffic), it compares the summary directly against the source text along two independent axes: did the summary invent anything that isn't in the source, and did it capture the information that actually matters from the source. If your team is shipping a summarization feature — meeting notes, support ticket digests, document Q&A, changelogs — this is the metric you reach for before you reach for anything fancier.

In this article we'll break down how SummarizationMetric actually computes its score, then walk through a complete, runnable example: setting it up, tuning it, and wiring it into a test suite you can run in CI.

The two things a good summary must do

DeepEval's SummarizationMetric splits "is this a good summary" into two separate, scoreable questions:

Alignment — does the summary contradict or hallucinate information that isn't supported by the source text? A summary that adds a statistic the original document never mentioned fails on alignment, even if it reads beautifully.
Coverage — does the summary include the information that a reader actually needs from the source? A summary that is 100% accurate but omits the main conclusion fails on coverage, even though it hallucinated nothing.

The final score is deliberately unforgiving:

summarization_score = min(alignment_score, coverage_score)

Taking the minimum instead of an average means you can't compensate for hallucination by writing an exhaustive summary, and you can't compensate for missing content by being scrupulously accurate about the three facts you did include. Both properties have to hold at the same time, which mirrors how a human reviewer actually judges a summary — a single confident hallucination usually disqualifies an otherwise strong summary, and a "safe" summary that just repeats the intro paragraph doesn't count as a good summary either, no matter how faithful it is.

How alignment_score is computed

The alignment score works the same way DeepEval's hallucination checks work under the hood. An LLM judge extracts a set of factual claims (truths) from the input (the source document), then checks each claim the summary makes against those truths. Claims in the summary that aren't supported by, or that contradict, the source text count against the score.

You can control how many truths get extracted from the source using truths_extraction_limit. This matters more than it sounds — for a 20-page transcript, DeepEval will rank extracted truths by apparent importance, and capping the count keeps the judge focused on the claims that matter rather than penalizing your summary for omitting a minor aside that happened to get extracted as "truth #47."

How coverage_score is computed

Coverage works differently, and it's the more interesting half of the metric. Rather than asking an LLM to eyeball "does this cover everything important" (a notoriously unreliable prompt), DeepEval generates a set of closed-ended, yes/no assessment questions derived from the source document — questions that should be answerable from the source text alone.

Each assessment question is then asked twice: once against the original input, and once against the summary. Coverage is the proportion of questions where the two answers agree. If the source text answers "yes" to "does the document mention a pricing change?" and the summary also answers "yes" to that question when read on its own, that's a covered point. If the summary can't answer it, or answers differently, that's a coverage miss.

You can either let DeepEval auto-generate these questions (and control how many with the n parameter) or supply your own assessment_questions list when you know exactly what a summary in your domain must preserve — say, a dollar amount, a deadline, or a named decision-maker.

Setting up SummarizationMetric

Install DeepEval and set your model provider key (OpenAI by default, though DeepEval supports swapping in other providers as the judge model):

pip install deepeval
export OPENAI_API_KEY="sk-..."

The minimum viable usage only needs an LLMTestCase with an input (the source document) and an actual_output (your model's summary):

from deepeval.test_case import LLMTestCase
from deepeval.metrics import SummarizationMetric

source_document = """
Q3 revenue grew 18% year-over-year to $42M, driven primarily by
enterprise contract renewals. Churn in the SMB segment rose to 6.2%,
up from 4.1% last quarter, largely attributed to a pricing change
introduced in July. The board approved a $10M share buyback program
and flagged supply chain costs as the primary risk heading into Q4.
"""

generated_summary = """
Q3 revenue was up 18% YoY to $42M on the back of enterprise renewals.
SMB churn increased to 6.2% following a July pricing change. The board
approved a $10M buyback and is watching supply chain costs for Q4.
"""

test_case = LLMTestCase(
    input=source_document,
    actual_output=generated_summary,
)

metric = SummarizationMetric(
    threshold=0.5,
    model="gpt-4",
    include_reason=True,
)

metric.measure(test_case)

print(metric.score)
print(metric.reason)
print(metric.score_breakdown)

Running metric.measure(test_case) triggers both sub-evaluations. metric.score gives you the final min() value, metric.reason gives a natural-language explanation the judge model generated for why the score landed where it did, and metric.score_breakdown exposes the alignment and coverage numbers separately — which is what you actually want when you're debugging a low score, since "0.4 overall" tells you almost nothing on its own.

Reading the score_breakdown to debug failures

This is where SummarizationMetric earns its keep over a single blended score. Suppose you get metric.score = 0.33. Without a breakdown, you'd be guessing whether the model hallucinated or under-covered. With score_breakdown, you might see something like:

{
    "Alignment": {
        "score": 1.0,
        "reason": "The summary contains no claims that contradict or extend beyond the source text."
    },
    "Coverage": {
        "score": 0.33,
        "reason": "The summary omits the SMB churn figure and the board's supply chain risk comment, which the source document treats as material."
    }
}

That tells you immediately: your model isn't hallucinating, it's under-summarizing. The fix is a prompt change (ask the summarizer to always include quantitative changes and named risks), not a hallucination-mitigation fix like retrieval grounding. Conversely, if alignment tanks and coverage is high, you know your summarizer is padding the summary with invented specifics — a very different bug with a very different fix.

Here's a version of the earlier example, but with a summary that hallucinates a number, to show how alignment catches it:

from deepeval.test_case import LLMTestCase
from deepeval.metrics import SummarizationMetric

source_document = """
Q3 revenue grew 18% year-over-year to $42M, driven primarily by
enterprise contract renewals. Churn in the SMB segment rose to 6.2%,
up from 4.1% last quarter, largely attributed to a pricing change
introduced in July. The board approved a $10M share buyback program
and flagged supply chain costs as the primary risk heading into Q4.
"""

hallucinated_summary = """
Q3 revenue grew 18% YoY to $42M. The company also announced it is
expanding into the European market next quarter, and SMB churn
improved slightly to 3.8%. The board approved a $10M buyback.
"""

test_case = LLMTestCase(
    input=source_document,
    actual_output=hallucinated_summary,
)

metric = SummarizationMetric(threshold=0.5, model="gpt-4", include_reason=True)
metric.measure(test_case)

print(f"Score: {metric.score}")
print(f"Passed: {metric.is_successful()}")
print(metric.score_breakdown)

Here the summary invents a European expansion the source never mentions, and misstates the churn direction (it went up, not down). Alignment should come back low because both claims are unsupported by or contradict the source, dragging the final min() score down regardless of how good coverage looks.

Using custom assessment_questions for domain-specific coverage

Auto-generated assessment questions are a reasonable default, but in production you usually know exactly what a summary in your domain is not allowed to drop. For a support-ticket summarizer, that might be the customer's account tier and whether a refund was requested. For a legal-document summarizer, it might be effective dates and party names. Supplying your own list turns coverage from "did it cover what an LLM decided was important" into "did it cover what we decided was important":

from deepeval.test_case import LLMTestCase
from deepeval.metrics import SummarizationMetric

ticket_transcript = """
Customer (Enterprise tier) reports that exported CSV files are
missing the 'last_login' column since the October 12 release.
They are requesting a refund for this billing cycle if the bug
is not fixed within 5 business days. Engineering has reproduced
the issue and expects a fix by October 17.
"""

agent_summary = """
Enterprise customer hit a bug where CSV exports are missing the
last_login column after the Oct 12 release. Engineering has
reproduced it and is targeting a fix for Oct 17. Customer has
asked for a refund if it isn't resolved within 5 business days.
"""

test_case = LLMTestCase(input=ticket_transcript, actual_output=agent_summary)

metric = SummarizationMetric(
    threshold=0.7,
    model="gpt-4",
    assessment_questions=[
        "Does the summary mention the customer's account tier?",
        "Does the summary mention that a refund was requested?",
        "Does the summary mention an expected fix date?",
    ],
    include_reason=True,
)

metric.measure(test_case)
print(metric.score)
print(metric.score_breakdown)

Because every assessment question is answered against both the source and the summary independently, this approach also catches a subtler failure mode: a summary that mentions a refund in passing but frames it incorrectly (e.g., says a refund was already issued instead of requested) will disagree with the source's answer to that specific question, and coverage will reflect it — something plain keyword matching would miss entirely.

Wiring it into a test suite with pytest

Ad hoc scoring is fine for exploration, but the actual value of DeepEval shows up when SummarizationMetric runs automatically on every pull request. Because DeepEval integrates with pytest, you can assert on the metric directly:

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import SummarizationMetric

def build_test_case(source, summary):
    return LLMTestCase(input=source, actual_output=summary)

@pytest.mark.parametrize("source,summary", [
    (
        "The API rate limit is 100 requests per minute per API key. "
        "Exceeding it returns a 429 status code with a Retry-After header.",
        "The API allows 100 requests per minute per key; going over "
        "returns a 429 with a Retry-After header.",
    ),
])
def test_summarization_quality(source, summary):
    test_case = build_test_case(source, summary)
    metric = SummarizationMetric(threshold=0.7, model="gpt-4")
    assert_test(test_case, [metric])

Run it the same way you'd run any pytest suite:

deepeval test run test_summarization.py

deepeval test run gives you DeepEval's own reporting on top of pytest's — pass/fail per test case, the score breakdown, and the judge's reasoning for each failure, so a CI failure comes with an explanation instead of just a red X. That's the difference between a summarization regression getting caught in code review versus getting caught by an angry customer three weeks after deploy.

Practical tuning notes

A few things worth knowing before you drop this into a real pipeline:

Pick a threshold you can defend, not 0.9 by default. SummarizationMetric uses min(alignment, coverage), which is already a strict combination. A threshold of 0.7–0.8 is a reasonable starting point for most use cases; ratchet it up once you've seen how your specific summarizer scores on real traffic.
Cap `truths_extraction_limit` for long documents. Without a cap, DeepEval will extract as many truths as the judge model finds, and a 30-page document can generate far more "facts" than any reasonable summary is expected to cover, unfairly punishing coverage.
Prefer custom `assessment_questions` once you know your domain. Auto-generated questions are a good starting point during prototyping, but hand-written ones encode what your team has actually decided is non-negotiable in a summary.
The judge model matters. Both alignment and coverage depend on an LLM acting as judge. Using a stronger model (e.g., gpt-4 over a smaller model) as the judge generally produces more reliable assessment questions and truth extraction, at the cost of slower and more expensive evaluation runs — a trade-off worth making for your CI suite even if your production summarizer itself uses a cheaper model.
Treat `score_breakdown` as your primary debugging tool. A single scalar score tells you something failed; the breakdown tells you what to fix.

Where this fits in a broader eval strategy

SummarizationMetric shouldn't be the only check on a summarization feature, but it's usually the first one worth automating because hallucination and omission are the two failure modes that actually hurt users — a summary that's slightly awkwardly worded is a UX nit, a summary that fabricates a number is a trust problem. Pairing it with DeepEval's other reference-free metrics (like faithfulness checks on RAG pipelines feeding the summarizer, or answer relevancy checks if the summary is itself an answer to a user query) gives you coverage across the pipeline rather than just at the final output.

If you're building this out for a real product, the pattern that works well in practice is: start with auto-generated assessment questions to get a baseline fast, watch score_breakdown on a sample of real outputs to see whether alignment or coverage is your actual bottleneck, then invest in custom assessment questions only for the domain facts you genuinely cannot afford to lose in a summary.

If you want a guided, hands-on walkthrough of this and the rest of DeepEval's metric suite — hallucination, answer relevancy, faithfulness, and building CI pipelines around all of them — our DeepEval Tutorial course on teachyou.ai covers it step by step, with real datasets and graded exercises instead of toy examples.