LLM Eval

Evaluating Summarization Quality: Metrics That Actually Work

Pramod Dutta · Jun 6, 2026 · 12 min read

Your summarizer passed ROUGE-L with a 0.61 score. It's also hallucinating a customer's refund amount.

This is the moment almost every team building an LLM summarization pipeline eventually hits. You ship a document summarizer, a meeting-notes generator, or a support-ticket condenser. You run it against a benchmark, get a respectable-looking ROUGE or BLEU score, and call it done. Then a user reports that the summary invented a number, flipped a decision from "approved" to "rejected," or dropped the one clause that actually mattered. The metric said the summary was fine. The summary was not fine.

This gap exists because most classic text-generation metrics were built for a different era of NLP — one where models produced short, templated outputs and "quality" mostly meant "close to the reference words." Modern LLM summarizers produce fluent, confident, and sometimes subtly wrong text. You need an evaluation strategy that catches fluency problems (which n-gram metrics still do fine) *and* faithfulness problems (which they miss almost entirely).

This article walks through what each family of summarization metrics actually measures, where each one breaks, and how to build an evaluation pipeline — including code you can run today — that catches the failures that matter in production. We'll end with how LLM-as-a-Judge fits into a mature eval stack, and where it doesn't.

Why "good English" and "faithful to the source" are different problems

Summarization quality has at least four independent dimensions, and it's worth naming them before picking metrics:

Fluency — is the text grammatical and readable?
Relevance — does it cover the important points, in proportion?
Coherence — does it read as a connected whole, not a stitched list of facts?
Faithfulness (factual consistency) — does every claim in the summary follow from the source document?

Here's the trap: fluency and relevance are the dimensions classic metrics were designed to measure, and they're also the dimensions where modern LLMs already excel by default. Faithfulness is the dimension where LLMs actually fail — via hallucination, over-generalization, or numeric drift — and it's the dimension that n-gram overlap metrics are structurally blind to.

Consider this pair:

Source: "The Q3 report shows revenue grew 4% year-over-year,
though the North American segment declined 2% due to
softer enterprise demand."

Summary A: "Revenue grew 4% YoY in Q3, driven partly by
weaker enterprise demand in North America."

Summary B: "Revenue grew 4% YoY in Q3, although the North
American segment declined 2% due to softer enterprise demand."

Summary A and Summary B share almost every word with Summary B being marginally more verbose. An n-gram metric will rate them nearly identically. But Summary A contains a factual inversion — it says weak enterprise demand *drove* growth, when the source says the opposite (it explains a regional *decline*). That's not a style problem. That's a summary you cannot ship, and ROUGE will not flag it.

ROUGE, BLEU, and why n-gram overlap runs out of road

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is still the most commonly reported summarization metric, largely due to inertia from the pre-LLM research era. ROUGE-N measures n-gram overlap between the generated summary and one or more reference summaries; ROUGE-L uses longest common subsequence.

from rouge_score import rouge_scorer

scorer = rouge_scorer.RougeScorer(
    ["rouge1", "rouge2", "rougeL"], use_stemmer=True
)

reference = (
    "The Q3 report shows revenue grew 4% year-over-year, "
    "though the North American segment declined 2% due to "
    "softer enterprise demand."
)
candidate = (
    "Revenue grew 4% YoY in Q3, driven partly by weaker "
    "enterprise demand in North America."
)

scores = scorer.score(reference, candidate)
for metric, result in scores.items():
    print(f"{metric}: precision={result.precision:.2f} "
          f"recall={result.recall:.2f} f1={result.fmeasure:.2f}")

Run that on the hallucinated Summary A above and you'll get a respectable ROUGE-1 F1 in the 0.4-0.5 range — high enough that a naive quality gate ("reject anything under 0.3") would let it through. BLEU has the same structural problem, compounded by the fact that it was designed for machine translation, where there's usually one clearly correct target sentence. Summarization is a much looser task: there are dozens of valid ways to compress the same document, and a good summary might use *none* of the reference's exact phrasing.

The three specific failure modes to remember:

Paraphrase blindness. A summary that perfectly preserves meaning using different words scores lower than a summary that copies source phrases while subtly distorting them.
No faithfulness signal. ROUGE only compares candidate to reference — never candidate to source document. A hallucination that happens to share vocabulary with the reference summary scores fine.
Reference-quality ceiling. Your ROUGE score is only as good as your reference summaries. Noisy or inconsistent references (common in scraped datasets) inject noise you can't distinguish from model error.

ROUGE and BLEU aren't useless — they're cheap, fast, deterministic, and fine as a coarse regression check ("did this change make outputs wildly shorter or longer than before"). Just don't use them as your quality gate.

BERTScore and embedding-based metrics: better, still not enough

BERTScore improves on n-gram overlap by comparing contextual embeddings of tokens rather than exact string matches, using cosine similarity between candidate and reference token embeddings (typically from a BERT-family model).

from bert_score import score

candidates = [
    "Revenue grew 4% YoY in Q3, driven partly by weaker "
    "enterprise demand in North America."
]
references = [
    "The Q3 report shows revenue grew 4% year-over-year, "
    "though the North American segment declined 2% due to "
    "softer enterprise demand."
]

P, R, F1 = score(candidates, references, lang="en", verbose=False)
print(f"BERTScore F1: {F1.item():.3f}")

This catches paraphrase better than ROUGE — "the company's income rose" and "revenue increased" will score as similar even with zero word overlap. That's a genuine improvement for measuring semantic relevance. But BERTScore is still fundamentally a candidate-vs-reference metric, not a candidate-vs-source metric. It has no mechanism to detect that "driven partly by weaker demand" reverses the causal direction of the source sentence, because both versions describe the same *topic* (revenue, North America, enterprise demand) with high embedding similarity even though the claim is inverted.

Embedding metrics also inherit a subtler problem: they're calibrated against whatever corpus the underlying encoder was trained on, and their scores don't map cleanly to "acceptable" or "unacceptable" for your domain. A 0.86 BERTScore on legal contract summaries and a 0.86 on customer support tickets don't mean the same thing about real-world usability.

Faithfulness metrics: measuring against the source, not the reference

This is the category that actually addresses the hallucination problem, and it's the one most teams skip because it takes more engineering effort. The key idea: instead of comparing summary to reference summary, you compare summary to source document.

QAGS-style methods (Question Answering for Generating Summaries) work by generating questions from the summary, answering them using both the summary and the source, and checking if the answers match. If they diverge, the summary likely contains unsupported content.

A simpler, very practical variant you can build today with an LLM:

import json

FAITHFULNESS_PROMPT = """You are checking whether a summary is
factually supported by its source document.

SOURCE DOCUMENT:
{source}

SUMMARY:
{summary}

Extract every factual claim in the summary as a short list.
For each claim, mark it as one of:
- SUPPORTED: directly stated or clearly implied by the source
- CONTRADICTED: the source says something different
- UNSUPPORTED: not mentioned in the source at all

Return JSON only, as a list of objects:
[{{"claim": "...", "verdict": "SUPPORTED|CONTRADICTED|UNSUPPORTED", "evidence": "..."}}]
"""

def check_faithfulness(client, source: str, summary: str) -> dict:
    prompt = FAITHFULNESS_PROMPT.format(source=source, summary=summary)
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    claims = json.loads(response.content[0].text)
    total = len(claims)
    supported = sum(1 for c in claims if c["verdict"] == "SUPPORTED")
    contradicted = [c for c in claims if c["verdict"] == "CONTRADICTED"]
    unsupported = [c for c in claims if c["verdict"] == "UNSUPPORTED"]
    return {
        "faithfulness_score": supported / total if total else 1.0,
        "contradictions": contradicted,
        "unsupported_claims": unsupported,
        "total_claims": total,
    }

This claim-decomposition approach is deliberately more granular than a single pass/fail judgment. A summary with nine supported claims and one contradicted claim (say, a wrong dollar figure) is a very different failure than a summary where four of ten claims are fabricated. Treating faithfulness as a *rate* over atomic claims, rather than a document-level binary, gives you a metric you can actually threshold and track over time — and it gives you the specific contradicted claim to show a human reviewer, instead of just a score.

Two things to watch when you build this yourself:

The extraction step is where errors hide. If your claim-extraction prompt is sloppy, it'll merge two claims into one or miss a claim entirely, and your faithfulness rate becomes meaningless. Test the extraction step in isolation before trusting the verdicts.
Long documents need chunked verification. If the source is longer than the judge model's effective context, don't just truncate it — retrieve the most relevant chunks per claim (a small BM25 or embedding search over source paragraphs works well) so "UNSUPPORTED" actually means unsupported, not "not in the truncated window I happened to send."

Coverage and redundancy: the metrics nobody talks about

Faithfulness gets deserved attention because hallucination is scary, but two other failure modes quietly wreck real deployments: poor coverage (the summary misses the point that mattered) and redundancy (the summary says the same thing three times to hit a length target).

A simple coverage check compares the key entities and claims in the source against what made it into the summary:

def coverage_score(source_claims: list[str], summary_claims: list[str],
                    embed_fn) -> float:
    """
    Rough coverage: fraction of source claims that have a
    semantically similar match in the summary.
    """
    import numpy as np

    source_vecs = embed_fn(source_claims)
    summary_vecs = embed_fn(summary_claims)

    covered = 0
    threshold = 0.80
    for s_vec in source_vecs:
        sims = [
            np.dot(s_vec, t_vec) /
            (np.linalg.norm(s_vec) * np.linalg.norm(t_vec))
            for t_vec in summary_vecs
        ]
        if max(sims, default=0) >= threshold:
            covered += 1

    return covered / len(source_claims) if source_claims else 1.0

The trick is deciding *which* source claims should count toward coverage. Not every sentence in a source document deserves a place in the summary — that's the whole point of summarizing. In practice, teams solve this by having a human (or a well-prompted LLM, treated as a first draft only) mark 5-10 "must include" claims per document during dataset creation, then measuring coverage against that curated set rather than every sentence in the source. This is more setup work than running an off-the-shelf metric, but it's the only way "coverage" means anything more than "word count."

Redundancy is cheaper to check — self-similarity between sentences within the summary itself, flagging near-duplicate sentences that inflate length without adding information. High redundancy is a strong signal your prompt or decoding settings are pushing the model to pad toward a target length.

Building a task-specific eval harness

None of the metrics above tell you the whole story alone. In production, the useful pattern is a small harness that runs several checks per summary and returns a structured report, not a single number.

from dataclasses import dataclass, field
from rouge_score import rouge_scorer


@dataclass
class SummaryEvalResult:
    rouge_l_f1: float
    faithfulness_score: float
    coverage_score: float
    contradictions: list = field(default_factory=list)
    passed: bool = False


def evaluate_summary(source: str, summary: str, reference: str,
                      llm_client, embed_fn,
                      must_include_claims: list[str]) -> SummaryEvalResult:
    scorer = rouge_scorer.RougeScorer(["rougeL"], use_stemmer=True)
    rouge = scorer.score(reference, summary)["rougeL"].fmeasure

    faithfulness = check_faithfulness(llm_client, source, summary)

    summary_sentences = [s.strip() for s in summary.split(".") if s.strip()]
    coverage = coverage_score(must_include_claims, summary_sentences, embed_fn)

    result = SummaryEvalResult(
        rouge_l_f1=rouge,
        faithfulness_score=faithfulness["faithfulness_score"],
        coverage_score=coverage,
        contradictions=faithfulness["contradictions"],
    )

    # Gate on the dimensions that actually matter for shipping.
    result.passed = (
        result.faithfulness_score >= 0.95
        and result.coverage_score >= 0.70
        and len(result.contradictions) == 0
    )
    return result

Notice the gating logic: ROUGE-L is *computed and logged* but not part of the pass/fail decision. That's deliberate. Track it for trend visibility (a sudden ROUGE drop across a batch is a useful early warning that something changed in your prompt or model version), but don't let it block a release, and don't let it wave through one either. Faithfulness and coverage are the two dimensions that correlate with "would a human using this summary make a bad decision because of it" — that's the bar that matters.

Where LLM-as-a-Judge fits — and where it doesn't

Everything above — QAGS-style faithfulness checking, claim extraction, coverage matching — already leans on an LLM to do the judging. That's not a coincidence; it's the honest acknowledgment that "is this summary good" is a semantic question that fixed-formula metrics were never going to answer well. LLM-as-a-Judge is the natural endpoint of this progression: instead of hand-rolling narrow checks for faithfulness or coverage separately, you use a capable model with a well-specified rubric to score summaries holistically, and you calibrate that judge against human ratings until it's trustworthy enough to run at scale.

The reason this works better than it sounds is that summarization evaluation is really a reading-comprehension task wearing a scoring costume — you have to actually understand both documents to judge whether one faithfully compresses the other, and that's precisely the kind of task LLMs are strong at. The reason it *fails* when done carelessly is also predictable: judge models have their own biases (preferring longer or more confident-sounding summaries, being lenient toward outputs that resemble their own writing style), and a judge prompt with vague criteria ("rate this summary 1-10") produces scores that don't reproduce and don't correlate with what your users actually care about.

The fixes are the same discipline you'd apply to any measurement system: give the judge a source document, not just a reference summary; ask for structured, claim-level verdicts rather than a single holistic number; run the judge multiple times or with multiple models on a sample and check agreement before trusting it in a gate; and periodically audit the judge's verdicts against real human review, especially whenever you change the judge model or prompt. Do that, and LLM-as-a-Judge stops being a hand-wavy shortcut and becomes the most reliable layer in your summarization eval stack — the one that actually catches the hallucinated refund amount that ROUGE would have happily scored at 0.61.