The LLM Evaluation Metrics Map

Picking llm eval metrics feels harder than it should be because the field ships a new metric every few months and nobody tells you which one replaces which. This article maps the whole space into six families: string-based, embedding-based, LLM-as-judge, task-specific (RAG and agents), classification-style, and human-in-the-loop. Read it once and you will be able to look at any evaluation problem and know which family to reach for, what each metric actually measures, and where it quietly breaks.

The short version: if your output has one correct answer, use exact match or a classifier. If it has many correct phrasings, use embedding similarity or an LLM judge. If it is grounded in retrieved documents, use faithfulness and context metrics. If it is an agent doing multi-step work, use trajectory and task-completion metrics. Everything below expands on why.

Why generic accuracy fails for LLM outputs

Traditional software testing checks one thing: does output equal expected output. LLM outputs are usually free text, and free text has no single correct string. "The capital of France is Paris" and "Paris is France's capital" are both right and share zero exact tokens in the same order. This is the root problem every llm eval metric tries to solve in its own way.

There are three separate things people conflate under "evaluation":

• Capability testing: does the model know the answer, in a controlled benchmark setting.
• Regression testing: did this prompt or model change break behavior that used to work.
• Production monitoring: is the live system producing acceptable outputs right now, on real traffic.

Different metrics suit different stages. A benchmark score is nearly useless for catching a broken system prompt in production; a real-time LLM judge is overkill for a nightly regression suite of 50,000 cases. Keep this distinction in mind as you read the rest of the map, because it will settle most "which metric should I use" arguments faster than the metric's math will.

String-based and lexical metrics

These are the oldest llm eval metrics, mostly inherited from machine translation and summarization research. They compare generated text to one or more reference texts using surface-level overlap.

Exact match (EM): output equals reference, normalized for case and whitespace. Good for QA tasks with a single short answer (dates, numbers, named entities). Useless for anything with paraphrase freedom.

F1 over tokens: precision and recall of overlapping tokens between output and reference, used heavily in extractive QA (SQuAD-style). More forgiving than EM because partial overlap still scores.

BLEU: n-gram precision with a brevity penalty, from machine translation. Penalizes short outputs, rewards outputs that reuse reference phrasing closely. Weak on paraphrase, weak on single-reference setups, still shows up in translation and code-generation pipelines because it is fast and deterministic.

ROUGE (1/2/L): n-gram and longest-common-subsequence recall, built for summarization. ROUGE-L in particular tolerates word reordering better than BLEU. Still purely lexical, so a summary that captures the right meaning in different words scores poorly.

Minimal runnable example comparing EM and F1 on a QA-style pair:

from collections import Counter

def normalize(text):
    return text.lower().strip().rstrip(".")

def exact_match(pred, ref):
    return int(normalize(pred) == normalize(ref))

def token_f1(pred, ref):
    pred_tokens = normalize(pred).split()
    ref_tokens = normalize(ref).split()
    common = Counter(pred_tokens) & Counter(ref_tokens)
    overlap = sum(common.values())
    if overlap == 0:
        return 0.0
    precision = overlap / len(pred_tokens)
    recall = overlap / len(ref_tokens)
    return 2 * precision * recall / (precision + recall)

pred = "Paris is the capital of France"
ref = "The capital of France is Paris"

print("EM:", exact_match(pred, ref))
print("F1:", round(token_f1(pred, ref), 3))

Notice EM returns 0 while F1 gives a reasonable partial score, because F1 only cares about the bag of tokens, not their order. That is the whole tradeoff of the lexical family in one example: cheap, deterministic, explainable, but blind to meaning.

Use string-based metrics when: the task has short, structured, mostly-canonical answers (extraction, classification-as-text, translation with a fixed reference set), or when you need a metric that runs in milliseconds with zero API cost across millions of rows.

Avoid them when: answers can be correct in many different phrasings, which covers most chat, summarization, and generation tasks people actually ship.

Embedding and semantic similarity metrics

These metrics replace token overlap with vector similarity, so paraphrases score well even without shared words.

Cosine similarity of embeddings: encode both texts with a sentence embedding model, compute cosine similarity. Simple, fast, works across most encoder models. The failure mode: it rewards topical similarity even when the actual claim is wrong. "Revenue grew 10%" and "Revenue fell 10%" will still land close together in embedding space because the sentences are structurally similar, so this metric alone will not catch a flipped sign or a wrong number.

BERTScore: instead of one vector per sentence, it computes token-level contextual embeddings and greedily matches each candidate token to its best-scoring reference token, then aggregates into precision/recall/F1. More sensitive to word-level substitutions than plain cosine similarity, still a similarity score rather than a correctness score.

Semantic answer similarity for QA: a specialization where you embed just the answer span, not the whole response, so verbose or hedging responses do not get penalized for extra words as harshly as pure BLEU/ROUGE would.

# Sketch using any sentence-embedding model you already have loaded
import numpy as np

def cosine_sim(vec_a, vec_b):
    vec_a, vec_b = np.array(vec_a), np.array(vec_b)
    return float(np.dot(vec_a, vec_b) / (np.linalg.norm(vec_a) * np.linalg.norm(vec_b)))

# embed_fn(text) -> list[float], swap in your embedding model of choice
def semantic_similarity(pred, ref, embed_fn):
    return cosine_sim(embed_fn(pred), embed_fn(ref))

Use embedding metrics when: you need a fast, referenceable score across paraphrase-heavy outputs (summaries, rewrites, translations) and you can tolerate that they measure "sounds similar" rather than "is factually correct."

Avoid them for: anything where getting a number, date, name, or polarity wrong is the exact failure you are trying to catch, since embedding similarity is famously insensitive to negation and numeric swaps.

LLM-as-judge metrics

This is the family that ate the evaluation world over the last few years, because it is the only approach flexible enough to score open-ended generation against criteria that are hard to write as code: helpfulness, tone, instruction-following, coherence, correctness against a rubric.

The pattern: you send the judge model the input, the output (and often a reference answer or rubric), and ask it to score or classify. Two shapes dominate:

Pointwise scoring: judge scores one output in isolation, usually 1-5 or pass/fail against explicit criteria. Cheap, parallelizable, but absolute scores drift across judge-model versions, so treat scores as relative within a single evaluation run, not as a stable long-term number.

Pairwise comparison: judge sees two outputs (e.g., new prompt vs. old prompt) and picks the better one, or ties. More reliable than pointwise scoring for A/B testing prompt or model changes, because relative judgments are easier for a judge model to make consistently than absolute ones.

A minimal pointwise judge call, structured so the score is easy to parse and log:

JUDGE_PROMPT = """You are grading a customer support response for correctness and tone.

Question: {question}
Reference answer: {reference}
Model response: {response}

Score the model response from 1 to 5:
5 = fully correct and matches the reference in substance, tone is professional
3 = partially correct or missing minor details
1 = incorrect or contradicts the reference

Reply with only the number."""

def judge_score(question, reference, response, llm_call):
    prompt = JUDGE_PROMPT.format(
        question=question, reference=reference, response=response
    )
    raw = llm_call(prompt)
    try:
        return int(raw.strip())
    except ValueError:
        return None  # log and inspect malformed judge outputs, don't silently drop them

Known failure modes to design around, not ignore:

• Position bias: in pairwise comparisons, judges lean toward whichever answer appears first. Fix by running both orderings and averaging, or randomizing order per example.
• Self-preference bias: a judge model tends to score outputs from its own model family slightly higher. If you are comparing models, use a judge from a different family than either candidate where possible.
• Verbosity bias: judges often rate longer answers as better independent of quality. Add an explicit rubric line telling the judge conciseness is a criterion, not a bonus.
• Rubric drift: vague criteria like "is this good" produce noisy, low-agreement scores. Every rubric line should be something a human could apply the same way twice.

Always calibrate a new judge prompt against a small set of human-labeled examples (50-100 is a reasonable start) before trusting it at scale. Report agreement between the judge and human raters, not just the judge's raw scores, so you know how much noise you're inheriting.

Use LLM-as-judge when: the quality dimension is genuinely subjective or too complex to encode as a formula (tone, helpfulness, instruction adherence, open-ended correctness), and you can afford the latency and API cost of a second model call per evaluation.

Avoid it as your only signal when: the criteria are actually objective (a number, a date, valid JSON, a specific entity). Use a deterministic check for those and reserve the judge for the genuinely fuzzy parts.

Task-specific metrics: RAG systems

Retrieval-augmented generation introduces failure modes that generic text metrics cannot see, because the output can be fluent and well-written while being completely unsupported by the retrieved context. RAG evaluation typically splits into retrieval-side and generation-side metrics.

Retrieval-side:

• Context precision: of the chunks retrieved, what fraction are actually relevant to the question. Low precision means your retriever is pulling noise that dilutes the context window and can distract the generator.
• Context recall: of the information needed to answer correctly, what fraction appears somewhere in the retrieved chunks. Low recall means the answer literally cannot be assembled from what was retrieved, no matter how good the generator is.

Generation-side:

• Faithfulness (a.k.a. groundedness): does every claim in the generated answer trace back to something in the retrieved context. This is the single most important RAG metric because it directly measures hallucination against your own knowledge base, independent of whether the answer happens to also be true in the real world.
• Answer relevancy: does the generated answer actually address the question asked, as opposed to being faithful to the context but off-topic (a common failure when the retriever pulls tangentially related chunks and the generator dutifully summarizes them instead of answering the question).

A simple faithfulness check pattern using an LLM judge, decomposing the answer into atomic claims first:

CLAIM_EXTRACTION_PROMPT = """List the individual factual claims made in this answer, one per line, as short standalone statements.

Answer: {answer}"""

CLAIM_CHECK_PROMPT = """Context: {context}

Claim: {claim}

Is this claim directly supported by the context above? Reply only: supported, contradicted, or unsupported."""

def faithfulness_score(answer, context, llm_call):
    claims_raw = llm_call(CLAIM_EXTRACTION_PROMPT.format(answer=answer))
    claims = [c.strip("- ").strip() for c in claims_raw.splitlines() if c.strip()]
    if not claims:
        return None
    supported = 0
    for claim in claims:
        verdict = llm_call(CLAIM_CHECK_PROMPT.format(context=context, claim=claim)).strip().lower()
        if verdict.startswith("supported"):
            supported += 1
    return supported / len(claims)

Decomposing into atomic claims before checking each one against the context produces far more reliable faithfulness scores than asking a judge to eyeball the whole answer at once, because "mostly grounded with one fabricated detail" is exactly the case a holistic judge tends to miss.

Use RAG-specific metrics whenever your system retrieves before generating, even if the retrieval step is simple keyword search rather than a vector database. Faithfulness and context recall will catch different failures (hallucination vs. missing information) so run both, not just one.

Task-specific metrics: agents and tool use

Agentic systems add another layer: correctness now depends on the sequence of actions taken, not just the final text. A few metrics specific to this setting:

• Tool selection accuracy: given a step where a tool call was needed, did the agent pick the correct tool from the available set.
• Argument correctness: did the agent call the right tool with the right arguments (right file path, right API parameters, right query).
• Task completion rate: across a suite of multi-step tasks, what fraction did the agent complete successfully end to end. This is the metric that matters most to users and the hardest to get a clean signal on, because "completion" for an open-ended task often needs a judge or a hand-written verifier per task.
• Trajectory efficiency: number of steps or tool calls taken versus a minimal reference trajectory. An agent that gets the right answer after twenty unnecessary tool calls has a real problem even though a pure completion-rate metric would call it a success.
• Groundedness under tool results: same idea as RAG faithfulness, applied to tool outputs instead of retrieved documents. Does the final answer actually reflect what the tools returned, or did the model ignore a tool error and answer anyway.

A minimal trajectory checker comparing an agent's tool call sequence against an expected one:

def trajectory_match(actual_calls, expected_calls, strict_order=True):
    """actual_calls / expected_calls: list of (tool_name, args_dict)"""
    if strict_order:
        return actual_calls == expected_calls
    # order-insensitive: did the agent make all the necessary calls, extras allowed
    return all(call in actual_calls for call in expected_calls)

Strict-order matching is the right default for tasks with a genuinely required sequence (must read a file before editing it). Order-insensitive matching suits tasks where several valid paths reach the same result. Pick per task, not per project, because mixing the two silently under- or over-penalizes agents depending on the task shape.

Classification-style metrics for structured outputs

When the LLM's job is closer to classification than open generation, standard machine learning metrics apply directly and are usually the right choice over an LLM judge:

• Precision, recall, F1 for tasks like intent classification, content moderation labels, or entity extraction with a fixed schema.
• Exact match on structured fields for JSON-schema outputs, function calls, or extraction tasks where each field has one correct value.
• Schema validity rate: separate from correctness, track what fraction of outputs are even parseable against the expected schema. A model that produces malformed JSON 5% of the time has a reliability problem that correctness metrics alone will hide, since malformed outputs often get silently dropped from the correctness calculation.

These metrics are cheap, deterministic, and reproducible, which makes them the right default whenever the task genuinely reduces to picking from a fixed set of valid outputs. Reach for an LLM judge only for the residual open-ended part of the task, not the whole thing.

Human evaluation, still the ground truth

Every automated llm eval metric above is ultimately a proxy for human judgment, calibrated (if you calibrated it at all) against a small human-labeled sample. Human evaluation stays essential for:

• Calibrating and auditing LLM judges, as covered above, on a recurring basis (judge and model versions both drift).
• Catching failure modes nobody thought to write a metric for. Automated metrics only measure what you told them to measure; humans reading raw transcripts catch the weird stuff.
• High-stakes domains (medical, legal, financial) where the cost of an automated metric's blind spot is too high to accept without spot-checking.

A practical pattern: sample one to two percent of production traffic weekly for human review, using the same rubric your LLM judge uses, and track agreement over time. When agreement drops, that is your signal the judge prompt or the underlying model has drifted and needs recalibration, not a signal to abandon automation entirely.

Choosing metrics for your use case

A quick decision path that covers most real projects:

1. Does the output have one correct value? (classification label, extracted field, yes/no) -> use exact match, F1, or schema validity. Skip the judge.
2. Is the output free text with many valid phrasings but no external grounding required? (summarization, rewriting, chat) -> use an LLM judge with an explicit rubric, backed by a small human-calibration set. Add embedding similarity as a cheap secondary signal for regression tracking between judge runs.
3. Does the output depend on retrieved context? -> add faithfulness and context recall/precision on top of whatever generation metric you picked in step 2. Faithfulness is non-negotiable for RAG; skipping it means you have no hallucination signal at all.
4. Does the system take multiple actions before answering? -> add task completion rate and trajectory metrics. Treat the final-answer quality metric (steps 1-3) as necessary but not sufficient.
5. Is this feeding a production monitoring dashboard rather than a one-time benchmark? -> favor metrics cheap enough to run on 100% of traffic (schema validity, deterministic checks, sampled LLM judge) over metrics that only make sense in a controlled offline batch (full BLEU/ROUGE suites against curated references).

Most production systems end up running a stack, not a single metric: deterministic checks for anything objective, an LLM judge for the subjective remainder, faithfulness metrics if retrieval is involved, and a thin slice of human review to keep the automated layer honest. The map above is for picking which layers you actually need, so you add complexity only where a real failure mode requires it, not by default.

FAQ

What is the best single llm eval metric to start with? There isn't one, and picking a single metric is usually the mistake. Start by writing down the specific failure modes you're worried about (wrong facts, off-topic answers, bad tone, hallucinated citations) and pick one metric per failure mode. A stack of three narrow metrics catches more real bugs than one broad score.

Is LLM-as-judge reliable enough to trust in production? It's reliable enough to be useful, not reliable enough to trust blindly. Calibrate every judge prompt against human-labeled examples before shipping it, re-check agreement periodically since judge and target models both drift over time, and design the rubric to counter known biases (position, verbosity, self-preference).

Do I still need BLEU or ROUGE in 2026? Only for narrow cases: fixed-reference translation pipelines, or as a cheap deterministic regression check when you already know an LLM judge agrees with humans on that task and you want a faster secondary signal between full judge runs. They should not be your primary quality metric for open-ended generation.

How is RAG evaluation different from general LLM evaluation? General metrics score the output against a reference or rubric. RAG evaluation adds a second axis entirely: does the output match what was actually retrieved. An answer can score well on relevance and fluency while being completely unfaithful to the retrieved context, which is why faithfulness has to be measured separately, not inferred from other scores.

How do I evaluate an agent that can complete a task multiple valid ways? Use order-insensitive trajectory matching or, more robustly, a task-completion verifier that checks final state (was the file edited correctly, did the API call succeed) rather than the exact sequence of steps. Reserve strict trajectory matching for tasks where the sequence itself is part of the requirement.

What sample size do I need to trust an LLM judge score? For calibrating a judge against humans, 50-100 examples is a reasonable starting point to estimate agreement, more if the task is high-variance. For ongoing regression testing between prompt or model changes, hundreds of examples per run reduce noise enough to trust a directional change, though the exact number depends on how much the underlying score naturally varies on your task.

Can I combine multiple metrics into one composite score? You can, but weight it carefully and keep the individual components visible in your logs. A composite score that blends faithfulness, relevance, and tone into one number is convenient for a dashboard but hides which specific failure mode caused a drop, which is exactly the information you need when a regression shows up.