DeepEval Metrics Explained: G-Eval, Faithfulness, Hallucination and More

Every team that ships an LLM feature eventually hits the same wall: the demo works, the vibes are good, and then someone asks "how do you know it's actually correct?" That question doesn't have a vibes-based answer. It needs a metric, a threshold, and a reason to trust the number. DeepEval exists to fill that gap, but it ships with more than a dozen metrics, and picking the wrong one is worse than picking none at all — you'll get a green checkmark on a broken pipeline. This is a metric-by-metric breakdown of the ones you'll actually use: what each one is scoring, when it's the right call, and the misinterpretation that trips up almost every team the first time they wire it in.

Before the breakdown, one framing point. Every metric in DeepEval that uses an LLM to produce a score is doing the same fundamental thing: asking a language model to judge another language model's output against some criteria. That's LLM-as-a-Judge, and it's worth holding onto as you read, because it explains both why these metrics are powerful and why they're never infallible.

G-Eval: your custom rubric, graded by an LLM

G-Eval is the metric you reach for when none of the pre-built metrics match your actual quality bar. Instead of a fixed definition of "good," you write a plain-language rubric — your own criteria — and G-Eval uses chain-of-thought reasoning to have an LLM judge apply that rubric consistently across every test case.

Under the hood, G-Eval doesn't just ask "rate this 1-10." It generates evaluation steps from your criteria, walks through those steps for each output, and then produces a score, usually normalized to a 0-1 range using the token probabilities of the judge's verdict. That extra structure is what separates it from just prompting "is this good?" — you get something closer to a rubric-following grader than a mood ring.

When is G-Eval the right metric? Anytime your quality definition is domain-specific and doesn't map cleanly to "did it hallucinate" or "was it relevant." Tone compliance, adherence to a style guide, whether a summary preserved the original's structure, whether a customer support reply followed a de-escalation policy — these are all G-Eval territory. If you can write the criteria in a sentence a smart human reviewer could apply, G-Eval can operationalize it.

Here's a conceptual example of defining custom criteria:

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

correctness_metric = GEval(
    name="Correctness",
    criteria=(
        "Determine whether the actual output is factually consistent "
        "with the expected output, and does not omit any critical "
        "numeric values or dates mentioned in the expected output."
    ),
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
        LLMTestCaseParams.EXPECTED_OUTPUT,
    ],
    threshold=0.7,
)

The criteria string is doing all the work here — it's your rubric in natural language, and the metric infers evaluation steps from it.

The common misinterpretation: teams treat a high G-Eval score as proof of factual correctness. It isn't. G-Eval is only as good as the criteria you wrote and the context you gave the judge. If your criteria is vague ("is this a good answer?"), you'll get an inconsistent, mushy score that looks precise because it's a number. Write criteria that are specific enough that two different human reviewers would apply them the same way — if they wouldn't agree, neither will the judge.

There's a second, quieter trap with G-Eval: criteria overload. Teams often try to cram five different quality dimensions into one criteria string — tone, factual accuracy, formatting, length, and policy compliance all at once — because it's easier than defining five metrics. Resist that. A single G-Eval metric answering one clear question gives you a diagnosable score. A single G-Eval metric answering five vague questions at once gives you a number that goes down and you have no idea why. If you catch yourself writing "and" three times in a criteria string, that's usually a sign you need three metrics, not one.

It's also worth deciding upfront which evaluation_params your criteria actually needs. If your rubric only cares about tone, you don't need to pass EXPECTED_OUTPUT or RETRIEVAL_CONTEXT — extra parameters the judge doesn't need can dilute the reasoning chain and, in some cases, invite the judge to grade against criteria you never asked for. Keep the inputs as tight as the question you're actually asking.

Faithfulness and Hallucination: does the output match the source

These two metrics get confused constantly because they sound like they're measuring the same thing. They're not, and the difference matters.

Faithfulness checks whether the claims in your output are actually supported by the retrieval context you gave the model. It's a RAG-shaped metric: you retrieved some documents, the model generated an answer, and Faithfulness asks "does every claim in that answer trace back to something in the retrieved context?" It works by extracting individual claims from the output and verifying each one against the context, then scoring the proportion that hold up.

Hallucination, in DeepEval's framing, is closer to the inverse question but with a different reference point — it checks the output against a provided context (which might be the full source document, not just retrieved chunks) and measures how much of the output contradicts or fabricates beyond what's grounded in that context. In practice, teams use Hallucination when they have a full-document ground truth and want to catch fabrication broadly, and Faithfulness specifically when evaluating a RAG pipeline's retrieval-to-generation link.

When is Faithfulness the right metric? Any RAG system where you're worried the generator is "creative" with the retrieved chunks — paraphrasing past the point of accuracy, filling gaps with plausible-sounding but ungrounded detail, or blending two retrieved facts into one wrong one. When is Hallucination the right metric? Anywhere you have a trusted reference document and need a broader fabrication check, including non-RAG generation tasks like summarization or report drafting.

The common misinterpretation: a perfect Faithfulness score does not mean the answer is correct — it means the answer is consistent with what was retrieved. If retrieval pulled the wrong documents, the model can be perfectly faithful to irrelevant or outdated information and still give a useless answer. Faithfulness is a check on the generator, not a check on the retriever. You need Contextual Precision and Contextual Recall for that half of the pipeline, which is exactly the next section.

There's a practical wrinkle worth flagging too: both Faithfulness and Hallucination depend heavily on how you chunked and passed context in the first place. If your retrieval context is bloated with near-duplicate or tangential chunks, claim extraction gets noisier, and the metric's judgment gets noisier with it. A model can also be "faithful" to context that itself contains stale or contradictory information — Faithfulness verifies internal consistency between output and context, not the truth of the context. Garbage in, faithfully summarized garbage out, and the metric will still score it well. If you suspect your source documents themselves are wrong, no generation-side metric will catch that; you need to audit the corpus, not the model.

Contextual Precision and Contextual Recall: judging the retriever, not the generator

This is the pair of metrics people skip because they're RAG-specific and feel like extra plumbing — until a customer gets a wrong answer and the postmortem reveals retrieval was the actual failure point, not generation.

Contextual Precision measures whether the relevant nodes in your retrieved context are ranked higher than the irrelevant ones. It's not asking "did you retrieve the right document," it's asking "among what you retrieved, is the useful stuff near the top?" This matters because most generators are recency- and position-biased — if the relevant chunk is buried at position eight of ten, the model may never use it properly, even though technically it was "in context."

Contextual Recall measures the opposite failure mode: does the retrieved context contain everything in the expected output? This one requires an expected/ground-truth answer to compare against — it checks whether the ideal answer's claims can all be traced back to something in what was retrieved. If your retriever consistently misses a needed fact, Contextual Recall is what surfaces it, even when Faithfulness and Hallucination look fine because the model didn't fabricate anything, it just didn't have what it needed.

When is Contextual Precision the right metric? When you're tuning a reranker or adjusting retrieval top-k and want to confirm the ranking, not just the recall set, actually improved. When is Contextual Recall the right metric? When you're debugging "the model gave an incomplete answer" complaints and need to know if it's a retrieval gap versus a generation gap.

The common misinterpretation: treating these as generator metrics. They are entirely about the retrieval step. A RAG pipeline can score badly on Contextual Precision/Recall while the generator is flawless, and teams sometimes waste a sprint tuning prompts when the actual fix is in the embedding model, chunking strategy, or reranker. Run these two before you touch generation-side metrics — if retrieval is broken, everything downstream is guesswork.

One more nuance worth internalizing: Contextual Precision and Contextual Recall pull in different directions when you tune top-k. Cranking up the number of retrieved chunks tends to help Recall — more surface area means fewer missed facts — but it actively hurts Precision, because you're diluting the ranked list with more irrelevant material for the generator to wade through. There's no universal "right" top-k; the correct setting is the one that keeps both metrics acceptable for your specific corpus and query patterns. If you only ever look at one of the two, you'll optimize yourself into either a retriever that misses facts or one that buries them, and both look fine on whichever single metric you weren't watching.

Answer Relevancy: did the response actually address the question

Answer Relevancy is deceptively simple and often the first metric teams reach for, sometimes to their detriment because it measures less than they assume. It checks whether the output is relevant to the input — does the response actually address what was asked, without padding, tangents, or answering a different question than the one posed. It typically works by extracting statements from the output and scoring what fraction are relevant to the input.

When is Answer Relevancy the right metric? Chatbots and QA systems where "on-topic-ness" is the failure mode you're worried about — the model goes on a tangent, answers a related-but-different question, or buries the actual answer under generic preamble. It's also a good cheap smoke test early in development, before you've built out full RAG evaluation, because it doesn't require retrieval context or an expected output — just input and actual output.

The common misinterpretation: a high Answer Relevancy score is not evidence of correctness. A model can give a perfectly relevant, on-topic, confidently wrong answer and still score well here, because relevancy and accuracy are orthogonal. Answer Relevancy tells you the model is answering the right question. It says nothing about whether the answer is true. Pair it with Faithfulness or a correctness-oriented G-Eval metric if truthfulness is what you actually care about — which, for most production systems, it is.

Teams also sometimes conflate low Answer Relevancy with a prompting problem when it's actually a retrieval problem in disguise. If the retrieved context genuinely doesn't contain anything relevant to the question, a well-behaved model will often hedge, caveat, or partially answer — and that hedging reads as "irrelevant" to the metric even though the model behaved correctly given bad input. Before you rewrite your system prompt to fix a relevancy dip, check what was actually retrieved for those failing test cases. Half the time the fix belongs in the retriever, not the prompt.

Task Completion and Tool Correctness: evaluating agents, not chatbots

Once your system stops being a single input-output text generator and starts calling tools, executing multi-step plans, or acting as an agent, the metrics above stop being sufficient. You need metrics that evaluate the trajectory, not just the final text.

Task Completion looks at the full sequence of steps an agent took — the tool calls, intermediate reasoning, and final output — and asks whether the original goal stated in the input was actually accomplished. This is an outcome-level metric: it doesn't care if the agent took an inefficient path, only whether it landed the task.

Tool Correctness is narrower and more mechanical: given the tools the agent called, were they the right tools, called with the right inputs, in a reasonable order? This one can be deterministic rather than LLM-judged in simpler cases — you can directly compare the tools_called list against an expected list — which makes it faster and cheaper to run at scale than most LLM-graded metrics.

When is Task Completion the right metric? End-to-end agent evaluation — did the coding agent actually fix the bug, did the research agent actually answer the brief, did the booking agent actually complete the reservation. When is Tool Correctness the right metric? Debugging why an agent failed, or regression-testing that a prompt or system change didn't break the agent's tool-selection logic. It's also useful in CI, where you want a fast, cheap check that doesn't need a full LLM judge for every commit.

The common misinterpretation: assuming Tool Correctness implies Task Completion, or vice versa. An agent can call all the right tools in the right order and still fail the task because of a bad final synthesis step — high Tool Correctness, low Task Completion. Just as often, an agent stumbles into the right answer through a wrong or roundabout tool sequence — low Tool Correctness, high Task Completion. Run both, and when they disagree, that disagreement is usually the most useful debugging signal you'll get out of either metric alone.

There's also a scaling argument for keeping Tool Correctness deterministic where you can. LLM-judged metrics cost tokens and latency, and if you're running an agent's regression suite on every pull request, that adds up fast. A deterministic Tool Correctness check — literally diffing the expected tool call sequence against the actual one — costs almost nothing and catches a large share of agent regressions: a renamed tool, a dropped required parameter, a step executed out of order after a refactor. Save the more expensive Task Completion judgment for a smaller, slower-running suite, and let the cheap mechanical check run on every commit.

Bias and Toxicity: safety metrics, not quality metrics

Bias and Toxicity are grouped together because they're both scoring for the presence of something you want at zero, not for degrees of quality. Bias checks whether the output contains prejudiced framing — gender, racial, political, or other demographic skew that shouldn't be there given the input. Toxicity checks for harmful, offensive, or abusive language in the output.

Both work by extracting opinions or statements from the output and scoring each against the relevant definition, similar in mechanism to Answer Relevancy's statement extraction, but judged against a safety rubric instead of a relevance one.

When are these the right metrics? Any customer-facing deployment, especially ones with no human in the loop before the output reaches an end user — support bots, content generation tools, anything moderating or summarizing user-submitted text. They're also worth running in CI as a gate, not just a monitoring dashboard metric, because a regression here is a reputational and possibly legal risk, not a quality nit.

The common misinterpretation: treating a passing Bias or Toxicity score as a general safety clearance. These metrics are narrow. A response can be free of toxic language and demographic bias while still being unsafe in other ways — giving harmful instructions, leaking PII, or violating a policy that has nothing to do with bias or toxicity as DeepEval defines them. Don't let a green light on these two lull you into skipping a dedicated safety or policy-adherence evaluation, whether that's a G-Eval custom criteria metric or a separate red-teaming pass.

It's also worth running Bias and Toxicity on more than just the final output. If your pipeline has intermediate generation steps — a draft that gets refined, a set of retrieved snippets that get summarized before final synthesis — bias introduced early can get diluted or reworded by later steps without actually being removed. Testing only the last-mile output can miss it. For anything high-stakes, sample intermediate stages too, not just the thing that ends up on screen.

Putting the metrics together

No single metric earns you the right to stop testing. A typical RAG pipeline needs Contextual Precision and Recall on the retriever, Faithfulness on the generator's grounding, Answer Relevancy on topicality, and probably a G-Eval custom criteria metric for whatever domain-specific correctness bar you actually care about. A typical agent needs Tool Correctness for the mechanics and Task Completion for the outcome, plus Bias and Toxicity if it's user-facing. Stacking two or three of these per pipeline is normal — treating any one of them as a full test suite is how "the eval passed" and "the product works" quietly stop meaning the same thing.

That gap is also why it's worth staying skeptical of any single number, including these. Every metric described here except Tool Correctness in its deterministic form is, underneath the label, an LLM-as-a-Judge call — a model reading your output and applying a rubric, whether that rubric is DeepEval's built-in definition or your own G-Eval criteria. Judges have their own blind spots, their own leniency biases, and their own failure modes when the criteria is ambiguous. Treat these metrics as strong, cheap-to-run signals that tell you where to look closer, not as a courtroom verdict. Use them the way you'd use a good code linter: trust the pattern it catches, but read the diff yourself before you ship.