DeepEval Custom Metrics: Writing Your Own Evaluation Logic

Why the built-in metrics run out fast

You install DeepEval, run AnswerRelevancyMetric and FaithfulnessMetric against your RAG pipeline, and for the first week it feels like magic. Then reality sets in. Your product manager wants to know if the chatbot's tone matches brand voice. Your compliance team wants a hard fail if the model recommends a competitor's product. Your support team wants to flag responses that give medical advice without a disclaimer. None of the fourteen or so built-in metrics in DeepEval's library cover any of that, because they can't — those are business rules, not generic NLP properties like relevancy or hallucination.

This is the point where most teams either give up on systematic evaluation and go back to eyeballing outputs, or they start writing their own scoring functions from scratch, reinventing test case management, threshold logic, and reporting along the way. Neither is necessary. DeepEval was built with exactly this gap in mind, and it exposes two clean paths for encoding custom judgment: GEval, a natural-language rubric you hand to an LLM judge, and BaseMetric, a fully programmatic metric class you write in Python. This article walks through both, with working code, so you can start scoring your application on the criteria that actually matter to your product instead of the criteria a benchmark paper happened to define.

The two flavors of custom metric

Before writing any code, it helps to understand what problem each approach solves, because picking the wrong one wastes time.

GEval is for criteria that are fundamentally judgment calls — things a human reviewer would need to read and reason about to score. "Is this response appropriately empathetic?" "Does this summary preserve the original's key numbers?" "Is the tone consistent with a senior engineer explaining something to a junior one?" These are hard to express as code but easy to express as a rubric, so GEval lets you describe the rubric in plain English and DeepEval handles turning that into a structured LLM-judge call, complete with chain-of-thought reasoning and a normalized score.

BaseMetric is for criteria that are deterministic or semi-deterministic — things you can actually compute. "Does the response contain a valid JSON object?" "Is the output under 200 tokens?" "Does the response avoid any of these ten banned phrases?" "Is the extracted phone number formatted as E.164?" You could technically ask an LLM to judge these too, but that's slower, costs money, and is less reliable than just writing the check. BaseMetric also lets you blend both — call an LLM inside your custom logic and combine that with deterministic checks in the same metric.

A third pattern worth knowing is DAG (Deep Acyclic Graph) metrics, which let you chain multiple sub-checks with conditional branching — useful when a "correct" answer depends on which category the input falls into. We'll touch on this at the end, but GEval and BaseMetric cover the vast majority of real-world needs.

Setting up your environment

Everything below assumes you have DeepEval installed and an LLM provider configured for judge calls.

pip install deepeval
export OPENAI_API_KEY="your-api-key-here"

DeepEval defaults to OpenAI models as the judge LLM, but you can swap in Anthropic, Azure, or a local model by passing a custom model argument to any metric — we'll show that later. Every metric in DeepEval, built-in or custom, operates on a LLMTestCase object, so let's define one we'll reuse throughout this article.

from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="What's your return policy on electronics?",
    actual_output=(
        "You can return electronics within 30 days of purchase, "
        "as long as the original packaging is intact and you have "
        "the receipt. Refunds are processed within 5-7 business days."
    ),
    expected_output=(
        "Electronics can be returned within 30 days with original "
        "packaging and proof of purchase."
    ),
    context=["Return window: 30 days. Requires original packaging and receipt."],
)

This is the same object you'd pass to AnswerRelevancyMetric or FaithfulnessMetric, and it's exactly what your custom metrics will consume too. That consistency is the whole point — custom metrics aren't a separate system, they're first-class citizens in the same evaluation pipeline.

Building your first GEval metric

Let's say you're building a customer support bot and you care about a very specific thing: does the response sound helpful without being condescending? That's subjective, hard to codify, but easy to describe. Here's a GEval metric for it.

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

tone_metric = GEval(
    name="Helpful Tone",
    criteria=(
        "Determine whether the actual output is helpful and respectful "
        "in tone, without being condescending, dismissive, or overly "
        "apologetic. A good response answers directly and treats the "
        "user as capable of understanding a normal explanation."
    ),
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
    ],
    threshold=0.7,
)

tone_metric.measure(test_case)
print(tone_metric.score)
print(tone_metric.reason)

A few details matter here. The criteria field is the actual rubric — write it the way you'd brief a human reviewer, not the way you'd write a regex. The evaluation_params list tells DeepEval which fields of your LLMTestCase the judge is allowed to look at; if you don't include LLMTestCaseParams.EXPECTED_OUTPUT, the judge won't see it, which is useful when you want to score tone independent of correctness. The threshold is the pass/fail cutoff on the 0-to-1 score DeepEval returns, and tone_metric.reason gives you the judge's actual chain-of-thought explanation for the score, which is invaluable when you're debugging why a metric passed or failed a case you disagreed with.

You can also use evaluation_steps instead of criteria when you want tighter control over exactly how the judge reasons, rather than letting DeepEval auto-generate steps from your criteria:

strict_tone_metric = GEval(
    name="Helpful Tone (Strict)",
    evaluation_steps=[
        "Check if the response directly answers the question asked",
        "Check if the response uses any dismissive phrases like 'obviously' or 'as I said'",
        "Check if the response over-apologizes with phrases like 'I'm so sorry' more than once",
        "Penalize heavily if the response answers a different question than what was asked",
    ],
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
    ],
    threshold=0.8,
    model="gpt-4o",
)

Notice the model argument — every DeepEval metric accepts this, so you can point tone checks at a cheaper, faster model while reserving your most capable model for something like faithfulness checking where judge quality matters more.

Writing a fully custom metric with BaseMetric

GEval is great until you need something deterministic, or you need to combine an LLM call with actual computation. That's what BaseMetric is for. Every custom metric class needs to implement measure, a_measure (the async version), is_successful, and expose a name property.

Here's a metric that checks whether a response correctly avoids mentioning competitor names — a hard compliance rule, not a judgment call.

from deepeval.metrics import BaseMetric
from deepeval.test_case import LLMTestCase

class NoCompetitorMentionMetric(BaseMetric):
    def __init__(self, competitors: list[str], threshold: float = 1.0):
        self.competitors = [c.lower() for c in competitors]
        self.threshold = threshold
        self.score = None
        self.reason = None
        self.success = None

    def measure(self, test_case: LLMTestCase) -> float:
        output_lower = test_case.actual_output.lower()
        found = [c for c in self.competitors if c in output_lower]

        if found:
            self.score = 0.0
            self.reason = f"Mentioned competitor(s): {', '.join(found)}"
        else:
            self.score = 1.0
            self.reason = "No competitor names found in output."

        self.success = self.score >= self.threshold
        return self.score

    async def a_measure(self, test_case: LLMTestCase) -> float:
        return self.measure(test_case)

    def is_successful(self) -> bool:
        return self.success

    @property
    def __name__(self):
        return "No Competitor Mention"

Using it is identical to using any built-in metric:

from deepeval import evaluate

metric = NoCompetitorMentionMetric(competitors=["Zendesk", "Intercom", "Freshdesk"])
evaluate(test_cases=[test_case], metrics=[metric])

Because a_measure just wraps measure here, this metric runs synchronously even in async batch evaluation, which is fine for pure string checks. If your custom logic calls an external API or an LLM, implement a_measure properly with async/await so DeepEval can run your metric concurrently across a large test suite instead of serializing everything — that difference matters a lot once your regression suite has a few hundred cases.

Combining computation and LLM judgment in one metric

The most useful custom metrics often blend both worlds — a deterministic pre-check that short-circuits obviously bad outputs, and an LLM call for the cases that need real judgment. Here's a metric for a support bot that must always include a case number in a specific format, and separately must sound apologetic when the sentiment of the query is negative.

import re
from deepeval.metrics import BaseMetric
from deepeval.models import DeepEvalBaseLLM
from deepeval.test_case import LLMTestCase

class CaseNumberFormatMetric(BaseMetric):
    """Deterministic check: response must include a case number like CASE-123456."""

    CASE_PATTERN = re.compile(r"CASE-\d{6}")

    def __init__(self, threshold: float = 1.0):
        self.threshold = threshold

    def measure(self, test_case: LLMTestCase) -> float:
        match = self.CASE_PATTERN.search(test_case.actual_output)
        self.score = 1.0 if match else 0.0
        self.reason = (
            f"Found case number: {match.group()}"
            if match
            else "No valid case number (format CASE-XXXXXX) found in output."
        )
        self.success = self.score >= self.threshold
        return self.score

    async def a_measure(self, test_case: LLMTestCase) -> float:
        return self.measure(test_case)

    def is_successful(self) -> bool:
        return self.success

    @property
    def __name__(self):
        return "Case Number Format"

Run several custom metrics together with evaluate, exactly the way you'd mix in built-in ones:

from deepeval import evaluate
from deepeval.metrics import GEval, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCaseParams, LLMTestCase

support_case = LLMTestCase(
    input="I've been charged twice for my subscription, please help.",
    actual_output=(
        "I'm very sorry about the duplicate charge — that's frustrating. "
        "I've opened CASE-482913 for you and refunded the extra charge, "
        "which should appear in 3-5 business days."
    ),
)

results = evaluate(
    test_cases=[support_case],
    metrics=[
        CaseNumberFormatMetric(),
        AnswerRelevancyMetric(threshold=0.7),
        GEval(
            name="Apologetic Tone for Complaints",
            criteria="Check that the response acknowledges the customer's frustration with an apology before offering a resolution.",
            evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
            threshold=0.7,
        ),
    ],
)

This is where DeepEval's design pays off: your compliance rule (case number format), a generic quality metric (relevancy), and your subjective brand-voice rule (apologetic tone) all run through the same evaluate call, produce comparable pass/fail results, and show up in the same test report.

Handling multi-turn and conversational metrics

If you're evaluating a chatbot rather than single-shot Q&A, you'll eventually need metrics that look at the whole conversation instead of one exchange. DeepEval supports this through ConversationalTestCase, and you can write custom conversational metrics by extending BaseConversationalMetric instead of BaseMetric.

from deepeval.metrics import BaseConversationalMetric
from deepeval.test_case import ConversationalTestCase

class NoRepeatedQuestionMetric(BaseConversationalMetric):
    """Fails if the assistant asks the user the same clarifying question twice."""

    def __init__(self, threshold: float = 1.0):
        self.threshold = threshold

    def measure(self, test_case: ConversationalTestCase) -> float:
        assistant_turns = [
            turn.content.strip().lower()
            for turn in test_case.turns
            if turn.role == "assistant" and "?" in turn.content
        ]
        has_duplicate = len(assistant_turns) != len(set(assistant_turns))

        self.score = 0.0 if has_duplicate else 1.0
        self.reason = (
            "Assistant repeated an identical question across turns."
            if has_duplicate
            else "No repeated questions detected."
        )
        self.success = self.score >= self.threshold
        return self.score

    async def a_measure(self, test_case: ConversationalTestCase) -> float:
        return self.measure(test_case)

    def is_successful(self) -> bool:
        return self.success

    @property
    def __name__(self):
        return "No Repeated Question"

This kind of check is nearly impossible to express as a single-turn metric, because the failure only exists across turns. Writing it as plain Python over the turns list is far more reliable than asking an LLM judge to "remember" earlier turns and compare them.

Wiring custom metrics into CI with assert and pytest

Once you trust a custom metric, the next step is making it a gate rather than a report. DeepEval integrates with pytest through assert_test, so you can fail a build the same way you'd fail any other test.

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase

from my_metrics import NoCompetitorMentionMetric, CaseNumberFormatMetric


def test_support_bot_avoids_competitor_mentions():
    test_case = LLMTestCase(
        input="Is your product better than the other guys?",
        actual_output="Our platform focuses on fast onboarding and 24/7 support.",
    )
    assert_test(
        test_case,
        [NoCompetitorMentionMetric(competitors=["Zendesk", "Intercom"])],
    )


def test_support_bot_always_generates_case_number():
    test_case = LLMTestCase(
        input="My order never arrived.",
        actual_output="I've created CASE-102938 to track your missing order.",
    )
    assert_test(test_case, [CaseNumberFormatMetric()])

Run it like any other suite:

deepeval test run test_support_bot.py

deepeval test run is a thin wrapper around pytest that adds DeepEval's reporting — a per-metric breakdown, the reason string from each metric, and (if you've set up an account) a synced dashboard. Because your custom metrics implement the same interface as built-ins, they get the same treatment: readable failure messages showing exactly which metric failed, its score, its threshold, and the reason your metric produced. That last part is why it's worth always populating self.reason with something specific — a failing CI run that just says "score 0.0 < threshold 1.0" tells you nothing, but one that says "Mentioned competitor(s): zendesk" tells you exactly what to fix.

Common pitfalls when writing custom metrics

A few mistakes show up repeatedly when teams start writing their own metrics, and they're worth calling out directly.

• Forgetting `is_successful()` reads from a computed `self.success`. DeepEval calls is_successful() after measure(), not instead of it — if you don't set self.success inside measure, your metric will crash or silently report the wrong pass/fail state.
• Leaving `a_measure` as a stub that raises `NotImplementedError`. This works fine until someone calls evaluate() with run_async=True (the default), at which point every one of your custom metrics needs a working async path.
• Using GEval for things that are actually deterministic. If you can write the check in five lines of Python, do that instead of paying for an LLM judge call — it's faster, cheaper, and won't have judge-to-judge variance across runs.
• Vague `criteria` strings in GEval. "Check if the response is good" produces inconsistent scores because the judge has to invent its own rubric. Be as specific as you'd be briefing a new hire: name the exact behaviors that should raise or lower the score.
• Not pinning the judge model. If you don't explicitly pass model= to a GEval metric, you're relying on DeepEval's default, which can change between library versions. Pin it explicitly for any metric you plan to trend over time, so a score drop in your dashboard reflects your app getting worse, not your judge model changing.
• Skipping DAG metrics when you actually need branching logic. If "correct" depends on the input category (a refund request needs a case number, a general question doesn't), don't try to cram that into one GEval criteria string — use DAGMetric to route test cases through different sub-metrics based on a decision node.

Bringing it together

Custom metrics are what turn DeepEval from "a library that scores RAG pipelines" into "the evaluation layer for whatever you're actually building." GEval covers the fuzzy, judgment-based criteria your product and support teams care about, expressed as a rubric instead of code. BaseMetric covers the deterministic rules — formats, banned phrases, structural checks — that should never depend on an LLM's mood that day. And because both plug into the exact same evaluate(), assert_test(), and CI workflow as DeepEval's built-in metrics, you're not maintaining a second evaluation system — you're extending the one you already have.

Start small: pick one rule your team currently checks by hand during code review or QA, and encode it as a custom metric this week. Once it's running in CI, you'll find the next one, and the one after that, until "does this response meet our bar" stops being a question anyone has to answer by re-reading transcripts.

If you want a structured, hands-on path through this — building GEval and BaseMetric classes for a real multi-agent project, wiring them into pytest and CI, and learning the DAG metric patterns for branching evaluation logic — check out the DeepEval Tutorial course on teachyou.ai. It walks through the exact patterns in this article with a working codebase you extend module by module.