Ragas Custom Metrics: Extending the Framework for Your Use Case

Why the Built-In Metrics Eventually Run Out

Every team that adopts Ragas goes through the same arc. You start with faithfulness, answer_relevancy, and context_precision, wire them into a CI job, and for a few weeks it feels like you've solved evaluation. Then someone asks a question the built-in metrics were never designed to answer: "does this answer follow our support team's tone guidelines?" or "did the model correctly cite the SKU number instead of hallucinating a similar one?" or "does this legal RAG assistant avoid giving definitive advice on jurisdiction-specific questions?"

None of the stock metrics know about your domain. faithfulness checks whether claims in the answer are grounded in retrieved context, which is generic and useful, but it has no idea that your product requires every answer mentioning a medication to include a dosage disclaimer, or that your internal style guide bans phrases like "as an AI" in customer-facing responses. This is precisely the gap custom metrics are built to close.

Ragas was designed from day one to be extended rather than just consumed. The same abstractions that power Faithfulness and ContextRecall internally are exposed to you as a public API: Metric base classes, LLM-based scoring prompts, rubric-based graders, and plain Python functions wrapped as metrics. This article walks through all of them, with working code, so you can go from "the default metrics don't fit" to "we have a metric that reflects exactly what our product needs" in an afternoon.

By the end you'll know how to build an LLM-as-judge metric from scratch, how to use RubricsScore for graded criteria, how to wrap non-LLM Python logic as a metric, how to test your metrics before trusting them, and how the async execution model affects what you can safely do inside _ascore.

The Metric Base Classes: What You're Actually Extending

Ragas metrics are Python classes that implement a scoring interface. Understanding the class hierarchy is the first step, because which base class you inherit from determines what you need to implement.

At the root, every metric implements Metric, an abstract base with a name attribute and a _ascore (or _single_turn_ascore / _multi_turn_ascore) coroutine that returns a float. Ragas ships several concrete bases you'll typically extend instead of the raw abstract class:

• MetricWithLLM — for metrics that need an LLM to do the judging. Your metric gets an llm attribute (a wrapped BaseRagasLLM) that you call inside your scoring logic.
• MetricWithEmbeddings — for metrics that need embedding similarity (used less often for custom metrics, more common under the hood in things like AnswerSimilarity).
• SingleTurnMetric — the interface for metrics scored over a single SingleTurnSample (question, answer, contexts, ground truth).
• MultiTurnMetric — the interface for metrics scored over a full conversation (MultiTurnSample), useful for agent trajectory evaluation.

In practice, most custom metrics you'll write inherit from both MetricWithLLM and SingleTurnMetric, because most RAG evaluation questions are "given this one question/answer/context tuple, is this true?"

Here's the actual shape of a minimal custom metric:

from dataclasses import dataclass, field
from ragas.metrics.base import MetricWithLLM, SingleTurnMetric
from ragas.dataset_schema import SingleTurnSample
from ragas.prompt import PydanticPrompt
from pydantic import BaseModel

class DisclaimerInput(BaseModel):
    question: str
    answer: str

class DisclaimerOutput(BaseModel):
    contains_disclaimer: bool
    reason: str

class DisclaimerPrompt(PydanticPrompt[DisclaimerInput, DisclaimerOutput]):
    instruction = (
        "Determine if the answer includes an appropriate medical "
        "disclaimer whenever it recommends a dosage, medication, or "
        "treatment. Respond with contains_disclaimer=true only if the "
        "disclaimer is present and specific, not generic boilerplate."
    )
    input_model = DisclaimerInput
    output_model = DisclaimerOutput

@dataclass
class DisclaimerPresence(MetricWithLLM, SingleTurnMetric):
    name: str = "disclaimer_presence"
    _required_columns: dict = field(
        default_factory=lambda: {"SINGLE_TURN": {"user_input", "response"}}
    )

    async def _single_turn_ascore(self, sample: SingleTurnSample, callbacks) -> float:
        prompt_input = DisclaimerInput(
            question=sample.user_input, answer=sample.response
        )
        result = await self.disclaimer_prompt.generate(
            data=prompt_input, llm=self.llm, callbacks=callbacks
        )
        return 1.0 if result.contains_disclaimer else 0.0

    def __post_init__(self):
        self.disclaimer_prompt = DisclaimerPrompt()

This is a full, working custom metric. It declares its required columns, defines a structured prompt with typed input/output via PydanticPrompt, and implements _single_turn_ascore to call the LLM and convert the structured response into a float score. The _required_columns declaration matters — Ragas uses it during evaluate() to validate your dataset has the fields the metric needs before running, so failures show up immediately instead of halfway through a batch.

Building an LLM-Based Custom Metric with `PydanticPrompt`

The example above used PydanticPrompt, which is the backbone of every LLM-graded metric in Ragas (including the built-in ones — Faithfulness itself is essentially a PydanticPrompt for claim decomposition plus a second one for verification). Understanding this pattern well pays off because it's reusable across dozens of custom metrics.

The core idea: instead of hand-writing a prompt string and parsing free text back out, you define Pydantic models for input and output, and PydanticPrompt handles serializing the input into the prompt template and parsing the LLM's JSON response into your output model. This gives you type safety and removes an entire class of bugs around regex-parsing LLM output.

Let's build something more realistic: a metric that scores whether an answer correctly cites a source document by name when the question asks for a source.

from dataclasses import dataclass, field
from pydantic import BaseModel, Field
from ragas.metrics.base import MetricWithLLM, SingleTurnMetric
from ragas.dataset_schema import SingleTurnSample
from ragas.prompt import PydanticPrompt

class CitationInput(BaseModel):
    question: str
    answer: str
    retrieved_contexts: list[str]

class CitationOutput(BaseModel):
    citation_required: bool = Field(
        description="Whether the question asks for a source or citation"
    )
    citation_correct: bool = Field(
        description="Whether the answer cites a document that actually "
        "appears in retrieved_contexts"
    )
    explanation: str

class CitationCorrectnessPrompt(PydanticPrompt[CitationInput, CitationOutput]):
    instruction = (
        "Given a question, an answer, and the retrieved contexts, decide "
        "two things: (1) does the question require the answer to cite a "
        "source document, and (2) if so, does the answer correctly name "
        "a document that is actually present in retrieved_contexts. "
        "Do not reward citations to documents that do not appear in the "
        "provided contexts."
    )
    input_model = CitationInput
    output_model = CitationOutput
    examples = []

@dataclass
class CitationCorrectness(MetricWithLLM, SingleTurnMetric):
    name: str = "citation_correctness"
    _required_columns: dict = field(
        default_factory=lambda: {
            "SINGLE_TURN": {"user_input", "response", "retrieved_contexts"}
        }
    )

    def __post_init__(self):
        self.prompt = CitationCorrectnessPrompt()

    async def _single_turn_ascore(self, sample: SingleTurnSample, callbacks) -> float:
        result = await self.prompt.generate(
            data=CitationInput(
                question=sample.user_input,
                answer=sample.response,
                retrieved_contexts=sample.retrieved_contexts or [],
            ),
            llm=self.llm,
            callbacks=callbacks,
        )
        if not result.citation_required:
            return 1.0  # not applicable, don't penalize
        return 1.0 if result.citation_correct else 0.0

Two details matter here. First, the examples list on the prompt class is where few-shot examples go — leaving it empty works, but for anything nuanced (like judging tone or partial correctness) adding two or three worked examples dramatically improves consistency across runs, exactly the way few-shot examples improve any LLM classification task. Second, notice the "not applicable" branch — a common mistake in custom metrics is forcing every sample into a binary pass/fail even when the criterion doesn't apply, which silently drags down your aggregate score and makes the metric untrustworthy. Decide explicitly how not-applicable cases should score, and document it.

Using `RubricsScore` for Graded, Multi-Level Criteria

Not every judgment is binary. Sometimes you want a 1-to-5 scale for "how helpful is this response," graded against explicit criteria rather than a vague "rate this 1-5" prompt that different LLM calls will interpret inconsistently. Ragas provides RubricsScore for exactly this, and it's less code than building a custom PydanticPrompt from scratch when a rubric is all you need.

from ragas.metrics import RubricsScore

helpfulness_rubrics = {
    "score1_description": (
        "The response is off-topic, factually wrong, or fails to address "
        "the user's question at all."
    ),
    "score2_description": (
        "The response addresses the question but is incomplete or "
        "contains a notable factual error not supported by context."
    ),
    "score3_description": (
        "The response is correct and grounded in context but is generic "
        "or omits details the user would reasonably expect."
    ),
    "score4_description": (
        "The response is correct, grounded, and reasonably complete, "
        "with only minor room for improvement in clarity or detail."
    ),
    "score5_description": (
        "The response is fully correct, grounded in the retrieved "
        "context, complete, and clearly written."
    ),
}

helpfulness_metric = RubricsScore(
    name="helpfulness",
    rubrics=helpfulness_rubrics,
)

RubricsScore takes your rubric descriptions and an LLM, then asks the model to pick the score whose description best matches the sample, rather than asking it to invent a number from an underspecified scale. This is the same technique used for AspectCritic under the hood, and it's worth using any time your evaluation criteria are genuinely graded rather than pass/fail.

You wire it into evaluate() exactly like any other metric, but it needs an LLM assigned first if you didn't pass one at construction:

from ragas import evaluate
from ragas.llms import llm_factory

judge_llm = llm_factory("gpt-4o")
helpfulness_metric.llm = judge_llm

results = evaluate(
    dataset=my_eval_dataset,
    metrics=[helpfulness_metric],
    llm=judge_llm,
)
print(results.to_pandas()[["user_input", "helpfulness"]])

Rubric metrics are especially good for stakeholder-facing scores — "helpfulness," "tone alignment," "completeness" — because the rubric descriptions themselves are readable by non-engineers, and you can put them in front of your product or support team to sign off on before you trust the number.

Wrapping Plain Python Logic: Non-LLM Custom Metrics

Not every custom metric needs an LLM call. If your criterion is deterministic — a regex check, a JSON schema validation, a check against a list of banned phrases, a length constraint — running it through an LLM is slower, costs money, and is less reliable than just writing the check in Python. Ragas supports this cleanly by letting you inherit from SingleTurnMetric without MetricWithLLM at all.

import re
from dataclasses import dataclass, field
from ragas.metrics.base import SingleTurnMetric
from ragas.dataset_schema import SingleTurnSample

BANNED_PHRASES = ["as an ai", "as a language model", "i cannot guarantee"]

@dataclass
class BannedPhraseCheck(SingleTurnMetric):
    name: str = "banned_phrase_free"
    _required_columns: dict = field(
        default_factory=lambda: {"SINGLE_TURN": {"response"}}
    )

    async def _single_turn_ascore(self, sample: SingleTurnSample, callbacks) -> float:
        text = sample.response.lower()
        hit = any(phrase in text for phrase in BANNED_PHRASES)
        return 0.0 if hit else 1.0

    def _score(self, row: dict) -> float:
        # sync fallback some Ragas versions call directly
        text = row["response"].lower()
        hit = any(phrase in text for phrase in BANNED_PHRASES)
        return 0.0 if hit else 1.0

This runs in microseconds per sample, needs zero API calls, and is completely deterministic — the same input always produces the same score, which is a property LLM-judged metrics fundamentally cannot guarantee. A good rule of thumb: if you can write the check as a regex, a length comparison, a JSON schema validator, or a lookup against a fixed list, do it in Python. Reserve the LLM judge for genuinely semantic judgments — factual grounding, tone, relevance — that no amount of string matching can capture.

You can mix deterministic and LLM-based metrics in the same evaluate() call without any special handling:

from ragas import evaluate

results = evaluate(
    dataset=my_eval_dataset,
    metrics=[BannedPhraseCheck(), DisclaimerPresence(), helpfulness_metric],
    llm=judge_llm,
)

Ragas runs each metric independently against every sample and assembles the results into one dataframe, regardless of whether a given metric needed the LLM or not.

Handling Multi-Turn and Agent Trajectory Metrics

If you're evaluating an agent rather than a single-shot RAG answer, SingleTurnMetric isn't the right interface — you need MultiTurnMetric, which is scored against a MultiTurnSample containing a full list of messages (human, AI, tool calls, tool results) instead of a flat question/answer pair.

from dataclasses import dataclass, field
from ragas.metrics.base import MetricWithLLM, MultiTurnMetric
from ragas.dataset_schema import MultiTurnSample

@dataclass
class ToolCallEfficiency(MetricWithLLM, MultiTurnMetric):
    name: str = "tool_call_efficiency"
    _required_columns: dict = field(
        default_factory=lambda: {"MULTI_TURN": {"user_input"}}
    )

    async def _multi_turn_ascore(self, sample: MultiTurnSample, callbacks) -> float:
        tool_calls = [
            msg for msg in sample.user_input if getattr(msg, "type", None) == "tool"
        ]
        # simple heuristic: penalize redundant identical tool calls
        seen = set()
        redundant = 0
        for call in tool_calls:
            key = (getattr(call, "name", None), str(getattr(call, "args", "")))
            if key in seen:
                redundant += 1
            seen.add(key)
        if not tool_calls:
            return 1.0
        return max(0.0, 1.0 - (redundant / len(tool_calls)))

This pattern is what you reach for when evaluating agentic RAG pipelines — a retrieval agent that can call a search tool multiple times, a re-ranker tool, and a final answer-generation step. You're no longer just judging the final answer; you're judging the trajectory that produced it. Custom multi-turn metrics are how you encode things like "did the agent call the same tool twice with identical arguments" or "did the agent ask a clarifying question when the request was ambiguous," which no single-turn metric can see.

Testing Custom Metrics Before You Trust Them

An LLM-judged metric is itself a piece of software with bugs, and the most dangerous bug is a metric that looks reasonable but is silently miscalibrated — always scoring near 1.0 regardless of input, or flipping on prompt wording it shouldn't be sensitive to. Before wiring a new custom metric into CI, build a small labeled test set by hand: five or six examples you're confident should score high, and five or six you're confident should score low or fail.

import asyncio
from ragas.dataset_schema import SingleTurnSample

async def sanity_check(metric, cases):
    for sample, expected in cases:
        score = await metric.single_turn_ascore(sample)
        flag = "OK" if abs(score - expected) < 0.5 else "MISMATCH"
        print(f"{flag}: got {score:.2f}, expected {expected:.2f}")

cases = [
    (
        SingleTurnSample(
            user_input="What's the dosage for ibuprofen?",
            response=(
                "Typical adult dosage is 200-400mg every 4-6 hours. "
                "This is general information, not medical advice — "
                "consult a physician before use."
            ),
        ),
        1.0,
    ),
    (
        SingleTurnSample(
            user_input="What's the dosage for ibuprofen?",
            response="Take 400mg every 4 hours.",
        ),
        0.0,
    ),
]

asyncio.run(sanity_check(DisclaimerPresence(llm=judge_llm), cases))

Run this sanity check every time you change the underlying prompt or swap the judge LLM to a different model or version — LLM judges are not stable across model upgrades, and a metric calibrated against one model's judgment can drift meaningfully when the provider updates the model behind the scenes. Treat your labeled cases as a regression suite for the metric itself, not a one-time validation.

Aggregating Custom Metrics into a Composite Score

Once you have several custom metrics, a common next step is combining them into one composite quality score for dashboards or release gates. Ragas doesn't prescribe a single way to do this, since the right weighting is domain-specific, but a clean pattern is to compute your individual metrics first, then apply your own aggregation function over the resulting dataframe.

from ragas import evaluate

results = evaluate(
    dataset=my_eval_dataset,
    metrics=[DisclaimerPresence(), CitationCorrectness(), BannedPhraseCheck()],
    llm=judge_llm,
)

df = results.to_pandas()

weights = {
    "disclaimer_presence": 0.4,
    "citation_correctness": 0.4,
    "banned_phrase_free": 0.2,
}

df["composite_score"] = sum(df[col] * w for col, w in weights.items())
release_ready = (df["composite_score"] >= 0.85).mean() >= 0.95
print(f"95th-percentile release gate passed: {release_ready}")

This keeps each metric individually interpretable — you can still see exactly why a sample failed — while giving you a single number to gate a deploy or a release checklist on. Resist the temptation to bake the weighting logic into the metric class itself; keeping composition at the dataframe level means you can re-weight for different product surfaces (support bot vs. internal knowledge assistant) without touching the metric implementations.

Common Pitfalls When Writing Custom Metrics

A few mistakes show up repeatedly in custom Ragas metrics, and they're worth naming directly.

• Forgetting `_required_columns`. If you skip this, evaluate() will attempt to run your metric on a dataset missing a field it needs, and you'll get a confusing runtime error deep inside _ascore instead of a clear validation error up front.
• Blocking calls inside `_ascore`. Since Ragas runs metrics concurrently across your dataset using asyncio, a synchronous requests.get() or a blocking file read inside your scoring coroutine will stall the entire event loop and destroy your evaluation throughput. Use httpx.AsyncClient or wrap blocking calls with asyncio.to_thread.
• Not handling missing or empty fields. Real datasets have empty retrieved_contexts lists and empty ground truths more often than test data does. Guard against None and empty lists explicitly rather than assuming every sample is fully populated.
• Conflating "not applicable" with "failed." As shown in the citation example, a metric that can't apply to a given sample should not automatically score it as zero — decide the semantics explicitly and keep it consistent.
• Over-trusting a single LLM judge. For any metric feeding a release gate, consider running it with two different judge models occasionally and checking agreement, the same way you'd want two human reviewers to agree on a rubric before trusting it as ground truth.

Wrapping Up

Custom metrics are where Ragas stops being a generic RAG-evaluation library and starts being your team's actual quality bar. The framework gives you three composable building blocks — PydanticPrompt-based LLM judges for semantic questions, RubricsScore for graded criteria, and plain Python SingleTurnMetric subclasses for deterministic checks — and they all plug into the same evaluate() call alongside the built-in metrics you already use. Start with the narrowest, most deterministic check you can write in Python; reach for an LLM judge only when the judgment genuinely requires semantic understanding; and test every custom metric against hand-labeled cases before it touches a release gate.

If you want to go deeper — building a full evaluation harness, wiring custom metrics into CI, and learning how to calibrate LLM judges against human raters — our Ragas Tutorial course on teachyou.ai walks through all of it hands-on, from your first faithfulness score to production-grade custom metrics running in a real pipeline.