Ragas

Ragas Cost Considerations: Managing LLM Calls During Evaluation

Ira Menon · May 8, 2026 · 13 min read

The Evaluation Bill Nobody Budgets For

Teams building RAG systems get very disciplined about production inference costs. They cache aggressively, pick the cheapest model that clears the bar, and monitor token spend on every user query. Then they run a Ragas evaluation sweep over a few hundred test cases, using GPT-4 class models as the judge for every metric, and the bill arrives looking nothing like what they expected.

This is not a bug in Ragas. It is a structural property of LLM-as-judge evaluation that catches people off guard because the mental model is wrong. You are not paying for one inference per test case — you are paying for one inference per metric per test case, and some metrics internally issue several calls just to compute a single score. Multiply that across a real evaluation set, add a few metrics, run it during CI on every pull request, and the "evaluation" line item can rival the actual application's LLM spend.

This article walks through where the calls actually go, how to estimate cost before you run anything, and the concrete levers — model selection, metric selection, sampling, and caching — that bring evaluation spend under control without gutting the signal you're trying to get out of Ragas in the first place.

Why Ragas Multiplies LLM Calls

Ragas metrics fall into two rough categories: statistical/embedding-based (cheap) and LLM-based (expensive). Most of the metrics people actually care about — faithfulness, answer relevancy, context precision, context recall, answer correctness — are LLM-based, meaning they use an LLM to decompose, judge, or score the components of a RAG output.

Take faithfulness as an example. Under the hood, it typically does not make one call. It first prompts the LLM to break the generated answer down into a list of discrete claims or statements. Then it makes a second call (or a batch of calls) asking the LLM to verify each of those claims against the retrieved context. If your answer has six factual claims, you might be looking at one decomposition call plus a verification pass that touches all six claims, sometimes as separate calls depending on the implementation and prompt strategy.

Now stack metrics. A typical evaluation run for a RAG pipeline uses four or five metrics: faithfulness, answer relevancy, context precision, context recall, and maybe answer correctness. Each of those has its own call pattern:

Faithfulness: claim extraction call + claim verification call(s)
Answer relevancy: generates several synthetic questions from the answer, then embeds and compares them to the original question — LLM calls for question generation, embedding calls for similarity
Context precision: judges each retrieved chunk's relevance to the question, often one call per chunk or a batched call across chunks
Context recall: breaks the reference answer into statements and checks each against retrieved context, similar shape to faithfulness
Answer correctness: combines a factual overlap check with a semantic similarity check, itself invoking sub-metrics

Do the arithmetic on a modest evaluation set. Say you have 300 test samples, 5 metrics, and each metric averages 2 LLM calls per sample (a conservative estimate given claim-level decomposition). That is 300 × 5 × 2 = 3,000 LLM calls for a single evaluation run. Run that nightly, or on every PR in CI, and the calls compound fast — not because Ragas is inefficient, but because rigorous LLM-as-judge scoring inherently requires multiple reasoning steps per sample.

Estimating Cost Before You Run Anything

The single highest-leverage habit is estimating cost before kicking off a full run, especially the first time you evaluate a new dataset size or metric combination. You don't need a perfect model — you need an order-of-magnitude sanity check that stops you from accidentally burning budget on a typo'd loop or an evaluation set that's ten times larger than you thought.

A simple back-of-envelope approach:

# Rough cost estimator for a Ragas evaluation run
# Numbers are illustrative placeholders — plug in your actual
# provider's per-token pricing before trusting this.

def estimate_ragas_cost(
    num_samples: int,
    metrics: list[str],
    avg_calls_per_metric: float,
    avg_input_tokens_per_call: int,
    avg_output_tokens_per_call: int,
    input_price_per_1k: float,
    output_price_per_1k: float,
) -> dict:
    total_calls = num_samples * len(metrics) * avg_calls_per_metric
    total_input_tokens = total_calls * avg_input_tokens_per_call
    total_output_tokens = total_calls * avg_output_tokens_per_call

    input_cost = (total_input_tokens / 1000) * input_price_per_1k
    output_cost = (total_output_tokens / 1000) * output_price_per_1k

    return {
        "total_calls": total_calls,
        "total_input_tokens": total_input_tokens,
        "total_output_tokens": total_output_tokens,
        "estimated_cost_usd": round(input_cost + output_cost, 2),
    }


estimate = estimate_ragas_cost(
    num_samples=300,
    metrics=["faithfulness", "answer_relevancy", "context_precision", "context_recall"],
    avg_calls_per_metric=2,
    avg_input_tokens_per_call=800,
    avg_output_tokens_per_call=150,
    input_price_per_1k=0.003,
    output_price_per_1k=0.015,
)

print(estimate)

Run this before every meaningful change to your eval set size, metric list, or context length — context length in particular is easy to forget, because faithfulness and context precision both feed the retrieved chunks into the prompt, and long chunks inflate the input token count far more than people expect.

The point of this exercise isn't precision — it's catching the "wait, this is 3x bigger than I thought" moment before it shows up on an invoice, not after.

Pick the Right Judge Model for Each Metric

The default instinct is to use your most capable model as the judge for everything, on the theory that a stronger judge means more trustworthy scores. That's true up to a point, but it ignores that different metrics demand different amounts of reasoning.

Context precision — judging whether a retrieved chunk is relevant to a question — is a relatively simple classification task. A smaller, cheaper model can do this reliably. Faithfulness and answer correctness, which require multi-step claim decomposition and nuanced comparison, benefit more from a stronger reasoning model.

Ragas supports configuring a different LLM per metric, which means you don't have to run every metric through the same (expensive) model:

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

# Strong model reserved for metrics that need deeper reasoning
strong_judge = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))

# Cheaper model for simpler classification-style judgments
light_judge = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))

faithfulness.llm = strong_judge
answer_relevancy.llm = light_judge
context_precision.llm = light_judge
context_recall.llm = strong_judge

results = evaluate(
    dataset=my_eval_dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)

This single change — routing the "easy" metrics to a smaller model and reserving the expensive model for the metrics that actually need it — routinely cuts evaluation spend by more than half, because context precision and answer relevancy tend to dominate call volume (one judgment per retrieved chunk, one per generated question) while faithfulness and correctness are comparatively call-light per sample.

Validate this tradeoff once, deliberately: run a subset of your eval set through both the cheap and expensive judge for a given metric, and check whether the score distributions and pass/fail decisions actually diverge. If they don't, you've confirmed the cheaper model is safe for that metric going forward. If they do diverge meaningfully, keep the stronger model there and save the substitution for metrics where it held up.

Don't Evaluate Every Metric on Every Sample

Not every metric needs to run on every single test case. This sounds obvious once stated, but most teams default to running the full metric suite across the entire dataset every time, because that's what the quickstart tutorial shows.

A more deliberate structure looks like this:

Smoke tier — a small, fast, cheap subset (20-30 samples) run on every commit or PR, using only your one or two most important metrics (usually faithfulness, since ungrounded answers are the highest-severity failure mode for RAG).
Regression tier — the full metric suite run on a medium-sized representative sample (100-200 samples), triggered on merges to your main branch, not every commit.
Full evaluation tier — the complete dataset with the complete metric suite, run on a schedule (nightly or weekly) or before a release, when cost is justified by the decision it informs.

This tiered structure means the expensive, comprehensive run happens only when the stakes justify it, while fast feedback during development stays cheap. It mirrors how most teams already think about unit tests versus full integration suites — the mistake is treating every Ragas run as if it needs to be the integration suite.

import os

def get_eval_tier() -> str:
    """Decide which tier of evaluation to run based on CI context."""
    if os.getenv("CI_EVENT") == "pull_request":
        return "smoke"
    if os.getenv("CI_EVENT") == "merge_to_main":
        return "regression"
    return "full"


TIER_CONFIG = {
    "smoke": {"sample_size": 25, "metrics": ["faithfulness"]},
    "regression": {
        "sample_size": 150,
        "metrics": ["faithfulness", "answer_relevancy", "context_precision"],
    },
    "full": {
        "sample_size": None,  # entire dataset
        "metrics": [
            "faithfulness",
            "answer_relevancy",
            "context_precision",
            "context_recall",
            "answer_correctness",
        ],
    },
}

tier = get_eval_tier()
config = TIER_CONFIG[tier]
print(f"Running {tier} tier: {config}")

Sample Smartly Instead of Evaluating Everything

If your evaluation dataset has grown to thousands of examples, evaluating all of them on every run is often unnecessary. A well-chosen random sample, refreshed periodically, gives you a statistically reasonable read on system quality at a fraction of the cost.

The trick is doing this deliberately rather than just lopping off the first N rows, which tends to introduce bias if your dataset was built or ordered in any structured way (e.g., grouped by topic, or appended chronologically as new failure cases were added).

import random

def stratified_sample(dataset, sample_size, category_key="category", seed=42):
    """Sample proportionally across categories so rare-but-important
    slices (e.g. multi-hop questions, edge cases) aren't starved out
    by a naive random sample."""
    random.seed(seed)
    categories = {}
    for row in dataset:
        categories.setdefault(row[category_key], []).append(row)

    total = len(dataset)
    sampled = []
    for category, rows in categories.items():
        proportion = len(rows) / total
        n = max(1, round(sample_size * proportion))
        sampled.extend(random.sample(rows, min(n, len(rows))))

    return sampled

Stratifying by category (question type, difficulty, source document, whatever taxonomy your test set already has) ensures that when you cut your sample size to save money, you're not accidentally dropping coverage of the hard cases that actually reveal regressions. A pure random sample from an imbalanced dataset can silently stop testing your edge cases altogether.

Re-run the full dataset periodically — weekly or before major releases — to catch anything the sampled runs might have missed, but treat the sampled run as the default for day-to-day iteration.

Cache Aggressively, Especially During Development

A huge and easily avoidable source of wasted spend is re-scoring the same input twice. During development, it's common to run an evaluation, tweak an unrelated part of the pipeline, and re-run the entire suite — including samples whose retrieval and generation output didn't actually change.

Caching the LLM judge calls keyed on the actual inputs (question, answer, context, and metric) means you only pay for genuinely new judgments:

import hashlib
import json
import os

CACHE_DIR = ".ragas_cache"
os.makedirs(CACHE_DIR, exist_ok=True)


def cache_key(metric_name: str, question: str, answer: str, contexts: list[str]) -> str:
    payload = json.dumps(
        {"metric": metric_name, "question": question, "answer": answer, "contexts": contexts},
        sort_keys=True,
    )
    return hashlib.sha256(payload.encode()).hexdigest()


def get_cached_score(metric_name, question, answer, contexts):
    key = cache_key(metric_name, question, answer, contexts)
    path = os.path.join(CACHE_DIR, f"{key}.json")
    if os.path.exists(path):
        with open(path) as f:
            return json.load(f)["score"]
    return None


def set_cached_score(metric_name, question, answer, contexts, score):
    key = cache_key(metric_name, question, answer, contexts)
    path = os.path.join(CACHE_DIR, f"{key}.json")
    with open(path, "w") as f:
        json.dump({"score": score}, f)

Wrap your metric evaluation call with a check against this cache before invoking the judge LLM. In practice, this matters most in two scenarios: iterative debugging (where you re-run the same failing sample repeatedly while fixing a prompt) and CI, where unchanged test cases across commits don't need to be re-judged if neither the pipeline output nor the reference data has changed. Some teams key the cache on a hash of the pipeline version too, so a genuine pipeline change correctly invalidates the cache while an unrelated code change doesn't.

Watch Context Length, Not Just Call Count

It's tempting to focus entirely on the number of calls, but input token count matters just as much for cost, and it's easy to overlook. Context precision and faithfulness both feed retrieved chunks directly into the judge prompt. If your retriever returns five chunks of 500 tokens each, that's 2,500 tokens of context added to every single judgment call for that sample — before you've counted the question, the answer, or the prompt template's own instructions.

A few practical levers here:

Truncate or summarize long contexts before passing them to the judge if your retrieval chunks are unusually large, as long as truncation doesn't strip the exact passage the answer depends on.
Reduce top-k for evaluation runs if your production system retrieves more chunks than are actually needed to judge groundedness — you can evaluate against a slightly narrower context window than what production serves, as a cost/fidelity tradeoff, and validate periodically that the narrower window doesn't change your conclusions.
Batch chunk-level judgments where the Ragas implementation supports it, rather than issuing one call per chunk — check the specific metric's configuration options for a batching or grouping parameter.

None of these are exotic techniques. They're the same token-hygiene practices you'd apply to production prompts, just redirected at your evaluation pipeline, which is easy to forget because it doesn't feel like "real" traffic.

Building a Cost-Aware Evaluation Habit

The underlying fix here isn't a single trick — it's treating evaluation cost as a first-class constraint you design around, the same way you already do for production inference. A few habits make this durable:

Log actual token usage per evaluation run, not just the final scores, so you have real data instead of guesses when someone asks why the bill went up.
Set a budget per tier (smoke, regression, full) and alert when a run exceeds it — this catches accidental scope creep, like someone adding a sixth metric without realizing what it does to total calls.
Review your metric list quarterly. Metrics get added during a specific investigation and then never removed, quietly taxing every future run for a question you stopped needing to ask.
Prefer fewer, well-chosen metrics over a maximal suite. Faithfulness and context precision catch the majority of RAG failure modes; answer correctness and context recall add real value but at real cost, and are often better suited to the periodic full-tier run than every commit.

None of this requires exotic infrastructure. It requires treating your evaluation harness with the same cost discipline you'd apply to any other LLM-calling system in production — because functionally, that's exactly what it is.

Where This Fits in Your Broader Evaluation Strategy

Cost control isn't a reason to skip rigorous evaluation — it's what makes rigorous evaluation sustainable enough that you actually keep running it. A comprehensive eval suite that gets disabled after the first surprising bill teaches you nothing. A tiered, cost-aware setup that runs reliably on every PR, every merge, and every release gives you a durable signal you can trust over the life of the project.

If you're setting up Ragas for the first time, or find your evaluation costs have crept past what you budgeted, the underlying skill is the same one that makes any evaluation pipeline useful: understanding what each metric actually costs to compute, and matching that cost to how much the decision it informs actually matters.

If you want a structured, hands-on walkthrough of setting up Ragas correctly from scratch — including metric selection, judge model configuration, and building the kind of tiered evaluation pipeline described here — the Ragas Tutorial course on teachyou.ai covers exactly this, with working code you can adapt directly to your own RAG pipeline.