Evaluating RAG vs Evaluating Agents: Different Metrics, Different Rules

The eval you built for RAG will lie to you about your agent

A team we worked with had a solid RAG eval pipeline. Retrieval precision, faithfulness scoring, answer relevance — the works. It caught regressions before they hit production. Then they wrapped that same pipeline around a new agent that could call five tools, write to a database, and retry on failure. The eval said everything was fine. Production said otherwise: the agent was looping on a broken tool call, silently swallowing an error, and returning a confident-sounding answer built from stale data.

Nothing in the RAG eval was wrong. It just wasn't measuring the thing that broke. This is the trap: RAG and agents both "answer questions with an LLM," so it's tempting to reuse the same eval harness for both. But a RAG pipeline is fundamentally a retrieve-then-generate function — one hop, mostly deterministic shape, a single point of failure between "did we find the right documents" and "did we say true things about them." An agent is a control-flow problem with an LLM inside it — multiple hops, branching decisions, external side effects, and failure modes that only exist because the system can take actions, not just produce text.

This article is about where those two evaluation problems diverge, why the metrics that work beautifully for one are close to useless for the other, and what a working eval stack looks like for each. We'll use concrete metrics, a runnable example, and enough detail that you can build this today rather than just nod along.

Why RAG eval is fundamentally a retrieval problem wearing a generation costume

Every RAG failure traces back to one of three places: the retriever pulled the wrong chunks, the generator ignored the right chunks, or the generator said something the chunks don't support. That's it. There's no branching, no tool selection, no multi-step planning to go wrong. This narrow failure surface is why RAG evaluation has converged on a fairly standard set of metrics.

Retrieval metrics answer: did we find the right stuff?

• Context precision — of the chunks retrieved, how many were actually relevant to the query
• Context recall — of the chunks that exist and are relevant, how many did we actually retrieve
• Hit rate / MRR (mean reciprocal rank) — did a relevant chunk show up at all, and how high did it rank

Generation metrics answer: given what we retrieved, did the model say something true and useful?

• Faithfulness (a.k.a. groundedness) — is every claim in the answer traceable back to the retrieved context, or did the model add things from its own memory
• Answer relevance — does the answer actually address the question asked, independent of whether it's grounded
• Answer correctness — when you have a reference answer, does the generated answer match it semantically

Here's a minimal but real faithfulness check using an LLM-as-a-judge pattern, which is the practical way most teams implement this metric (there is no closed-form formula for "is this grounded"):

import json
from openai import OpenAI

client = OpenAI()

FAITHFULNESS_PROMPT = """You are evaluating whether an answer is faithful to the
given context. Break the answer into individual factual claims, then decide for
each claim whether it is directly supported by the context.

Context:
{context}

Answer:
{answer}

Return JSON: {{"claims": [{{"claim": str, "supported": bool}}]}}
"""

def score_faithfulness(context: str, answer: str) -> float:
    resp = client.chat.completions.create(
        model="gpt-4.1",
        messages=[{"role": "user", "content": FAITHFULNESS_PROMPT.format(
            context=context, answer=answer
        )}],
        response_format={"type": "json_object"},
    )
    result = json.loads(resp.choices[0].message.content)
    claims = result["claims"]
    if not claims:
        return 1.0
    supported = sum(1 for c in claims if c["supported"])
    return supported / len(claims)

This is a single-hop measurement. You run it once per query-answer pair, it's cheap, and it's deterministic enough (with temperature 0 and a fixed rubric) to trend over time in a dashboard. RAG eval is, structurally, a batch job: run N queries through the pipeline, score each output independently, average the scores, done.

Why agent eval breaks that model entirely

An agent doesn't produce one output from one input. It produces a *trajectory* — a sequence of decisions, tool calls, intermediate observations, and (often) multiple LLM calls before anything reaches the user. Evaluating only the final answer is like grading a road trip purely on whether the car eventually arrived, ignoring that it took four wrong exits, ran out of gas twice, and got there three hours late by accident.

This changes what you need to measure:

• Was the plan reasonable given the goal, not just did the final output look plausible
• Did the agent pick the right tool at each decision point, and did it pass the right arguments
• Did it recover from a failed tool call, or did it silently give up, hallucinate a result, or loop
• How many steps and tokens did it take — an agent that gets the right answer in 14 steps when 3 would do is a cost and latency bug, not a correctness win
• Did it stop when it should have stopped — asking for missing permissions, refusing an unsafe action, or admitting it couldn't complete the task, instead of confabulating a success

None of this shows up if you only score the final text output. You need trajectory-level evaluation, which means capturing and inspecting the full sequence of steps, not just the last message.

Here's what that looks like in practice — evaluating a single agent run by walking its trajectory:

from dataclasses import dataclass

@dataclass
class Step:
    tool_name: str
    tool_input: dict
    tool_output: str
    reasoning: str  # the agent's stated reasoning before the call

def evaluate_trajectory(steps: list[Step], goal: str, final_answer: str) -> dict:
    issues = []

    # 1. Tool selection: was each chosen tool appropriate for its stated reasoning?
    for i, step in enumerate(steps):
        if step.tool_name == "search_docs" and "calculate" in step.reasoning.lower():
            issues.append(f"step {i}: called search_docs for a math task")

    # 2. Redundant work: did the agent call the same tool with the same args twice?
    seen = set()
    for i, step in enumerate(steps):
        key = (step.tool_name, str(step.tool_input))
        if key in seen:
            issues.append(f"step {i}: repeated identical tool call, likely a loop")
        seen.add(key)

    # 3. Silent failure handling: did a tool error get surfaced or swallowed?
    for i, step in enumerate(steps):
        if "error" in step.tool_output.lower() or "failed" in step.tool_output.lower():
            next_step = steps[i + 1] if i + 1 < len(steps) else None
            if next_step is None and "error" not in final_answer.lower():
                issues.append(f"step {i}: tool error not acknowledged in final answer")

    return {
        "num_steps": len(steps),
        "num_tool_calls": len(steps),
        "issues": issues,
        "passed": len(issues) == 0,
    }

This is a toy version, but it demonstrates the real point: agent eval logic inspects the *process*, not just the *product*. Production-grade versions of this usually combine rule-based checks (loop detection, argument schema validation, step budgets) with an LLM judge that reads the whole trajectory and rates plan quality, similar to how a human reviewer would read a support ticket transcript rather than just the resolution note.

Task success rate means something different in each world

In RAG, "success" is usually binary-ish and query-local: given this question, is the answer correct and grounded. You can compute it over a fixed eval set of Q&A pairs and get a stable number.

In agent systems, "success" has to account for partial credit and for tasks that are legitimately open-ended. Consider an agent whose job is "resolve this customer refund request." Possible outcomes:

• Fully resolved, correct amount, correct account — full success
• Resolved but needed to ask the user one clarifying question first — success, different path
• Escalated to a human because policy required manager approval — this is *correct behavior*, not a failure, even though the agent didn't "complete" the task itself
• Issued the refund to the wrong account — task success by a shallow check (a refund happened), catastrophic failure by any real standard

If your eval only checks "did a refund get issued," you'll score the last case as a win. This is why agent evals need task-specific success criteria defined up front, usually as a checklist per task rather than a single pass/fail flag:

def score_refund_task(trajectory, final_state) -> dict:
    checks = {
        "correct_account": final_state["refund_account"] == final_state["order_account"],
        "correct_amount": abs(final_state["refund_amount"] - final_state["order_total"]) < 0.01,
        "escalated_if_over_threshold": (
            final_state["order_total"] <= 500
            or final_state["status"] == "escalated_to_human"
        ),
        "no_duplicate_refund": final_state["refund_count"] <= 1,
    }
    return {"checks": checks, "score": sum(checks.values()) / len(checks)}

Notice this eval encodes business policy (the $500 escalation threshold) directly into the scoring function. That's normal for agent eval and rare for RAG eval — RAG scoring is mostly about textual and semantic correctness, while agent scoring is frequently about whether the system respected real-world operational constraints, approval limits, and safety rules that have nothing to do with language quality.

Retrieval quality vs tool-use quality are not the same skill to measure

It's tempting to treat "did the agent call the right tool" as the agent-world equivalent of "did the retriever find the right chunk." They rhyme, but the failure surface is bigger for tools.

A retriever has one job: rank candidates by relevance. A tool call has at least three ways to go wrong that retrieval doesn't:

1. Wrong tool selected — equivalent to retrieval failure
2. Right tool, wrong arguments — e.g., calling get_weather(city="Springfield") without specifying which of the 30 U.S. cities named Springfield — this has no retrieval analog
3. Right tool, right arguments, but the agent misinterprets the output — e.g., the tool returns {"status": "pending"} and the agent reports the task as complete

Here's a simple tool-call correctness checker that covers argument validation, which is the piece teams most often skip:

from jsonschema import validate, ValidationError

def check_tool_call(tool_name: str, tool_input: dict, tool_registry: dict) -> list[str]:
    errors = []
    spec = tool_registry.get(tool_name)
    if spec is None:
        errors.append(f"unknown tool: {tool_name}")
        return errors

    try:
        validate(instance=tool_input, schema=spec["input_schema"])
    except ValidationError as e:
        errors.append(f"schema violation: {e.message}")

    for required_context_key in spec.get("requires_context", []):
        if required_context_key not in tool_input:
            errors.append(f"missing contextual arg: {required_context_key}")

    return errors

Run this as a pre-check on every tool call in every trajectory in your eval set, and log the error rate per tool. In practice this surfaces bugs fast — usually one or two tools account for most of the schema violations, often because the tool's docstring or description in the system prompt is ambiguous about a parameter's format.

Latency and cost are eval dimensions for agents, not afterthoughts

A RAG pipeline's cost is roughly fixed per query: one retrieval call, one generation call. You can estimate cost and latency from request volume without much variance.

An agent's cost is a function of how many steps it takes, and that number is not fixed — it depends on how well the agent reasons, how often tools fail and need retries, and whether it gets stuck in unproductive loops. This means cost and latency need to be first-class eval metrics for agents, tracked per task type, not just monitored in production after the fact.

A practical pattern: define a step budget per task category during eval, and flag any trajectory that blows past it even if the final answer is correct.

STEP_BUDGETS = {
    "simple_lookup": 3,
    "multi_step_booking": 8,
    "research_and_summarize": 12,
}

def check_efficiency(task_category: str, trajectory_length: int) -> dict:
    budget = STEP_BUDGETS.get(task_category, 10)
    return {
        "within_budget": trajectory_length <= budget,
        "overage": max(0, trajectory_length - budget),
        "budget": budget,
        "actual": trajectory_length,
    }

An agent that passes every correctness check but consistently blows through its step budget by 3x is a real problem — it means your production latency and API bill are 3x what they need to be, and it usually points to a specific reasoning gap (the agent re-checks something it already knows, or it can't tell when a subtask is actually done).

Where the two evals genuinely overlap

It's not all divergence. Some things matter in both worlds, and it's worth naming them so you don't over-rotate into thinking these are unrelated disciplines.

• Groundedness still matters for agents that read documents. If your agent calls a search_knowledge_base tool mid-trajectory and then writes a summary, that summary needs the same faithfulness check you'd run on a RAG answer. The agent context just makes it one sub-check among several, not the whole eval.
• LLM-as-a-judge is the backbone technique for both. Whether you're scoring "is this answer grounded in the context" or "was this trajectory a reasonable way to solve the task," you're usually asking a strong model to read something long and structured and return a structured judgment. The prompt engineering discipline — clear rubric, forced structured output, few-shot examples of good and bad — transfers directly.
• Human-labeled eval sets are non-negotiable for both. Automated metrics drift from what users actually care about if you never re-anchor them against real human judgments. RAG teams often skip this because a public benchmark like a Q&A dataset feels sufficient; agent teams skip it because trajectories are tedious to label. Both are mistakes.
• Regression testing before deploy is the actual point. For RAG this looks like a fixed eval set of queries run against every retriever/prompt change. For agents it looks like a fixed set of task scenarios (with tool mocks) run against every prompt/planning change. The mechanics differ, the discipline is identical: never ship a change you haven't run through the eval set.

Building an eval set that actually stresses the right failure modes

For RAG, a good eval set needs:

• Queries with a known, verifiable answer (so you can check correctness, not just plausibility)
• Deliberately adversarial queries where the right answer requires synthesizing multiple chunks, not just retrieving one
• Queries with no good answer in the corpus, to check whether the system says "I don't know" instead of hallucinating

For agents, a good eval set needs something extra: scenarios that force branching and failure recovery, not just happy-path completions. Concretely:

• Tasks where a required tool returns an error partway through, to check recovery behavior
• Tasks where the "obvious" first tool choice is a trap (returns data that looks relevant but is subtly wrong), to check whether the agent validates before trusting
• Tasks that should be refused or escalated, to check the agent doesn't just try to please by completing everything
• Long-horizon tasks that require the agent to remember an earlier decision several steps later, to check for context/state loss

If your agent eval set only contains clean, single-path tasks, you will not find your production bugs before your users do. The single most common mistake we see teams make is building the agent eval set by taking the RAG eval set's queries and wrapping them in tool calls — that only tests the happy path, which was never the risky part.

Putting it together: a two-track eval pipeline

In practice, the systems that get this right run two distinct pipelines rather than forcing one framework to cover both:

def run_eval_suite(system_type: str, eval_set: list, pipeline_fn):
    results = []
    for case in eval_set:
        output = pipeline_fn(case["input"])

        if system_type == "rag":
            results.append({
                "context_precision": score_context_precision(output.context, case["query"]),
                "faithfulness": score_faithfulness(output.context, output.answer),
                "answer_correctness": score_correctness(output.answer, case["reference"]),
            })
        elif system_type == "agent":
            results.append({
                "trajectory": evaluate_trajectory(output.steps, case["goal"], output.answer),
                "task_success": case["scorer"](output.trajectory, output.final_state),
                "efficiency": check_efficiency(case["category"], len(output.steps)),
            })

    return results

The RAG branch scores each case independently and averages cleanly into a dashboard number. The agent branch produces a richer, structured result per case that usually needs a human (or an LLM judge with a detailed rubric) to weigh trade-offs — a slightly inefficient trajectory that correctly escalates a risky action is probably a better outcome than an efficient one that doesn't.

Closing: the judge you build has to match the system you're judging

The underlying reason RAG and agent evaluation diverge isn't stylistic — it's structural. RAG is a function: query in, grounded answer out, one hop. Agents are processes: goal in, sequence of decisions and side effects out, many hops, many places to go quietly wrong. Metrics designed for a one-hop function will always under-measure a multi-hop process, no matter how well they're implemented.

If you take one thing from this: don't grade an agent on its final answer alone, and don't over-engineer trajectory analysis for a system that only ever does one retrieve-and-generate pass. Match the eval to the shape of the system.

And whichever track you're building, you'll eventually hit the same wall — a metric you can't compute with a formula, because "is this grounded," "was this a reasonable plan," or "did this refusal make sense" all require judgment, not arithmetic. That's the case for LLM-as-a-Judge: using a capable model, with a tight rubric and structured output, as the scoring mechanism itself. It's not a shortcut around building a real eval — it's the only practical way to scale the judgment calls that both RAG and agent evaluation ultimately come down to.