Evaluating Tool-Calling Accuracy: Did the Agent Pick the Right Tool?

The demo worked. Production didn't.

Every agent builder has lived this moment. You wire up three tools — search_orders, get_refund_policy, issue_refund — hand the agent a system prompt, run five demo queries, and it nails every one. You ship it. Two weeks later a support lead forwards you a transcript where a customer asked "can I get my money back for the broken headphones" and the agent called search_orders with the wrong customer ID, then confidently called issue_refund anyway, using an order number it never actually looked up.

Nothing crashed. No exception was thrown. The JSON was perfectly valid. And yet the agent did the wrong thing at the most consequential step in the entire interaction.

This is the blind spot that swallows most agent projects: teams eval the final answer's tone, format, and helpfulness, but never directly check whether the agent picked the right tool, with the right arguments, at the right moment. Tool selection is the load-bearing wall of any agentic system. If the wall is cracked, it doesn't matter how nicely the rest of the house is painted. This article is about building an evaluation harness specifically for tool-calling accuracy — what to measure, how to measure it, and where teams get it wrong.

Why tool-calling accuracy is a different problem than answer quality

Most LLM evaluation intuition comes from text-generation tasks: is the summary accurate, is the tone right, did it follow instructions. Tool calling breaks that mental model in three ways.

First, tool calls are structured decisions with a discrete correct answer space, not open-ended text. At any given turn, there is usually a small, enumerable set of "acceptable" tool calls — sometimes exactly one, sometimes two or three that are equally valid. This means you can score tool selection with much stricter precision/recall metrics than you'd ever apply to free text, and you should, because "close enough" tool calls are often not close enough at all. Calling cancel_subscription instead of pause_subscription isn't a stylistic quibble; it's a different real-world action with different consequences.

Second, errors compound downstream in a way text errors don't. If an agent's prose response is slightly off, the user reads it and moves on. If an agent's tool call is slightly off — wrong parameter, wrong tool, wrong order — the error propagates into a tool result, which becomes context for the next reasoning step, which produces another decision built on a false premise. A single bad get_customer call with the wrong customer_id can poison every downstream reasoning step in a ten-turn agent trajectory.

Third, the failure modes are invisible in the final output. A well-trained model is very good at producing a fluent, confident final answer even when it was built on the wrong tool call. This is the most dangerous property of tool-calling failures: they hide behind good writing. You cannot catch them by reading the last message in the transcript. You have to look at the trace.

The four things you actually need to measure

Tool-calling accuracy isn't one number. Teams that try to collapse it into a single pass/fail score end up with an eval that's technically "green" while the agent quietly misbehaves in production. Break it into four distinct questions.

1. Tool selection accuracy — given the conversation state, did the agent call the correct tool, or one of the correct tools if there are multiple valid ones? This is the headline metric, usually reported as accuracy, precision, or an F1 score against a labeled "gold" tool for each turn.

2. Argument correctness — given that the agent called the right tool, were the parameters right? This splits further into:

• Required parameters present (did it forget customer_id?)
• Correct values (did it pass 2026-07-03 when the user said "next Tuesday" and next Tuesday is actually 2026-07-07?)
• No hallucinated parameters (did it invent a discount_code field that isn't in the schema, or worse, a value that isn't real?)

3. Sequencing and dependency correctness — in multi-step tasks, did the agent call tools in an order that respects real dependencies? You cannot call issue_refund before search_orders succeeds and returns a valid order ID. An agent that calls the right tools individually but in the wrong order is arguably worse than one that picks a slightly suboptimal tool, because sequencing errors often mean the model isn't actually tracking state.

4. Abstention and no-call correctness — did the agent correctly recognize when *no* tool call was needed at all? This is the most commonly skipped dimension. Agents that are over-eager to call tools will fire search_web for questions they could answer from context, wasting latency and cost and sometimes injecting irrelevant noise into their own context window. Under-eager agents will try to answer from memory when they should have looked something up. Both are tool-calling errors even though neither is a "wrong tool" in the traditional sense.

A mature eval suite reports all four separately. A single blended "tool accuracy: 91%" number tells you almost nothing actionable. You don't know if that nine percent failure is a selection problem, an argument problem, or a sequencing problem, and the fix for each is completely different.

Building a gold-labeled test set

You cannot evaluate tool-calling accuracy without ground truth. The single highest-leverage investment here is a well-constructed labeled dataset, and it's worth being disciplined about what goes into it.

Start by mining real transcripts — support logs, internal dogfooding sessions, anything where a human agent or your existing system actually handled the request. For each turn, label:

• the ideal tool call, or explicit "no tool call" if that is correct
• the ideal arguments
• alternative acceptable tool calls, if any exist — this matters more than people think, because many real queries have two or three defensible tool choices

Then deliberately construct adversarial cases, because production traffic will contain them even if your logs don't yet:

• Near-duplicate tools. If your toolkit has update_user and update_user_profile, you need test cases that specifically probe whether the model confuses them.
• Underspecified requests. "Cancel my order" when the user has three open orders. Does the agent ask a clarifying question, or does it guess and call the tool on the wrong order?
• Tool-shaped language that isn't a tool-shaped intent. "I could really use a refund right about now" said sarcastically inside a complaint, versus an actual refund request.
• Multi-tool tasks with a required order. "Check if this SKU is in stock and if so place the order." The agent must call check_inventory before place_order, and must not call place_order if the check comes back negative.

A good rule of thumb: for every one "happy path" example in your test set, aim for at least two adversarial or edge-case examples. Happy-path cases are what agents already handle well because they resemble the demo. Edge cases are what breaks in production, and they are what your eval set is actually for.

Instrumenting the harness

Here is a minimal but real evaluation harness in Python. It assumes your agent framework exposes the list of tool calls it made for a given turn — most frameworks, whether you are using the Anthropic Messages API directly, a LangGraph agent, or a custom loop, let you inspect this from the raw response.

from dataclasses import dataclass, field
from typing import Any, Optional

@dataclass
class ToolCall:
    name: str
    arguments: dict[str, Any]

@dataclass
class GoldExample:
    turn_id: str
    conversation: list[dict]
    expected_tool: Optional[str]          # None means "no tool call expected"
    expected_args: dict[str, Any] = field(default_factory=dict)
    acceptable_alternatives: list[str] = field(default_factory=list)
    required_arg_keys: list[str] = field(default_factory=list)

def score_tool_selection(gold: GoldExample, actual: Optional[ToolCall]) -> dict:
    result = {"turn_id": gold.turn_id, "selection_correct": False,
              "args_correct": None, "notes": []}

    # Case 1: gold says no tool call was needed
    if gold.expected_tool is None:
        result["selection_correct"] = actual is None
        if actual is not None:
            result["notes"].append(f"unnecessary call to {actual.name}")
        return result

    # Case 2: gold expects a tool call, but agent made none
    if actual is None:
        result["notes"].append(f"missing call, expected {gold.expected_tool}")
        return result

    # Case 3: check tool name against gold plus accepted alternatives
    valid_tools = {gold.expected_tool, *gold.acceptable_alternatives}
    result["selection_correct"] = actual.name in valid_tools
    if not result["selection_correct"]:
        result["notes"].append(f"called {actual.name}, expected one of {valid_tools}")
        return result

    # Case 4: tool was right, now check required arguments
    missing = [k for k in gold.required_arg_keys if k not in actual.arguments]
    mismatched = {
        k: (gold.expected_args[k], actual.arguments.get(k))
        for k in gold.expected_args
        if k in actual.arguments and actual.arguments[k] != gold.expected_args[k]
    }
    result["args_correct"] = not missing and not mismatched
    if missing:
        result["notes"].append(f"missing required args: {missing}")
    if mismatched:
        result["notes"].append(f"mismatched arg values: {mismatched}")

    return result


def run_eval(examples: list[GoldExample], agent_fn) -> dict:
    results = [score_tool_selection(ex, agent_fn(ex.conversation)) for ex in examples]
    n = len(results)
    selection_acc = sum(r["selection_correct"] for r in results) / n

    scored_for_args = [r for r in results if r["args_correct"] is not None]
    args_acc = (
        sum(r["args_correct"] for r in scored_for_args) / len(scored_for_args)
        if scored_for_args else None
    )

    failures = [r for r in results if not r["selection_correct"] or r["args_correct"] is False]

    return {
        "n": n,
        "tool_selection_accuracy": round(selection_acc, 3),
        "argument_accuracy": round(args_acc, 3) if args_acc is not None else None,
        "failures": failures,
    }

The important design choice here is that the harness reports tool selection accuracy and argument accuracy as two separate numbers, and it keeps a failures list with human-readable notes rather than just a pass rate. When you run this weekly against your regression set, you want to be able to skim the failures list and immediately see the pattern. "Five of seven failures this week were the model confusing update_user and update_user_profile" is actionable in a way "87% pass rate" is not.

For sequencing correctness in multi-turn tasks, extend this to compare the full ordered list of tool calls against a gold sequence, using something like a Levenshtein-style edit distance over the call sequence rather than exact match. That lets you distinguish "called things in a slightly different but still valid order" from "skipped a required step entirely."

Where exact-match scoring breaks down

The harness above works cleanly when arguments are exact strings or numbers. It falls apart the moment arguments involve anything that has multiple valid representations — dates, free-text search queries, natural-language filter descriptions passed into a tool.

If the user says "book something for next Tuesday" and your gold argument is "2026-07-07", exact string match works fine because dates are deterministic once you fix "today." But if the tool takes a search_query string like "waterproof running shoes size 10 under $100", there is no single correct string. The agent might reasonably produce "waterproof running shoes" with separate size and max_price parameters, or fold everything into the query string, depending on how it interpreted the schema. Exact match will fail cases that are functionally correct.

The fix is a tiered scoring approach rather than one universal comparator:

• Exact match for enums, IDs, booleans, and anything with a canonical single value.
• Normalized match for dates, phone numbers, and currency — normalize both sides to a canonical form before comparing, parsing the date string and comparing the resulting date objects rather than the raw strings.
• Semantic match via LLM-as-a-judge for free-text arguments where multiple phrasings are valid. You give the judge the gold intent and the actual argument value and ask a narrow yes/no question: does this argument value correctly capture the same constraint as the gold description?

The mistake to avoid is reaching for LLM-as-a-judge everywhere by default. It's slower, costs money, and, critically, introduces its own noise into your eval, which defeats the purpose of having ground truth in the first place. Use deterministic comparison wherever the argument type allows it, and reserve the judge for the genuinely ambiguous free-text cases.

Catching the tools that look right but aren't

Some of the most damaging tool-calling errors are ones that pass a naive "was the tool name correct" check but are still wrong in ways that matter operationally.

Redundant calls. The agent calls search_orders three times in a row with identical arguments because it "forgot" it already had the result in context. This doesn't fail a selection-accuracy check, since the tool was arguably correct each time, but it's a real cost and latency problem, and often a sign that your context management or system prompt needs work. Track a call redundancy rate — duplicate calls with identical arguments within a single trajectory — as a separate metric.

Right tool, stale context. The agent calls get_account_balance correctly, but does so using an account ID it extracted three turns ago before the user switched accounts mid-conversation. The tool name is right, the argument even looks plausible, but it's wrong because the world state moved and the agent didn't track it. Catching this requires your gold examples to include multi-turn state changes specifically, not just single-turn snapshots.

Confidently wrong on missing information. The agent needed a zip_code to call get_shipping_estimate, didn't have one, and instead of asking the user or calling a lookup tool, invented "00000" or reused an unrelated zip code from earlier context. This is the tool-calling equivalent of hallucination, and it's arguably the most dangerous category because it produces a tool call that executes successfully and returns a real but meaningless result. Your harness should specifically test what happens when required information is absent from the conversation, and score whether the agent asks for it versus fabricates it.

Running this continuously, not once

A tool-calling eval you run once before launch is a snapshot, not a safety net. Tool-calling accuracy degrades over time for reasons that have nothing to do with your test set going stale.

Tool schemas change: someone renames a parameter or adds a new required field, and nobody re-runs the eval against the new schema. New tools get added to the same agent, and the larger the tool inventory grows, the more the model has to disambiguate between similarly-named or similarly-described tools — accuracy tends to degrade as toolkits grow, especially past fifteen or twenty tools available to a single agent. Underlying model versions change when you upgrade to a new release, and tool-calling behavior is one of the areas most sensitive to model-version differences even when general capability improves.

Treat the tool-calling eval as a regression suite that runs on every prompt change, every schema change, and every model swap, the same discipline you would apply to unit tests in a codebase. Store failures over time and watch the trend line per tool, not just the aggregate. A tool that was 98 percent correct last month and is 91 percent this month is telling you something specific broke, even if your overall average still looks fine because other tools compensated.

It's also worth splitting your eval set into a held-out regression set that never changes, so you can compare apples to apples over time, and a rolling set sourced from recent production failures, so you keep closing the gap on whatever is actually breaking right now. Teams that only maintain the first set end up with a beautiful trend line on an eval that no longer resembles their real traffic.

Judging what "correct" even means

Not every disagreement between your gold label and the agent's actual tool call is a bug. Sometimes the agent found a genuinely better path than the one your labeler wrote down six months ago. Maybe it combined two tool calls into one because a newer tool supports batch arguments, or it asked a clarifying question in a case where your gold label assumed the agent should have guessed. This is exactly where LLM-as-a-Judge earns its place in the pipeline, not as a replacement for deterministic scoring, but as an adjudication layer for the failures your harness flags, deciding whether a mismatch is a real defect or a legitimate alternative path your gold set simply didn't anticipate. Feed the judge the conversation, the gold rationale, and the agent's actual tool call, and ask it a narrow, specific question rather than "was this good." Narrow judgments are the ones you can trust and audit. Used this way, the judge doesn't replace your ground truth; it keeps your ground truth honest as the agent, the tools, and the traffic all keep evolving.