How to Evaluate AI Agents: Trajectories, Outcomes and Cost

To evaluate AI agents you need to measure three separate things: whether the agent produced the right end result (outcomes), how it got there (trajectories), and what it spent along the way (cost in tokens, steps and latency). Most teams grade only the final answer, which is why agents that demo well still fall apart in production: the transcript can claim success while the database says otherwise, and a "passing" run can quietly cost ten times what the task is worth. This guide covers all three axes with concrete metrics, working code for a small evaluation harness, and the judgment calls you will face when you wire agent evals into CI.

Everything here assumes you are shipping an agent that calls tools in a loop: a support agent that reads orders and issues refunds, a coding agent that edits files and runs tests, a research agent that searches and synthesizes. The techniques apply whether you built on the OpenAI Agents SDK, LangGraph, the Claude tool use API, Pydantic AI, or a hand-rolled while loop. The framework does not change what you measure.

Why You Cannot Evaluate AI Agents Like Plain LLMs

Classic LLM evaluation is one shot: one input, one output, one grade. Summarize this document, classify this ticket, extract these fields. You can grade thousands of those with string checks or a judge model and be done by lunch.

An agent run is a loop: the model reasons, picks a tool, gets an observation back, and repeats, sometimes for dozens of steps. That changes evaluation in five concrete ways.

• Errors compound. A step that is 95 percent reliable gives you roughly 0.95 to the power of 20, about 36 percent, across a 20 step task. Per-step quality that sounds fine produces end-to-end behavior that is not fine.
• There are many valid paths. One run looks up the order first, another checks the refund policy first, and both finish correctly. Exact-match grading punishes healthy variation and rewards memorized scripts.
• Agents have side effects. They write files, send emails, mutate databases. The thing to grade is the state of the world after the run, not the words in the chat window.
• Agents narrate actions they never took. A depressingly common failure mode is the agent replying "I have issued the refund" with no refund tool call anywhere in the trajectory. If you grade transcripts alone, this scores as a pass.
• Nondeterminism is structural. Temperature 0 does not make agent runs reproducible: tool results, timestamps, retrieval ordering and provider-side model updates all vary. You are evaluating a distribution of behaviors, not a single run, which means multiple trials per task is not optional.

Once you accept those five facts, the shape of a proper agent eval follows directly: programmatic outcome checks against a real or simulated environment, trajectory scoring for diagnosis, cost accounting for economics, and enough repeated trials to make the numbers mean something.

The Three Axes: Outcomes, Trajectories and Cost

A useful rule of thumb: gate releases on outcomes, debug with trajectories, set budgets with cost.

• Outcomes answer "did the job get done". This is the number the business cares about and the only number worth blocking a deploy on.
• Trajectories answer "how did it get done". They are diagnostic: they tell you where failures begin, whether successes were luck, and what specifically to fix.
• Cost answers "was it worth doing this way". Two agents with identical success rates can differ several-fold in tokens and latency, and that difference is your margin.

Skipping any one axis has a predictable failure mode. Outcomes without trajectories: you know the pass rate dropped from last week but have no idea why. Trajectories without outcomes: you polish process elegance that nobody pays for. Either without cost: you ship an agent whose unit economics do not survive contact with real traffic volume.

Outcome Evaluation: Did the Job Actually Get Done

Define success as executable checks against the environment, not as a string the agent should say. For a coding agent, success is "the hidden test suite passes on the modified repo". For a support agent, success is "a refund record exists for the right order with the right amount, and a confirmation email was queued". For a research agent, success is "the answer contains the verifiable fact and cites a fetched source".

The pattern is always the same: after the run, inspect the environment.

def check_refund_outcome(env, case):
    order = env.db.get_order(case["order_id"])
    return (
        order.status == "refunded"
        and order.refund_amount == order.total
        and env.outbox.has_email_to(case["customer_email"])
    )

Notice what this does not do: it never reads the agent's final message. The environment is the source of truth. If you also care about the customer-facing reply (you usually do), grade that separately with a judge, covered below, but never let a well-worded reply substitute for a missing state change.

Because runs are stochastic, run every case multiple times and report two different aggregates:

• pass@k: the task counts as solved if at least one of k trials succeeds. This is the right metric when retries are cheap and you have a verifier that can pick the good attempt, for example a coding agent whose output is checked by tests before merging.
• pass^k: the task counts as solved only if all k trials succeed. This is the reliability metric, popularized by the tau-bench line of work on customer service agents, and it is the honest number for anything customer-facing where you do not get retries. A 90 percent single-run success rate compounds to about 59 percent over five independent runs (0.9 to the power of 5), which is why an agent that "usually works" feels so much worse in production than its eval score suggested.

Four trials per case is a reasonable floor. If your suite is small, more trials per case buys you more signal than more cases of the same flavor.

Trajectory Evaluation: How to Evaluate AI Agents Step by Step

A trajectory is the ordered record of everything the agent did: model turns, tool calls with their arguments, and the observations that came back. Outcome checks tell you that a run failed. The trajectory tells you where and why, and it also exposes ugly successes: runs that reached the right answer through twelve redundant searches and two hallucinated detours.

There are four standard ways to score a trajectory against a reference, in increasing order of leniency.

1. Exact match: the actual tool sequence equals the expected sequence, same tools, same order, nothing extra. Brittle. Reserve it for strictly procedural workflows where deviation is itself a defect, like compliance flows.
2. In-order match: every expected tool appears in the actual sequence in the right relative order, extra calls allowed. This is the best default when ordering carries meaning, for example check_refund_policy must precede issue_refund.
3. Any-order match: every expected tool appears somewhere in the run. Use it when the steps are independent lookups.
4. Precision and recall over tool calls: recall asks "did it call the tools the task needed", precision asks "how much of what it called was actually needed". Low recall predicts failure. Low precision predicts cost blowups and flailing.

All four fit in a dozen lines:

def tool_recall(actual, expected):
    called = set(actual)
    return len(called & set(expected)) / len(expected)

def tool_precision(actual, expected):
    if not actual:
        return 0.0
    needed = set(expected)
    return sum(1 for t in actual if t in needed) / len(actual)

def in_order(actual, expected):
    it = iter(actual)
    return all(tool in it for tool in expected)

The in_order function uses a Python idiom worth knowing: "tool in it" consumes the iterator as it searches, so the all() succeeds only if the expected tools appear as a subsequence of the actual sequence.

Two refinements matter in practice. First, grade arguments, not just tool names. An agent calling issue_refund with the wrong amount matches by name and fails the customer. Assert on the critical arguments (ids, amounts, recipients) and ignore free-text ones. Second, detect degenerate patterns directly:

import json

def repeated_calls(trajectory):
    seen = {}
    for step in trajectory:
        key = (step.tool, json.dumps(step.args, sort_keys=True))
        seen[key] = seen.get(key, 0) + 1
    return {k: n for k, n in seen.items() if n > 2}

The same tool with identical arguments three or more times almost always means a stuck loop, and it is cheaper to flag it structurally than to make a judge model notice it.

Finally, classify failures into a small taxonomy and track the distribution across versions: wrong tool, wrong arguments, hallucinated success, stuck loop, premature give-up (asking the user for something a tool could fetch), and budget blowout (right direction, ran out of steps). A prompt change that converts wrong-tool failures into budget blowouts is progress you would never see in a single pass-rate number.

LLM-as-Judge Without Fooling Yourself

Programmatic checks cannot grade everything. Tone of a support reply, faithfulness of a summary to fetched sources, whether an escalation message contains the right context: these need a judge model. Judges are genuinely useful and genuinely easy to misuse, so adopt some discipline.

• Grade with a rubric of separate binary criteria, not a single 1 to 10 score. Models are poor at absolute scoring and much better at answering "does this reply promise anything outside the policy, yes or no".
• Show the judge the trajectory when judging process. A judge that only sees the final answer will happily bless hallucinated success.
• Use a different model family for the judge than for the agent when you can. Self-preference bias, judges favoring outputs that sound like themselves, is a documented failure mode in the evaluation literature.
• Pin the judge: fixed model version, temperature 0, fixed rubric. An unpinned judge turns your eval history into noise.
• Calibrate before trusting. Hand-label 30 to 50 runs yourself, run the judge on the same runs, and inspect every disagreement. Iterate on the rubric until the remaining disagreements are rare and explainable, and re-check calibration whenever you change the rubric or the judge model.

A rubric prompt that works well as a starting point:

You are grading one run of a customer support agent.

Task given to the agent:
{task}

Final reply sent to the customer:
{reply}

Tool calls the agent made, in order, with results:
{trajectory}

Policy excerpt the agent must follow:
{policy}

Grade each criterion PASS or FAIL with one sentence of evidence:
1. resolution: the customer's actual problem is resolved or correctly escalated
2. grounding: every factual claim in the reply is supported by a tool result above
3. policy: the reply promises no refund, discount, or exception outside the policy excerpt
4. tone: professional, specific, no invented apologies for things that did not happen

Return JSON:
{"resolution": "...", "grounding": "...", "policy": "...", "tone": "...", "notes": "..."}

Treat judge scores as one signal among several, never as the gate by themselves. The deploy gate should stay programmatic: environment-state outcomes plus judge criteria that you have calibrated against human labels.

Cost Metrics: The Axis Everyone Ignores

An agent eval that does not track spend is half an eval. Log, per run: input tokens split into cached and uncached (providers price them very differently, and long agent trajectories are exactly where prompt caching pays off), output tokens, number of model calls, number of tool calls, retries, and wall-clock latency.

Then compute the one number that should drive model and design choices:

cost_per_solved_task = total_spend_across_all_runs / tasks_solved

The numerator includes failed runs deliberately. Failures still burn tokens, and an agent that fails 30 percent of the time pays for those failures whether or not you count them. This is also why a cheaper model with a slightly lower success rate sometimes wins on unit economics, and why "smaller model plus a programmatic verifier plus one retry" is a configuration always worth benchmarking against "bigger model, one shot". Run your same suite across the candidate configurations and compare success rate against cost per solved task; the right choice is frequently not the one with the highest raw pass rate.

Latency deserves the same treatment: report p50 and p95 wall-clock per task, not the mean, because agent latency is dominated by step count and the tail is what users feel. And make budgets part of the definition of success. A run that exceeds max_steps, max_tokens, or a timeout is a failure even if it was heading the right way, because in production you would have killed it. Evaluating with unlimited budgets while running production with limits is grading a different system than the one you ship.

A Minimal Harness to Evaluate AI Agents in CI

You do not need a platform to start. A JSONL file of cases, an isolated environment factory, a runner, and a summary function will carry you a long way, and every hosted tool you might adopt later (LangSmith, Langfuse, Braintrust, Arize Phoenix, W&B Weave) maps onto the same shape.

One case per line, with setup fixtures and expectations together:

{"id": "refund-001",
 "prompt": "Customer jane@example.com wants a refund on order 4312, it arrived broken.",
 "setup": {"orders": [{"id": 4312, "email": "jane@example.com", "total": 49.0, "status": "delivered"}]},
 "expected_tools": ["lookup_order", "check_refund_policy", "issue_refund"],
 "ordered": true,
 "max_steps": 8}

The runner builds a fresh environment per trial so cases can never contaminate each other, executes the agent, and records everything:

import json, time, statistics

def run_suite(agent_factory, cases, trials=4):
    results = []
    for case in cases:
        for t in range(trials):
            env = make_env(case["setup"])   # fresh, isolated state per trial
            agent = agent_factory(env)
            start = time.time()
            run = agent.run(case["prompt"], max_steps=case["max_steps"])
            results.append({
                "case": case["id"],
                "trial": t,
                "outcome": check_outcome(env, case),
                "recall": tool_recall(run.tools, case["expected_tools"]),
                "precision": tool_precision(run.tools, case["expected_tools"]),
                "in_order": in_order(run.tools, case["expected_tools"]),
                "steps": len(run.steps),
                "tokens": run.usage.total_tokens,
                "latency_s": round(time.time() - start, 2),
            })
    return results

def summarize(results, trials):
    by_case = {}
    for r in results:
        by_case.setdefault(r["case"], []).append(r["outcome"])
    pass_rate = statistics.mean(r["outcome"] for r in results)
    pass_all = statistics.mean(all(v) for v in by_case.values())
    mean_tokens = statistics.mean(r["tokens"] for r in results)
    print(f"pass rate: {pass_rate:.2%}  pass^{trials}: {pass_all:.2%}  tokens/run: {mean_tokens:.0f}")

Wiring this into CI is mostly policy decisions:

• Run a smoke subset (10 to 20 cases, 2 trials) on every pull request, and the full suite nightly. Full agent suites are too slow and too expensive for every commit.
• Gate on regressions against a committed baseline file, not on absolute thresholds. Fail the build if pass rate drops beyond your noise band or cost per solved task rises above budget. With 4 trials on 50 cases, swings of a few percentage points are noise; look at which specific cases flipped before believing a delta.
• Pin model versions in the eval environment. A provider-side model update should show up as a deliberate re-baseline, not as a mystery regression on a Tuesday.
• Keep expectations out of the agent's reach. If expected_tools or assert logic ever leaks into a prompt or a retrieval index, the eval silently becomes a memorization test.
• Use mocked or containerized tools. Evals that hit live third-party APIs inherit their flakiness and their bills, and they make trials non-independent.

Public Agent Benchmarks and What They Actually Tell You

Public benchmarks measure models under a particular scaffold. Your eval suite measures your product. Both are useful; only one predicts your production behavior.

The ones worth knowing in 2026: SWE-bench Verified for coding agents resolving real GitHub issues against hidden tests, tau-bench and its follow-up tau2-bench for customer service tool use under policy constraints (the home of pass^k), WebArena for browser agents on realistic web tasks, GAIA for general assistant questions requiring tools, OSWorld for full computer use, and Terminal-Bench for agents working in a shell. When a lab announces a model, these are the numbers in the launch post.

Read all of them with three caveats. Contamination: benchmarks published before a model's training cutoff may be partially memorized. Harness sensitivity: the same model scores meaningfully differently under different agent scaffolds, so a leaderboard entry is a model-plus-harness score, not a model score. Goodharting: once a benchmark drives marketing, it stops being a neutral measurement.

The practical workflow: use public benchmarks to shortlist two or three candidate models for your agent's task family, then decide with your own suite. Fifty cases distilled from your real traffic will evaluate AI agents against your actual bar better than any leaderboard, because they encode your tools, your policies, and your users' phrasing.

From Offline Evals to Production Monitoring

Offline evals are necessary and not sufficient. Real traffic contains inputs your suite never imagined, and agent regressions love to hide in the gap. Close the loop in three steps.

First, trace everything in production: every model call, tool call, argument set, token count and latency, tied to a run id. The OpenTelemetry GenAI semantic conventions cover this, and the tracing platforms named above all ingest them. A trajectory you did not record is a failure you cannot diagnose.

Second, run online evaluation on a sample. Apply the same programmatic checks and the same calibrated judge rubrics to a slice of production traces, asynchronously, and alert on drift in the success proxy, step count, or token spend per task. Step count creeping up over a week is often the first visible symptom of a degraded tool or a changed upstream API.

Third, turn incidents into cases. Every production failure that reaches a human should become a new eval case with a fixture reproducing it, the same way a good engineering team turns bugs into regression tests. This is how a 20 case starter suite grows into a few hundred cases that actually cover your traffic, and it is the compounding loop that makes the whole practice pay off.

For risky changes, put a canary in front: route a small percentage of traffic to the new agent version, compare outcome proxies and cost side by side, then roll forward or back with data instead of vibes.

Common Mistakes That Invalidate Agent Evals

• Grading transcripts instead of environment state, which awards passes to narrated, non-existent actions.
• Running one trial per case. With agent-level variance, a 50 case suite at one trial ranks two versions roughly at coin-flip reliability.
• Using the same model as agent and judge without calibration, then treating the judge score as ground truth.
• Evaluating with unlimited step and token budgets while production enforces limits.
• Letting the eval set leak into prompts, few-shot examples, or a retrieval corpus.
• Freezing the suite at launch instead of feeding production failures back into it.
• Reacting to pass-rate deltas that are inside the noise band for the suite size, and re-rolling evals until the number looks good.

A Practical Starting Point

If you have an agent and no evals, this is the first afternoon of work: write 20 cases from real or realistic tasks, each with fixtures and a programmatic outcome check. Add tool recall, precision and an in-order check from the snippets above. Run every case 4 times, report pass rate, pass^4, tokens per run and p95 latency. Commit the baseline, wire the suite into nightly CI with a regression gate, and add a calibrated judge only for the qualities code cannot check. From then on, every production incident becomes case number 21, 22, 23. Within a quarter you will trust your eval dashboard more than your demo, which is exactly the point.

FAQ

How many test cases do I need to evaluate AI agents?

Start with 20 to 50 cases that mirror real tasks, run 4 or more trials each, and grow the suite from production failures. A small suite of real cases with repeated trials beats a large synthetic suite run once, because agent variance, not case count, is usually the dominant noise source early on.

What is the difference between pass@k and pass^k?

pass@k counts a task as solved if at least one of k attempts succeeds; it fits workflows with cheap retries and a verifier, like code that must pass tests. pass^k counts a task as solved only if all k attempts succeed; it measures consistency and is the honest metric for customer-facing agents that get one shot per user.

Should I use LLM-as-judge or code-based checks?

Code first, always, for anything verifiable: state changes, test results, required tool calls, argument correctness. Use a judge only for genuinely subjective qualities like tone and faithfulness, with a binary-criteria rubric, a pinned model, and calibration against hand-labeled runs. Gate deploys on the programmatic checks.

How do I evaluate multi-agent systems?

The same three axes apply. Grade the system on end-to-end outcomes, treat sub-agent invocations as tool calls in the orchestrator's trajectory, and trace across agent boundaries so cost and latency attribute to the run as a whole. Per-sub-agent scores are diagnostic detail, not the headline number.

How often should agent evals run?

A smoke subset on every pull request, the full suite nightly, and a deliberate full run plus re-baseline whenever you change the model, a system prompt, or a tool contract. Continuous online sampling in production catches what the offline schedule misses.

Do I need an eval framework, or can I build my own?

The harness in this guide is a few hundred lines and covers outcomes, trajectories and cost. Frameworks and platforms (DeepEval, promptfoo, Inspect, OpenAI Evals, LangSmith, Langfuse, Braintrust, Arize Phoenix, W&B Weave) add trace UIs, dataset management, hosted judges and team workflows. A sensible path: start homegrown so you understand your metrics, adopt a platform when reporting and trace inspection become the bottleneck.