Evaluating Agent Trajectories, Not Just Outputs

Agent trajectory evaluation is the practice of scoring the entire path an agent takes to complete a task, not just the final output. A coding agent can produce a passing test suite while having deleted and rewritten an unrelated file along the way. A research agent can return a correct summary after querying the wrong API three times and getting lucky on the fourth. If you only grade the end state, you miss all of that. This article walks through why output-only evals fail for agents, what a trajectory actually consists of, and how to build a practical evaluation pipeline around it.

Why grading only the final output is not enough

A traditional LLM eval treats the model as a function: input goes in, text comes out, you score the text. That model works fine for single-turn tasks like classification or summarization. It breaks down the moment your system is an agent, meaning it plans, calls tools, observes results, and loops.

Consider a customer support agent tasked with refunding an order. Two runs both end with "refund issued" as the final message. In run A, the agent looked up the order, verified the refund policy, called the refund API once, and confirmed the amount. In run B, the agent called the refund API three times because it misread the response format, issued three partial refunds instead of one, and only the last message happened to say the right thing. Output-only grading scores both runs identically. Trajectory grading catches the difference immediately.

This matters more as agents get longer horizons. A single tool call has a small blast radius. A twenty-step agent loop compounds every small mistake: a wrong file path, a misparsed JSON blob, a retry that should have been a hard stop. The failure modes that matter in production are almost always visible in the trajectory before they show up in the output, and sometimes they never show up in the output at all because the agent got lucky.

What counts as a trajectory

A trajectory is the ordered sequence of everything the agent did between receiving a task and finishing it. In most agent frameworks (Claude Agent SDK, LangGraph, OpenAI's Responses API with tools, a custom ReAct loop) this sequence is a list of turns, and each turn typically contains one or more of:

• A reasoning or planning step (visible chain-of-thought, a scratchpad, or an internal "thinking" block)
• A tool call: the tool name, the arguments, and the raw request
• A tool result: the observation returned to the agent, including errors
• A message to the user, if the agent talks mid-task
• A final answer or terminal action

Logged as a transcript, a trajectory looks like this for a simple "fix the failing test" coding agent:

turn 1: reasoning -> "need to see which test is failing"
turn 1: tool_call -> run_tests()
turn 1: tool_result -> "FAIL: test_parse_date expected '2026-01-01' got None"
turn 2: reasoning -> "parse_date probably returns None on invalid format"
turn 2: tool_call -> read_file("src/dates.py")
turn 2: tool_result -> <file contents>
turn 3: reasoning -> "the regex doesn't handle single-digit months"
turn 3: tool_call -> edit_file("src/dates.py", diff)
turn 3: tool_result -> "edit applied"
turn 4: tool_call -> run_tests()
turn 4: tool_result -> "PASS"
turn 4: final_answer -> "Fixed the date parser, all tests pass."

Everything in that transcript is gradable material. The final answer is one line out of nine. If your eval pipeline only looks at that last line, you are throwing away almost all the signal about whether the agent reasoned correctly, used tools efficiently, and avoided unsafe actions.

The dimensions worth scoring

Trajectory evaluation is not one score, it is a small set of orthogonal checks. In practice, five dimensions cover most agent evals:

Task completion. Did the agent actually achieve the goal, and is the environment left in the state it should be in? This is the closest analog to output-only grading, but it checks end state (files changed, database rows written, tickets closed) rather than just the text the agent said.

Tool call correctness. Were the right tools called, with the right arguments, in a sensible order? A common failure is calling search_orders(customer_id=None) because the agent hallucinated a missing field instead of asking for it or looking it up first. You can check this with exact-match rules for deterministic tools (did it call delete_file at all? it never should have) and with an LLM judge for fuzzier cases (was this the most efficient tool choice given what was known at that step?).

Recovery behavior. Agents will hit errors: a rate limit from an API, a malformed JSON response, a tool that returns an empty result. What matters is not that an error occurred but what the agent did next. Did it retry sensibly, fall back to an alternative tool, or ask the user for help? Or did it loop the same failing call five times, or silently give up and fabricate a plausible-sounding answer?

Efficiency. Two trajectories that both complete the task are not equal if one took 4 tool calls and the other took 40. Step count, token count, and wall-clock time are all cheap proxies worth tracking, especially once you're paying per-token for a production agent fleet.

Safety and policy adherence. Did the agent stay inside its permission boundary? This is the dimension most teams under-invest in until an incident forces the issue. A refund agent that can technically call issue_refund with an arbitrary amount needs a trajectory check that verifies the amount never exceeds the original order total, independent of whether the final message looks correct.

Building a trajectory eval pipeline

The mechanics break into three stages: capture, scoring, and aggregation.

1. Capture the full transcript

You cannot evaluate what you do not log. Every tool call, its arguments, its raw result (including errors), and every reasoning step the agent exposes needs to be recorded with enough structure to replay later. If you're using the Claude Agent SDK, the message stream already gives you this as structured events, so the main job is persisting them rather than reconstructing them from a flat text log.

A minimal schema per trajectory:

{
  "trajectory_id": "run_8841",
  "task": "Fix the failing date parser test",
  "turns": [
    {"type": "reasoning", "content": "..."},
    {"type": "tool_call", "tool": "run_tests", "args": {}},
    {"type": "tool_result", "content": "...", "is_error": false},
    ...
  ],
  "final_state": {"files_changed": ["src/dates.py"], "tests_passing": true},
  "metadata": {"model": "claude-sonnet-5", "duration_ms": 8421, "step_count": 4}
}

Store this even for successful runs in production, not just eval runs. The trajectories that mattered most in most postmortems are the ones nobody thought to capture until after the incident.

2. Score with a mix of rules and LLM judges

Some checks should be deterministic code, not model calls. "Did the agent call delete_production_db?" is a grep, not a judgment call. Reserve LLM-as-judge scoring for things that genuinely need judgment: was this tool call reasonable given the prior context, did the agent's stated reasoning match what it actually did next, was the recovery from the error sensible.

A practical pattern is a two-pass judge: first pass extracts discrete claims from the trajectory ("agent claimed the test failed due to X", "agent chose to retry rather than escalate"), second pass checks each claim against the actual transcript. This catches a specific and common failure: an agent's natural-language narration of its own actions drifting from what it actually did, which a naive "does the final answer sound right" check will never catch.

Example rubric prompt for a judge scoring tool call correctness, given the transcript above:

You are scoring one agent trajectory for tool call correctness.
Given the full turn-by-turn transcript, answer for each tool call:
1. Was this tool call necessary given what was known at that point?
2. Were the arguments correct and non-hallucinated?
3. Was there a more efficient tool or argument choice available?
Score 1-5 per call, then an overall trajectory score.
Cite the specific turn number for any deduction.

Requiring turn-level citations forces the judge to ground its score in the transcript instead of pattern-matching on the final answer, which is the same failure mode you're trying to eliminate.

3. Aggregate and diff

A single trajectory score is not very actionable on its own. The value shows up when you compare trajectories across model versions, prompt changes, or tool set changes. Two aggregate views are worth building early:

• Per-dimension trend lines: plot task completion, tool call correctness, and step count separately over time or across model versions. A prompt change that improves completion but doubles step count is a real tradeoff you want visible, not hidden inside one blended score.
• Trajectory diffing: for a fixed benchmark task, diff the trajectory before and after a change. If a new system prompt causes the agent to call read_file twice instead of once before every edit, that shows up immediately in a diff view and would be invisible in an output-only eval.

Common pitfalls

Scoring reasoning text as if it were ground truth. An agent's chain-of-thought is not a reliable narration of what it did or why. Treat it as one more piece of evidence, cross-check it against the actual tool calls, don't grade it in isolation.

One giant LLM judge prompt. Asking a single judge call to simultaneously score completion, efficiency, safety, and tool correctness produces noisy, unstable scores. Split it into separate judge passes per dimension, even if that costs more tokens. The stability is worth it.

Ignoring non-terminating trajectories. Agents that hit a step limit or timeout without finishing are a distinct failure category from agents that finish with a wrong answer. Bucket them separately: both compress into "task failed" if you're not careful, and they need different fixes.

No environment reset between eval runs. If your agent operates against a shared sandbox (a filesystem, a test database), state leaking between runs will corrupt your trajectory comparisons. Snapshot and restore the environment before every eval run, the same way you would for integration tests.

Benchmarking only the happy path. Most trajectory-eval suites are built from tasks the agent is expected to complete cleanly. The trajectories that reveal the most about production readiness are the ones seeded with a broken tool, a rate-limited API, or an ambiguous instruction. Build a handful of these adversarial cases into every suite.

A small worked example

Say you're evaluating two versions of a research agent that answers questions using a web search tool and a code execution tool. The task: compute the year-over-year change in a specific metric from a dataset the agent needs to find and download.

Version A's trajectory: searches once, finds the dataset, downloads it, writes and runs one script, returns the answer. Four tool calls.

Version B's trajectory: searches, downloads a dataset, writes a script that throws a key error because it guessed a column name wrong, does not inspect the error message, writes an entirely new script from scratch guessing different column names, gets lucky, returns the same numeric answer as version A.

Output-only eval: both pass, identical score. Trajectory eval: version B gets flagged on tool call correctness (guessed instead of inspecting the error) and on efficiency (double the tool calls for the same task). That is exactly the signal you want before shipping a model or prompt change, because in a slightly different run, version B's blind guess on the second attempt might not land on the right column name at all, and you would ship a regression that your output-only suite never caught.

FAQ

Is trajectory evaluation the same as tracing or observability? They overlap but serve different purposes. Tracing and observability tools (OpenTelemetry-based agent tracing, vendor dashboards) are built to capture and visualize what happened, largely for humans debugging a single run. Trajectory evaluation is the scoring layer on top: turning that captured trace into a repeatable, comparable score you can track across versions and use as a regression gate.

Do I need a separate judge model for every dimension? Not necessarily a separate model, but separate calls. Using the same judge model with a tightly scoped prompt per dimension (completion, tool correctness, recovery, efficiency, safety) is far more reliable than one combined prompt asking for five scores at once. Keep the prompts short and grounded in specific turn citations.

How many eval trajectories do I need before the scores are trustworthy? There is no fixed number, but treat it like any other statistical sample: a handful of tasks run once each will bounce around too much to trust for a regression gate. Running your benchmark tasks multiple times, to capture the agent's own run-to-run variance, and covering a spread of task types including a few adversarial ones with broken tools or ambiguous instructions, matters more than raw task count.

Can I use trajectory evals in CI, or is this only for offline research? Both. A small, fast subset of trajectory checks (deterministic rules on tool calls, step-count budgets, safety boundary checks) can run in CI on every prompt or tool change and gate merges. The heavier LLM-judge passes are usually better suited to a nightly or pre-release run against a fuller benchmark, since they cost more and take longer.

What is the single highest-leverage first step if I only have output-only evals today? Start logging full transcripts for every agent run in production, even before you build any scoring on top. Teams that skip this step almost always regret it the first time they need to debug a subtle regression and discover they only kept the final answer. The scoring pipeline can be built incrementally once the data exists.