Agent Observability: Tracing Every Decision an Agent Makes

Why your agent's biggest bug will be invisible

Somewhere in production right now, an agent is making a decision you can't see. It called a tool, got a result, reasoned about that result, called another tool, and eventually returned an answer — or didn't. When a user reports "it just gave me the wrong refund amount," you're left staring at a single input and a single output, with an entire chain of reasoning collapsed into a black box in between.

This is the defining operational problem of agentic systems. A traditional web request has a clean, mostly linear path: request in, a few function calls, database query, response out. An agent's execution path is dynamic. It branches based on model output, retries based on tool failures, and sometimes loops for a dozen steps before converging. You cannot debug what you cannot see, and print statements scattered across a codebase do not scale past a demo.

Agent observability is the discipline of making every decision an agent makes inspectable — not after the fact through guesswork, but as structured, queryable data captured while the agent runs. This article walks through what that actually means in practice: the anatomy of a trace, how to instrument spans and events, what to log versus what to redact, how to catch silent failures like tool-call hallucination, and how to turn traces into dashboards that catch regressions before your users do.

What "observability" means for an agent, specifically

Observability for traditional software rests on three pillars: logs, metrics, and traces. Those pillars still apply to agents, but each one needs to be reinterpreted.

• Logs capture discrete events — a tool was called, a retry happened, a guardrail fired. For agents, logs need to include the *reasoning context* around the event, not just the event itself.
• Metrics aggregate numbers over time — latency, token usage, cost per request, tool error rate. For agents, the interesting metrics are often behavioral: how many steps did it take to finish, how often did it call the same tool twice in a row, how often did it ask for human input.
• Traces reconstruct the full journey of a single request across every component it touched. For agents, a trace is not just "which services were hit" — it's the entire decision tree: every prompt sent to the model, every tool invocation, every intermediate thought, and every branch the agent didn't take.

The critical addition agents bring is decision-level granularity. A database call either succeeds or fails; there's no ambiguity about why it happened. An agent's tool call happens because the model decided, based on a prompt and some context, that this was the right next step. If that decision was wrong, you need to see the exact context that led to it — the system prompt, the conversation history at that point, the available tools, and the model's raw output before any parsing.

Without that, "observability" degrades into "logging the final answer," which tells you a bug exists but nothing about where it lives.

There's also a fourth pillar that traditional systems rarely need but agents can't do without: evaluation context. A metrics dashboard tells you *that* latency spiked; a trace tells you *where* in the call graph it spiked. Neither tells you whether the agent's underlying decision was actually correct given the information it had. That's a qualitative judgment, and it usually requires a human — or a second model acting as a judge — reviewing the trace alongside the ground truth. Good agent observability tooling treats this as a first-class workflow: traces should be easy to pull into an evaluation set, annotate with a verdict, and feed back into your test suite, not just stare at in a dashboard.

The anatomy of an agent trace

Think of a trace as a tree. The root span covers the entire user request. Each significant unit of work inside it — a model call, a tool invocation, a retrieval step, a sub-agent delegation — becomes a child span. Spans have a start time, an end time, a status, and attributes describing what happened.

A well-instrumented agent trace typically has this shape:

Trace: handle_refund_request (root span)
├── span: llm_call (planning)
│     attributes: model=gpt-4.1, prompt_tokens=812, completion_tokens=140
├── span: tool_call (lookup_order)
│     attributes: order_id=ORD-9931, latency_ms=212, status=success
├── span: llm_call (reasoning over order data)
│     attributes: model=gpt-4.1, prompt_tokens=1204, completion_tokens=96
├── span: tool_call (check_refund_policy)
│     attributes: latency_ms=88, status=success
├── span: llm_call (decision)
│     attributes: decision=approve_refund, confidence=high
└── span: tool_call (issue_refund)
      attributes: amount=49.99, latency_ms=340, status=success

Each span in that tree is a unit you can query independently: "show me every issue_refund span where amount exceeds $500 and confidence was low." That query is impossible if all you have is a single log line per request.

Here's what that looks like as actual instrumentation code, using a simple span-based tracer you can drop into any Python agent loop without adopting a full observability platform on day one:

import time
import uuid
import json
from contextlib import contextmanager
from dataclasses import dataclass, field

@dataclass
class Span:
    name: str
    span_id: str
    parent_id: str | None
    start_ts: float
    end_ts: float | None = None
    attributes: dict = field(default_factory=dict)
    status: str = "in_progress"

class Tracer:
    def __init__(self):
        self.spans: list[Span] = []
        self._stack: list[str] = []

    @contextmanager
    def span(self, name: str, **attributes):
        span_id = str(uuid.uuid4())
        parent_id = self._stack[-1] if self._stack else None
        s = Span(
            name=name,
            span_id=span_id,
            parent_id=parent_id,
            start_ts=time.time(),
            attributes=attributes,
        )
        self.spans.append(s)
        self._stack.append(span_id)
        try:
            yield s
            s.status = "success"
        except Exception as e:
            s.status = "error"
            s.attributes["error_message"] = str(e)
            raise
        finally:
            s.end_ts = time.time()
            self._stack.pop()

    def export(self) -> str:
        return json.dumps([s.__dict__ for s in self.spans], default=str)


tracer = Tracer()

def run_agent_step(user_query: str):
    with tracer.span("handle_request", query=user_query):
        with tracer.span("llm_call", phase="planning") as plan_span:
            plan = call_model(user_query)
            plan_span.attributes["completion_tokens"] = plan.usage.completion_tokens

        with tracer.span("tool_call", tool="lookup_order") as tool_span:
            result = lookup_order(plan.order_id)
            tool_span.attributes["order_id"] = plan.order_id
            tool_span.attributes["result_status"] = result.status

        with tracer.span("llm_call", phase="decision") as decision_span:
            decision = call_model_with_context(user_query, result)
            decision_span.attributes["decision"] = decision.action

    return tracer.export()

This is deliberately minimal — no external dependencies, just nested context managers that build a span tree with parent-child relationships preserved through a stack. In production you'd swap this for OpenTelemetry or a vendor SDK, but the *shape* of the data — nested spans with attributes and status — stays the same regardless of backend.

Instrumenting the reasoning layer, not just the tools

Most teams that add "observability" to their agent stop at tool calls. They log every API request and response, which is useful, but it misses the layer where most agent bugs actually originate: the reasoning between tool calls.

The model's raw output before you parse it into a structured action is one of the highest-value things you can capture. If your agent parses a JSON tool call out of a model response and that parsing fails, you need the raw text to understand why — was it a formatting quirk, a hallucinated field name, or a genuine misunderstanding of the task?

def parse_tool_call(raw_output: str, span: Span) -> dict:
    span.attributes["raw_model_output"] = raw_output[:2000]  # cap for storage
    try:
        parsed = json.loads(raw_output)
        span.attributes["parse_status"] = "clean"
        return parsed
    except json.JSONDecodeError:
        # Fall back to extracting a JSON block from noisy output
        import re
        match = re.search(r"\{.*\}", raw_output, re.DOTALL)
        if match:
            span.attributes["parse_status"] = "recovered"
            return json.loads(match.group(0))
        span.attributes["parse_status"] = "failed"
        raise ValueError(f"Could not parse tool call from: {raw_output[:200]}")

Notice the parse_status attribute. Over a thousand requests, if recovered and failed climb as a share of total calls, that's a leading indicator of prompt drift or a model version change — long before users start complaining. This is the kind of signal that never shows up if you only log final answers.

Also worth capturing at this layer: the full set of tools available to the model at decision time, and which one it actually chose. Agents frequently misfire not because the model is "wrong" in the abstract, but because two tools have overlapping descriptions and the model picks the less appropriate one. You can't diagnose that without knowing the full menu it was choosing from.

There's a second layer worth instrumenting that teams often skip entirely: the intermediate scratchpad or chain-of-thought summary, if your agent architecture produces one. Many agent frameworks separate an internal "reasoning" field from the final structured action — a short explanation of *why* the model chose this tool over the alternatives. Capturing that field, even truncated, turns a mystery decision into a readable sentence. When a support engineer is triaging a bad refund three weeks after the fact, "the model reasoned that the customer's order qualified under the extended holiday return window" is worth more than any amount of raw JSON. Treat this reasoning trace as a first-class attribute on the span, not a debug-only nicety you strip out before shipping.

Structured events versus free-text logs

A recurring mistake is treating agent logs like traditional application logs — strings with some interpolated variables. That works for humans skimming a terminal, but it fails the moment you want to aggregate or alert on anything.

Instead, emit structured events with a consistent schema:

import logging
import json
from datetime import datetime, timezone

logger = logging.getLogger("agent.events")

def emit_event(event_type: str, trace_id: str, span_id: str, **payload):
    event = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "event_type": event_type,
        "trace_id": trace_id,
        "span_id": span_id,
        **payload,
    }
    logger.info(json.dumps(event))

# Usage inside the agent loop
emit_event(
    "tool_selected",
    trace_id="trc_8831",
    span_id="spn_0042",
    tool_name="issue_refund",
    available_tools=["issue_refund", "escalate_to_human", "deny_refund"],
    model_confidence="high",
)

emit_event(
    "guardrail_triggered",
    trace_id="trc_8831",
    span_id="spn_0043",
    guardrail_name="max_refund_amount",
    threshold=500,
    attempted_value=750,
    action_taken="blocked",
)

Structured events like these plug directly into log pipelines (Elasticsearch, ClickHouse, BigQuery, whatever you already run) without a custom parser. You can build a dashboard panel that's literally count(event_type = "guardrail_triggered") group by guardrail_name and get a live view of which safety rails are actually earning their keep.

A practical taxonomy of event types worth standardizing across your codebase:

• llm_call_started / llm_call_completed — with token counts and latency
• tool_selected — which tool, out of what menu, at what confidence
• tool_call_completed — success/failure, latency, retry count
• guardrail_triggered — which rule, what value, what action
• human_handoff — why the agent escalated
• state_transition — for agents with an explicit state machine or plan
• final_answer_emitted — the terminal event for the trace

Keeping this taxonomy small and consistent matters more than making it exhaustive. Ten well-named event types that every engineer on the team actually uses beat forty ad hoc ones that only the original author remembers.

Correlation IDs: the thread that ties it all together

None of the above works unless every span and event carries a trace ID that survives across process boundaries, retries, and — critically — across sub-agents. If your architecture has a planner agent delegating to a researcher agent delegating to a writer agent, all three need to propagate the same trace ID so you can reconstruct the whole chain as one story instead of three unrelated fragments.

import contextvars

trace_id_var = contextvars.ContextVar("trace_id", default=None)

def get_or_create_trace_id() -> str:
    current = trace_id_var.get()
    if current is None:
        current = f"trc_{uuid.uuid4().hex[:12]}"
        trace_id_var.set(current)
    return current

def call_sub_agent(sub_agent_fn, *args, **kwargs):
    trace_id = get_or_create_trace_id()
    # Propagate explicitly through the call, not just via contextvars,
    # since sub-agents may run in a different process or thread.
    kwargs["_trace_id"] = trace_id
    return sub_agent_fn(*args, **kwargs)

If your agents run across process boundaries (a queue-based worker, a separate microservice per agent role), pass the trace ID as an explicit header or message attribute rather than relying on in-memory context — contextvars doesn't survive a process hop. HTTP calls should carry it as a header like X-Trace-Id; message queue payloads should carry it as a field. The rule is simple: any time control crosses a boundary, the trace ID crosses with it, explicitly, not implicitly.

Catching silent failures: hallucinated tool calls and infinite loops

The most dangerous agent failures are the silent ones — the agent doesn't crash, it just does the wrong thing confidently. Observability's job is to surface these before a user does.

Two patterns worth building detection for directly into your tracing layer:

• Hallucinated tool calls. The model asks to call a tool that doesn't exist, or calls a real tool with parameters that don't match its schema. Log every attempted call before validation, not just the ones that pass — the rejected ones are exactly the signal you need to catch a model regression.
• Loop and repetition detection. An agent that calls the same tool with the same arguments three times in a row is almost certainly stuck, not making progress. This is cheap to detect from span data alone.

def detect_repetition(spans: list[Span], window: int = 3) -> list[str]:
    """Flag traces where the same tool+args combo repeats back to back."""
    warnings = []
    tool_spans = [s for s in spans if s.name == "tool_call"]
    for i in range(len(tool_spans) - window + 1):
        chunk = tool_spans[i:i + window]
        signature = {
            (s.attributes.get("tool"), json.dumps(s.attributes.get("args", {}), sort_keys=True))
            for s in chunk
        }
        if len(signature) == 1:
            warnings.append(
                f"Repeated identical tool call {window}x starting at span {chunk[0].span_id}"
            )
    return warnings

Run this as a post-processing check on completed traces, and alert whenever it fires. In practice, loop detection like this catches a meaningful class of production incidents — agents burning through budget or rate limits while making zero actual progress — well before a cost alert or a user complaint would.

Redaction and cost: observability has a budget too

Two practical constraints will bite you if you ignore them until it's too late.

Redaction. Agent traces routinely contain full prompts, tool arguments, and model outputs — which means they routinely contain PII, credentials, and other sensitive data that flowed through the conversation. Build redaction into the tracer itself, not as an afterthought applied at the dashboard layer:

• Strip or hash fields you know are sensitive (emails, card numbers, order-linked personal data) before the span is persisted, not after.
• Cap the length of any raw text field you store — full multi-thousand-token prompts are rarely worth storing in full; a truncated version plus a content hash for later lookup is usually enough.
• Keep a separate, more tightly access-controlled store for full unredacted traces if you truly need them for deep debugging, and default everything else to the redacted version.

Cost. Tracing itself is not free. Every span, every attribute, every structured event is data you're writing and eventually querying, and at agent-scale request volumes with multi-step loops, trace volume can dwarf your actual application logs by an order of magnitude. Sample aggressively for routine, successful traces, and capture everything for traces that error, get flagged by a guardrail, or exceed a latency threshold. You rarely need full detail on the ten thousand refund requests that went perfectly; you need full detail on the three that didn't.

Turning traces into dashboards that catch regressions

Raw traces are for debugging one incident. Dashboards are for catching the *pattern* before it becomes ten incidents. Once you have structured spans and events flowing somewhere queryable, a handful of panels do most of the work:

• Step count distribution — how many spans per trace, plotted as a histogram. A sudden rightward shift means the agent started looping or over-planning.
• Tool error rate by tool name — which tools are failing, and whether that correlates with a recent API change on the tool side rather than the agent side.
• Parse recovery rate — the clean / recovered / failed split from the parsing example above, tracked over time and across model versions.
• Guardrail trigger rate by rule — rising trigger rates on a specific guardrail often mean the agent is drifting toward edge cases your prompt no longer covers well.
• Cost and latency per completed trace, broken out by decision path — not every path through your agent costs the same, and averaging across all of them hides the expensive branches.

None of these panels require exotic infrastructure. If your spans are structured JSON in any queryable store, these are all standard group by and percentile queries. The hard part was never the dashboard — it was getting disciplined about emitting structured, correlated data in the first place.

It's worth naming the failure mode that skipping this step leads to: teams that add tracing only after an incident tend to over-correct, instrumenting everything at maximum verbosity for a week, then quietly letting it rot because nobody built the dashboard layer on top. Observability that only exists as raw JSON sitting in a log bucket is barely better than no observability — someone has to actually look at it, and they will only look at it regularly if the dashboards make the patterns obvious without a bespoke query every time. Treat the dashboard as part of the same deliverable as the tracer, not a follow-up task for "later."

Building this muscle before you need it

The teams that regret not having agent observability are never the ones who set it up too early. They're the ones who shipped an agent to production, got a support ticket about a wrong decision, and spent two days trying to reconstruct what happened from a single log line and a user's paraphrased description of what the bot said.

Start small: wrap your agent loop in spans, give every trace a propagated ID, emit a handful of structured events at the decision points that matter, and redact before you persist. You don't need a vendor platform on day one — the tracer sketched out earlier in this article is genuinely enough to start seeing your agent's decisions instead of guessing at them. Add the dashboard layer once you have real traffic and real incidents to learn from.

If you want to go deeper into building agents that are debuggable by design — not just prompted well but instrumented well, with tracing, guardrails, and evaluation loops built in from the first commit — that's exactly the ground we cover hands-on in 30 Days of Hermes Agent. It's built around shipping a real agent, one that you can actually watch think, one decision at a time.