Agent Cost Optimization: Cutting Token Spend Without Losing Quality

Why your agent bill grows faster than your usage

Somewhere around the third week of running an agent in production, someone on the team opens the billing dashboard and asks a question nobody wants to answer: why did the token spend triple when the number of users only went up ten percent? The answer is almost never "the model got more expensive." It's that agents burn tokens in ways that are invisible until you go looking. Every tool call re-sends the conversation history. Every retry after a malformed JSON response re-sends the whole prompt again. Every "just to be safe" step where the agent re-reads a file it already read three turns ago quietly compounds.

This isn't a niche operational detail — it's the difference between an agent architecture that scales and one that gets shut down in a cost review. Agent cost optimization is the discipline of controlling that growth curve deliberately, instead of discovering it by accident when Finance forwards you an invoice. The good news is that the fixes are well understood, mostly mechanical, and rarely require touching your prompts' actual instructions. The bad news is that most teams don't apply them until the pain is already expensive.

This piece walks through where agent token spend actually goes, then through the concrete levers — caching, model routing, context pruning, batching, and structural changes to how agents plan and execute — that bring the bill down without making the agent dumber. We'll use real code patterns you can adapt today, not abstract advice.

Where the tokens actually go

Before optimizing anything, it helps to be precise about what you're paying for. In a typical agentic loop, tokens accumulate in four places:

• Input tokens on every turn. Most agent frameworks re-send the entire conversation history — system prompt, prior tool calls, prior tool results, prior assistant messages — on every single model call. A 10-turn agent loop with a 2,000-token system prompt and growing tool outputs can easily be paying for that system prompt ten times over.
• Tool output tokens. Search results, file contents, API responses, and stack traces get dumped into context verbatim. A single git diff or database query result can be tens of thousands of tokens, most of which the model never actually needs.
• Output tokens for over-verbose reasoning. Models asked to "think step by step" without constraints will happily produce long chains of reasoning even for simple lookups, and that reasoning gets billed.
• Retries and re-plans. Malformed tool calls, hallucinated function signatures, and failed validation all trigger another full round trip — another full context re-send plus new output.

The mistake most teams make is optimizing the fourth category first (better prompts, better error messages) when the first two categories are usually 70-80% of the actual spend. Fix the structural stuff first.

Lever one: prompt caching is not optional

If you're running an agent against Claude, OpenAI, or most major providers, and you are not using prompt caching, you are leaving money on the table before you've done anything else. Prompt caching lets you mark a prefix of your context — typically the system prompt, tool definitions, and any large static reference material — as cacheable, so repeated calls only pay full price the first time and a fraction of that price on subsequent calls within the cache window.

The mechanical requirement is that the cached prefix has to be identical across calls, and it has to come first in the context, before anything variable. This sounds obvious but it's the number one reason caching doesn't kick in for teams who "already tried it." If you interpolate a timestamp, a user ID, or a session-specific instruction into your system prompt, you've broken the cache on every single call.

import anthropic

client = anthropic.Anthropic()

# Static, reusable parts go into cache_control blocks.
# Anything that changes per-request (user query, session state)
# must come AFTER the cached blocks, never inside them.
system_blocks = [
    {
        "type": "text",
        "text": AGENT_SYSTEM_PROMPT,  # large, unchanging instructions
        "cache_control": {"type": "ephemeral"}
    },
    {
        "type": "text",
        "text": TOOL_USAGE_GUIDE,  # unchanging reference doc
        "cache_control": {"type": "ephemeral"}
    }
]

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system=system_blocks,
    tools=TOOL_DEFINITIONS,  # also stable — benefits from caching upstream of it
    messages=conversation_history
)

The practical checklist for making caching actually work:

• Put everything static — system instructions, tool schemas, few-shot examples, style guides — at the front of the prompt, in a fixed order, every time.
• Never inject dynamic content (current date, user name, session ID) into the cached prefix. Put it in the first user message instead.
• Structure multi-turn agent loops so the growing conversation history is appended after the cached prefix, not interleaved with it.
• Watch your cache hit rate in the API response metadata. If it's near zero, something in your "static" prompt is actually changing between calls — go find it.

For an agent that makes many tool calls per user request (the common case), this single change often cuts effective input cost by half or more, because the system prompt and tool definitions are the same tokens being re-billed on every loop iteration.

Lever two: route by task, not by default

The second-biggest lever is refusing to use your most expensive model for every step of an agent's work. Agent pipelines have a natural division of labor: planning and complex reasoning need a strong model, but classification, extraction, formatting, and simple tool-argument construction usually don't.

A common anti-pattern is wiring the entire agent loop — planner, tool-argument generator, summarizer, output formatter — to the same top-tier model because it was the easiest thing to configure. That's paying frontier-model prices for work a cheaper, faster model handles identically.

def get_model_for_task(task_type: str) -> str:
    """Route each agent sub-task to the cheapest model that
    reliably handles it. Reassess this mapping whenever you
    change providers or a new model tier ships."""
    routing = {
        "plan":            "claude-opus-4",     # complex multi-step reasoning
        "tool_call":       "claude-sonnet-5",    # structured, well-defined task
        "summarize":       "claude-haiku-4",     # compression, low ambiguity
        "extract_fields":  "claude-haiku-4",     # deterministic-ish extraction
        "final_response":  "claude-sonnet-5",    # user-facing quality matters
    }
    return routing.get(task_type, "claude-sonnet-5")

A few practical notes on making routing safe rather than reckless:

• Route on task shape, not vibes. "Does this step require judgment across multiple pieces of context, or is it a narrow transformation?" is a better question than "is this an important step?" Formatting a final answer for the user feels important but is often narrow.
• Add a confidence-based escalation path. If your cheap-model step returns something malformed, invalid JSON, or a confidence signal, retry once with the stronger model rather than looping on the weak one. This gives you most of the savings with a safety net.
• Measure quality per route, not just per model. Run your eval set through each routed sub-task and confirm the cheaper model's error rate on that specific narrow task is acceptable — not its general benchmark score.
• Revisit the routing table on a schedule. Model pricing and capability shift every few months; a routing table you set once and never touched is a routing table that's probably wrong today.

Routing done well typically saves 30-50% of total spend on multi-step agents, because most agent steps are narrow, structured tasks even when the overall goal is complex.

Lever three: prune context aggressively, not politely

Agents accumulate context like a junk drawer. Every tool result gets appended to the conversation and, absent explicit management, stays there for the rest of the run — even after the agent has extracted what it needed and moved on. This is the single largest source of "why is this simple task costing so much" complaints.

The fix is to treat context as a working set you actively manage, not an append-only log.

class ContextManager:
    """Keeps the agent's working context lean by summarizing
    or dropping tool outputs once they're no longer load-bearing."""

    def __init__(self, max_tool_output_tokens=800):
        self.max_tool_output_tokens = max_tool_output_tokens

    def compact_tool_result(self, tool_name: str, raw_output: str) -> str:
        if self._estimate_tokens(raw_output) <= self.max_tool_output_tokens:
            return raw_output

        if tool_name == "file_read":
            # keep the relevant slice, not the whole file
            return self._truncate_with_marker(raw_output, self.max_tool_output_tokens)
        if tool_name == "web_search":
            # keep titles + snippets, drop full page bodies
            return self._extract_snippets(raw_output)
        if tool_name == "db_query":
            # keep schema + first N rows, note the total count
            return self._summarize_rows(raw_output)

        return self._truncate_with_marker(raw_output, self.max_tool_output_tokens)

    def prune_history(self, messages: list, keep_last_n_full: int = 3) -> list:
        """Collapse older tool results into short summaries once
        the agent has moved past the step that needed them."""
        pruned = []
        for i, msg in enumerate(messages):
            is_recent = i >= len(messages) - keep_last_n_full
            if msg["role"] == "tool" and not is_recent:
                pruned.append({**msg, "content": self._one_line_summary(msg["content"])})
            else:
                pruned.append(msg)
        return pruned

    def _estimate_tokens(self, text: str) -> int:
        return len(text) // 4  # rough heuristic; swap for a real tokenizer

    def _truncate_with_marker(self, text: str, max_tokens: int) -> str:
        char_limit = max_tokens * 4
        if len(text) <= char_limit:
            return text
        return text[:char_limit] + "\n...[truncated, original length: {} chars]".format(len(text))

    def _extract_snippets(self, raw_output: str) -> str:
        # placeholder for real snippet-extraction logic
        return raw_output[:2000]

    def _summarize_rows(self, raw_output: str) -> str:
        # placeholder for real row-summarization logic
        return raw_output[:2000]

    def _one_line_summary(self, content: str) -> str:
        return content[:120] + ("..." if len(content) > 120 else "")

Two ideas do most of the work here. First, compact at the source — don't let a raw 40,000-token file dump enter the conversation at all; extract the slice the agent actually needs before it ever hits the context window. Second, decay old context — a tool result that mattered for step 2 usually doesn't need to still be sitting there verbatim at step 9; collapse it to a one-line summary once the agent has moved on, and keep full detail only for the most recent few turns.

This is also where retrieval matters: instead of stuffing an entire knowledge base or document set into context "to be safe," retrieve only the passages relevant to the current sub-task. An agent that re-fetches narrow, relevant slices on demand will almost always out-cost (and often out-perform) one that front-loads everything.

Lever four: stop paying for retries you caused yourself

A large, quiet chunk of agent spend comes from self-inflicted retries: the agent calls a tool with a malformed argument, the call fails validation, and the entire loop re-runs — re-sending the full context plus the error, then re-generating the reasoning from scratch. If this happens even 10% of the time across a high-volume agent, it's a meaningful tax.

The fix is tightening the contract between the model and your tools so failures are rarer, and making the failures cheap when they do happen.

from pydantic import BaseModel, ValidationError

class SearchArgs(BaseModel):
    query: str
    max_results: int = 10
    date_from: str | None = None

def validate_and_repair(tool_call: dict) -> tuple[dict | None, str | None]:
    """Validate tool arguments locally before spending a model
    call on execution. Return a short, specific error the model
    can fix in its NEXT message, instead of re-running the whole plan."""
    try:
        args = SearchArgs(**tool_call["arguments"])
        return args.model_dump(), None
    except ValidationError as e:
        # Give the model a minimal, precise correction hint —
        # not the full validation traceback, which burns tokens
        # and doesn't help the model fix it any faster.
        first_error = e.errors()[0]
        hint = f"Fix '{first_error['loc'][0]}': {first_error['msg']}"
        return None, hint

Practical guardrails that reduce retry-driven spend:

• Validate tool arguments locally, before the call executes, using strict schemas (Pydantic, Zod, JSON Schema). Catching a bad argument locally costs nothing; catching it after a failed API call costs a full round trip.
• Return short, actionable error messages, not stack traces. A precise one-line hint fixes the next generation faster and cheaper than a wall of text the model has to parse.
• Cap retries per tool call (two or three, not unlimited) and fall back to asking the user or escalating to a stronger model rather than looping.
• Log retry rate per tool as a first-class metric. A tool with a 20% failure rate is a cost problem hiding in plain sight, and it's usually fixable by tightening its schema or its description in the tool definition.

Lever five: batch and parallelize where the workflow allows it

Not every agent task needs a live, synchronous response. If you're running agents for report generation, data enrichment, bulk classification, or overnight processing jobs, batching APIs typically offer a substantial discount over standard synchronous calls in exchange for higher latency — often used for background pipelines and evaluation runs rather than interactive chat.

# Pseudocode pattern: route latency-tolerant workloads to batch processing
def submit_agent_job(task, urgency: str):
    if urgency == "interactive":
        return run_sync(task)          # standard pricing, immediate response
    elif urgency == "background":
        return submit_to_batch_api(task)  # discounted pricing, async result

The decision rule is simple: does a human need this answer in the next few seconds? If not, it belongs in a batch queue. Teams often run their entire evaluation suite, nightly data-labeling job, or content-generation backlog synchronously purely out of habit, when none of it needs to be. Separating "must respond now" from "can respond in an hour" is a free cost reduction that requires zero changes to prompt quality.

Parallelizing independent sub-tasks (rather than looping through them sequentially in one long agent conversation) also helps, because each parallel branch can carry a smaller, more focused context instead of one giant conversation accumulating everything from every sub-task.

Lever six: measure before and after, per task type

None of the above matters if you can't see whether it worked. Cost optimization without measurement is just vibes with extra steps, and it's easy to "fix" one thing while quietly making another worse — for instance, pruning context so aggressively that the agent starts failing tasks and triggering more retries, which erases the savings.

The minimum viable tracking setup:

• Tokens in / tokens out, per task type, not just per request. "Support ticket triage" and "contract summarization" have very different cost profiles and need to be tracked separately.
• Cache hit rate, tracked continuously — this is your canary for prompt-structure regressions that silently kill your caching savings.
• Cost per successful task completion, not cost per API call. A cheap model that needs three retries can cost more per completed task than an expensive model that succeeds on the first try.
• Quality score alongside cost, from your existing eval suite. Every optimization above should be validated against the same quality bar you had before — cost reduction that trades away correctness isn't optimization, it's just cutting corners with a spreadsheet to justify it.

def log_task_metrics(task_id, model, tokens_in, tokens_out,
                      cache_read_tokens, retries, success, eval_score):
    metrics_store.record({
        "task_id": task_id,
        "model": model,
        "tokens_in": tokens_in,
        "tokens_out": tokens_out,
        "cache_read_tokens": cache_read_tokens,
        "effective_cache_hit_rate": cache_read_tokens / max(tokens_in, 1),
        "retries": retries,
        "success": success,
        "eval_score": eval_score,
        "cost_per_success": estimate_cost(model, tokens_in, tokens_out) / max(int(success), 0.01)
    })

Run this for a week before you touch anything, then again after each lever you apply, one at a time. Changing caching, routing, and pruning all in the same deploy makes it impossible to know which change earned you the savings and which one quietly hurt quality.

Putting it together: a realistic optimization order

If you're starting from an unoptimized agent, the sequence that tends to produce the best results with the least risk is:

1. Turn on prompt caching correctly (static prefix, stable ordering, dynamic content moved out). This alone is often the biggest single win.
2. Add per-task-type cost and quality logging so every subsequent change is measurable.
3. Introduce model routing for the narrow, structured sub-tasks in your pipeline, keeping the strongest model only where reasoning quality is genuinely load-bearing.
4. Add context compaction and pruning for tool outputs, starting with whichever tool produces the largest average output.
5. Tighten tool-argument validation to cut self-inflicted retries.
6. Move any non-interactive workload to batch processing.

Each of these is independently valuable, and none of them require you to rewrite your agent's core logic or degrade the experience your users actually see. The instructions you give the model don't change. The plan the agent follows doesn't change. What changes is how much of the surrounding machinery — repeated system prompts, bloated tool outputs, misrouted model calls, and self-inflicted retries — you're willing to keep paying for by default.

Cost optimization for agents isn't a one-time cleanup, either. New tools get added, context windows grow, and the sub-task mix shifts as your agent takes on more responsibility. Treat the metrics from lever six as a permanent dashboard, not a one-off audit, and revisit the routing table and pruning thresholds whenever your usage pattern changes meaningfully.

If you want to go deeper on building agents that are efficient by design — not just patched after the fact — that's exactly the kind of production-grade agent engineering we cover hands-on in 30 Days of Hermes Agent, our course on building, evaluating, and running real agentic systems from first principles.