LLM Cost Optimization: A Production Engineer's Playbook

LLM cost optimization is not a one-time audit, it is an operating discipline. Teams that run inference at scale treat token spend the way they treat cloud compute: something you measure continuously, route intelligently, and cache aggressively. This guide walks through the concrete levers that move a real invoice: picking the right model for each call, using prompt caching correctly, batching non-urgent work, controlling output length, and building the observability that tells you where the money actually goes.

Everything here assumes you are calling a hosted API (the examples use Claude's Messages API, but the principles transfer to any provider). None of this requires you to sacrifice output quality across the board. Most cost overruns come from using one expensive model for every request regardless of difficulty, not from the model being "too smart."

Why LLM costs spiral in production

Three patterns account for most runaway bills:

• One model for everything. A team picks the most capable model available during a prototype and never revisits the choice. Classification, extraction, and simple Q&A end up paying frontier-model prices for tasks a cheaper model handles just as well.
• No caching of repeated context. System prompts, tool definitions, and retrieved documents get re-sent, re-tokenized, and re-billed on every single request, even when 90% of that content is identical turn to turn.
• Unbounded output. Without a length budget, models write longer responses than the task needs, especially on open-ended or agentic tasks. Output tokens are typically priced several times higher than input tokens, so verbosity is where waste hides.

Fixing these three things usually gets you 40-70% of the possible savings before you touch anything more advanced. The rest comes from batching, smarter routing, and killing silent cache invalidators.

Start with a token budget, not a model preference

Before optimizing anything, get real numbers. Guessing at token counts from character length is unreliable, and using another provider's tokenizer estimate (a common mistake) can be off by 15-30% or more. Use the provider's own token-counting endpoint against representative prompts, not a rule of thumb.

import anthropic

client = anthropic.Anthropic()

result = client.messages.count_tokens(
    model="claude-opus-4-8",
    system="You are a support assistant for a SaaS billing platform.",
    messages=[{"role": "user", "content": "Why was I charged twice this month?"}],
)

print(result.input_tokens)

Run this against a sample of your real traffic, not synthetic test cases. Real prompts include retrieved context, conversation history, and tool schemas that synthetic tests usually omit, so a "quick estimate" from a test script will almost always undercount.

Once you have real input and output token counts per request type, multiply by current per-token pricing to get a per-request cost. That number, not the sticker price of the model, is what you optimize against.

Model selection: match capability to task, not habit

The single highest-leverage decision is which model handles which request. Pricing tiers exist because capability and cost trade off, and most production traffic does not need the top tier.

A reasonable default framework, using Claude's current lineup as a concrete example:

• Highest-capability tier (Claude Opus 4.8): long-horizon agentic work, complex code generation, multi-step reasoning, tasks where a wrong answer is expensive to catch downstream. Priced around $5 per million input tokens and $25 per million output tokens.
• Balanced tier (Claude Sonnet 5): the workhorse for most production traffic, coding, summarization, structured extraction, chat. Priced around $3 per million input tokens and $15 per million output tokens (with an introductory rate close to $2/$10 through the middle of 2026 on some plans, worth checking current pricing before you lock in a cost model).
• Fast, cheap tier (Claude Haiku 4.5): classification, short-form extraction, routing decisions, anything latency-sensitive and semantically simple. Priced around $1 per million input tokens and $5 per million output tokens.

The mistake is treating this as a one-time choice per application. It should be a per-request routing decision.

Build a router, not a default.

A simple, effective pattern: use the cheapest model as a first-pass classifier that decides whether a request needs escalation.

def classify_complexity(user_message: str) -> str:
    """Cheap triage call. Returns 'simple' or 'complex'."""
    response = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=10,
        messages=[{
            "role": "user",
            "content": (
                "Classify this support request as SIMPLE (a factual lookup or "
                "single-step answer) or COMPLEX (requires multi-step reasoning "
                "or code). Reply with one word only.\n\n"
                f"Request: {user_message}"
            ),
        }],
    )
    label = response.content[0].text.strip().upper()
    return "complex" if "COMPLEX" in label else "simple"


def handle_request(user_message: str) -> str:
    tier = classify_complexity(user_message)
    model = "claude-opus-4-8" if tier == "complex" else "claude-sonnet-5"
    response = client.messages.create(
        model=model,
        max_tokens=2048,
        messages=[{"role": "user", "content": user_message}],
    )
    return response.content[0].text

The triage call costs a fraction of a cent and pays for itself the moment it routes even a handful of simple requests away from the expensive model. In practice, most support and internal-tool traffic skews heavily toward "simple," so this single change often cuts blended cost per request by half or more.

If your traffic has a predictable shape (a known ratio of classification vs. generation vs. agentic tasks), you can skip the dynamic classifier and hard-route by endpoint or feature flag instead. The classifier is worth it mainly when request complexity is unpredictable and mixed within the same traffic stream.

Do not default to the newest, largest model out of habit.

When a new model generation ships, it is tempting to swap every call site to the latest flagship immediately. Do the swap, but re-run your cost model afterward. Newer generations sometimes use a different tokenizer that counts the same text as more tokens even at flat or lower per-token pricing, so the actual dollar cost per request can move in either direction. Measure, do not assume.

Prompt caching: the highest ROI lever most teams skip

Prompt caching lets you avoid re-paying for content that repeats across requests: system prompts, tool definitions, retrieved documents, few-shot examples. A cached read typically costs a small fraction of the full input price, while writing to the cache costs a modest premium over a normal request. For any prompt prefix reused more than twice, caching is close to free money.

The mechanics matter. Caching works on an exact prefix match: everything up to a cache marker must be byte-identical to a previous request, or the cache misses entirely and you pay full price plus the write premium for nothing.

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LARGE_KNOWLEDGE_BASE_TEXT,
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": "What's our refund policy for annual plans?"}],
)

print(response.usage.cache_read_input_tokens)
print(response.usage.cache_creation_input_tokens)

Check those two usage fields on every response during development. If cache_read_input_tokens stays at zero across repeated calls with the same prefix, something is silently invalidating your cache and you're paying full price while believing you're saving money.

Common silent cache killers:

• A timestamp or request ID inside the system prompt. Even something as small as "Current date: 2026-07-09" interpolated into the system message changes the prefix on every single call.
• Non-deterministic JSON serialization. If your tool definitions or context blocks are built from a dictionary or set without sorted keys, the serialized order can vary between requests even when the content is logically identical.
• Changing the tool list mid-session. Tools render before the system prompt in the request, so adding, removing, or reordering even one tool invalidates every cache breakpoint that follows.
• Per-user or per-session content baked into the shared prefix. If you personalize the system prompt with a user's name, that prefix is no longer shared, and you lose cross-request caching entirely. Move that personalization after the cache boundary, into the user message or a later block.

Where to put the cache boundary.

Order matters: put stable, shared content first (system prompt, tool definitions, retrieved reference material) and volatile, per-request content last (the actual user question). Place the cache marker at the end of the stable block, not at the end of the whole prompt.

messages = [{
    "role": "user",
    "content": [
        {
            "type": "text",
            "text": SHARED_PRODUCT_DOCS,
            "cache_control": {"type": "ephemeral"},
        },
        {
            "type": "text",
            "text": user_question,  # varies every request, no marker here
        },
    ],
}]

If you marked the boundary at the very end of the prompt instead, every request would write a distinct cache entry that nothing ever reads back, and you'd pay the write premium on every call with zero benefit.

For long-running agent sessions with many tool calls per turn, be aware that cache lookups only search back a limited number of content blocks (roughly the last 20). In agentic loops with heavy tool use, place an intermediate cache marker every 10-15 blocks rather than relying on a single marker from the start of the session.

Batch what does not need to be instant

Not every LLM call is user-facing and latency-sensitive. Nightly report generation, bulk classification, backfilling embeddings-adjacent metadata, offline evaluation runs, all of these can tolerate completion within a few hours instead of a few seconds. Batch processing APIs typically run at roughly half the price of synchronous calls in exchange for asynchronous turnaround.

from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request

requests = [
    Request(
        custom_id=f"ticket-{ticket_id}",
        params=MessageCreateParamsNonStreaming(
            model="claude-haiku-4-5",
            max_tokens=50,
            messages=[{
                "role": "user",
                "content": f"Classify sentiment (positive/negative/neutral): {ticket_text}",
            }],
        ),
    )
    for ticket_id, ticket_text in overnight_tickets
]

batch = client.messages.batches.create(requests=requests)

Results come back keyed by custom_id, not in submission order, so build your result handler around that key rather than assuming positional ordering. Most batches complete well within an hour, with a generous outer window for the rare slow batch, so this fits naturally into nightly cron jobs or queue-driven backfills.

A good rule: if a human is not waiting on the response in real time, it belongs in a batch. Audit your call sites for anything that runs on a schedule or in a background worker and is still using the synchronous endpoint out of inertia.

Control output length deliberately

Output tokens cost several times more than input tokens on most pricing tiers, which means verbosity is one of the most expensive silent costs in an LLM application. A model that writes a six-paragraph answer to a question that needed two sentences is burning money on every single call, not just the occasional one.

Two levers matter here:

• Set `max_tokens` to a realistic ceiling for the task, not a generously high default "just in case." A classification task needs tens of tokens, not thousands. A short chat reply needs hundreds, not tens of thousands.
• Instruct the model explicitly on desired length and format. Models generally follow explicit length and formatting instructions closely; a plain instruction to be concise, skip preamble, and avoid restating the question usually produces measurably shorter, cheaper responses without losing the substance of the answer.

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=300,
    system=(
        "Answer in 2-4 sentences. Do not restate the question. "
        "Do not add a closing summary or offer of further help."
    ),
    messages=[{"role": "user", "content": user_question}],
)

If you are running an agentic workflow with effort or thinking controls available, treat those as cost dials too. Lower effort settings produce fewer, more consolidated tool calls and terser output, at some cost to thoroughness. For routine or well-scoped tasks, a lower effort setting is often indistinguishable in quality from the maximum setting but noticeably cheaper and faster. Reserve the highest effort tier for tasks where getting it wrong is expensive to discover later, and default everything else to a middle setting, adjusting up only when you measure a quality gap.

Streaming is a UX fix, not a cost fix, but it prevents a costly failure mode

Streaming does not change token pricing. What it does is prevent a specific expensive failure: a non-streaming request configured with a very large max_tokens value can hit HTTP timeout limits and fail entirely after the model has already generated (and you've been billed for) a large chunk of output, forcing a full retry that pays for the same tokens twice. For any request where output length could reasonably run long, stream it and use the SDK's "final message" helper to still get the complete response object at the end. This is a reliability fix that happens to protect your cost line, not a direct savings lever.

Build the observability that catches waste before the invoice does

You cannot optimize what you cannot see per request. The minimum viable cost observability setup logs, for every LLM call:

• Model used
• Input tokens, output tokens, cache read tokens, cache creation tokens
• Wall-clock latency
• A tag identifying the calling feature or endpoint

import logging

def logged_completion(model: str, feature_tag: str, **kwargs):
    response = client.messages.create(model=model, **kwargs)
    usage = response.usage
    logging.info(
        "llm_call feature=%s model=%s input=%d output=%d cache_read=%d cache_write=%d",
        feature_tag,
        model,
        usage.input_tokens,
        usage.output_tokens,
        usage.cache_read_input_tokens,
        usage.cache_creation_input_tokens,
    )
    return response

Feed this into whatever metrics pipeline you already run, and build one dashboard: cost per feature per day. This single view tends to surface the two most common problems immediately: a feature that quietly regressed from a cached, cheap-model call to an uncached, expensive-model call after a refactor, and a feature whose output length crept up over time as prompts accumulated instructions without anyone revisiting the length budget.

Set an alert on total daily spend and on cost-per-feature week-over-week change. A 3x spike in cost for one feature, caught the day it happens, is a five-minute fix. The same spike discovered at the end of the billing cycle is a much larger conversation.

Prompt engineering as a cost lever, not just a quality lever

A tighter, more specific system prompt often produces better *and* cheaper output, because the model spends fewer tokens exploring what you actually want. Two concrete techniques:

• Move from few-shot examples to explicit rules where possible. Few-shot examples are token-expensive on every single request (unless cached). If the pattern you're demonstrating can be stated as a direct instruction instead, that trades a repeated per-request cost for a one-time prompt-design cost.
• Cut instructions that are no longer load-bearing. Prompts accumulate cruft over months of iteration: guardrail language added for an edge case that got fixed elsewhere, formatting instructions superseded by a later instruction, redundant emphasis. Periodically re-read your production system prompts and remove anything that isn't earning its token cost. This is cheap to do and compounds on every request that prompt serves.

A practical rollout order

If you're starting from an unoptimized system, this is the order that tends to produce the fastest, lowest-risk wins:

1. Add usage logging to every call site (a day of work, zero risk, unlocks everything else).
2. Cache the largest shared prefixes (system prompts, tool definitions, retrieved reference docs).
3. Add a cheap-model triage step in front of any endpoint with mixed-complexity traffic.
4. Set realistic max_tokens ceilings and length instructions on every call site.
5. Move anything non-real-time to the batch API.
6. Re-audit system prompts for token bloat, and re-check your model choice after any generation upgrade.

Each step is independently valuable and none of them requires touching the others first, so you can parallelize this across a team rather than doing it as one long sequential project.

FAQ

Does using a cheaper model always mean lower quality? Not for tasks within that model's capability range. Classification, short extraction, routing, and simple Q&A are frequently handled just as well by a fast, low-cost model as by the most capable tier. Quality loss shows up when you push a genuinely hard, multi-step reasoning task onto a model sized for simple tasks, not when you right-size the model to the task.

How much can prompt caching actually save? It depends entirely on how much of your prompt is shared across requests and how often that shared prefix repeats. A system with a large, stable system prompt and high request volume against it can see the cached portion of input cost drop to a small fraction of the uncached price. A system where every prompt is mostly unique per-request content will see little benefit, because there's nothing stable to cache.

Is batch processing worth it for a small number of requests? The savings percentage is the same regardless of volume, but the operational overhead of managing an asynchronous batch job (polling for completion, handling out-of-order results) is more worth it once you have a meaningful, recurring volume of non-urgent requests. For a handful of ad hoc calls, the synchronous API is simpler and the savings are not worth the added complexity.

Should I set `temperature` low to save money? No. Sampling parameters like temperature control output randomness, not cost. They have no direct effect on token pricing. Cost control comes from model choice, caching, output length, and batching, not sampling settings.

How often should I re-audit my model choices? Whenever pricing changes, whenever a new model generation ships, and on a fixed quarterly cadence regardless of either. Traffic patterns shift over time even when nothing else changes, and a routing decision that was optimal six months ago may no longer be optimal against your current mix of request types.

What's the single biggest mistake teams make with LLM costs? Treating the model choice as fixed at prototype time and never building the observability to see per-feature, per-request cost. Without that visibility, every other optimization is a guess. Log usage from day one, even before you think you need to.