Prompt EngineeringLLM cost optimizationcontext windowprompt cachingtokenization

Prompt Token Optimization: A Practical Guide for Engineers Cutting LLM Costs

Pramod Dutta · Jun 23, 2026 · 13 min read

Prompt token optimization means reducing the number of tokens you send to and receive from a language model without degrading the quality of its output. For working engineers, this is not an academic exercise: token count drives latency, cost, and how much of the context window is left for the model to actually reason with. A production system that ships bloated prompts pays for it three times over, in dollars, in response time, and in accuracy, because a model buried under irrelevant text has less room to focus on what matters.

This guide walks through how tokenization actually works, where prompts bleed tokens without anyone noticing, and the concrete techniques (with code) to fix it: trimming instructions, restructuring few-shot examples, compressing retrieved context, using prompt caching, and choosing the right output format. Everything here is runnable today against any major model API.

Why prompt token optimization matters beyond cost

Every request to an LLM API is billed per token, input and output, usually at different rates. Input tokens are almost always cheaper than output tokens, sometimes by 3-5x, which changes your optimization priorities immediately: it is often cheaper to send a longer, more explicit prompt that forces a short, structured answer than to send a lean prompt that produces a rambling one.

But cost is only part of the story. Two other effects matter just as much:

Latency. Time to first token and total generation time both scale with token count. A prompt with 8,000 tokens of boilerplate instructions adds real, measurable latency before the model even starts on your actual question.
Context window pressure. Every model has a finite context window. Even on models with huge windows, performance on needle-in-a-haystack style tasks degrades as you fill more of that window with irrelevant material. This is sometimes called "context rot": the model's attention gets diluted, and it becomes more likely to miss or misweight the one paragraph that actually answers the question. Prompt token optimization is therefore also an accuracy lever, not just a cost lever.

If you are building anything with a retrieval step, an agent loop, or a long system prompt, all three of these compound. A chatbot that re-sends the full conversation history on every turn, an agent that appends every tool result to its context forever, or a RAG pipeline that stuffs in ten chunks when three would do, all pay this tax on every single call.

Understand tokenization before you optimize

You cannot optimize what you cannot measure. Tokens are not words and they are not characters; they are subword units produced by a byte-pair-encoding-style tokenizer. Roughly:

English prose: about 4 characters per token, or about 0.75 tokens per word.
Code: often *more* tokens per character than prose, because indentation, punctuation, and camelCase/snake_case identifiers split into more pieces.
JSON and other structured formats: the quotes, braces, colons, and repeated keys all cost tokens. A JSON array of ten objects with the same five keys repeats those key names ten times.

Use the model provider's actual tokenizer to count, not a word count estimate. Most Python SDKs expose a count_tokens helper, or you can use a standalone library:

pip install tiktoken

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def count_tokens(text: str) -> int:
    return len(enc.encode(text))

system_prompt = open("system_prompt.txt").read()
print(f"System prompt: {count_tokens(system_prompt)} tokens")

If you are calling Claude, use the Anthropic SDK's token counting endpoint directly, since it accounts for the exact tokenizer and message formatting the model will see:

import anthropic

client = anthropic.Anthropic()

result = client.messages.count_tokens(
    model="claude-sonnet-4-5",
    system=system_prompt,
    messages=[{"role": "user", "content": user_query}],
)
print(result.input_tokens)

Run this in CI on your prompt templates so a regression (someone pastes a full stack trace into the system prompt "just for now") shows up as a diff in token count, not as a surprise on the invoice.

Technique 1: Trim the system prompt ruthlessly

System prompts accumulate cruft over time. Every bug fix adds another sentence ("if the user asks about refunds, do NOT mention the old policy"), and nobody ever removes the sentence that fixed the bug before it. Audit your system prompt the way you'd audit dead code:

Delete instructions the model already follows by default. You do not need to tell a modern model "respond in a professional tone" or "do not use offensive language" unless you have evidence it's actually a problem for your use case.
Collapse repeated examples into one canonical pattern. If you have five few-shot examples that all demonstrate the same transformation with different data, keep two: one typical case, one edge case. More examples have diminishing returns past 3-5 for most tasks, and each one costs you tokens on every single request forever.
Move rarely-needed instructions out of the always-sent system prompt and into conditional context. If 5% of requests are refund-related, don't ship refund policy text in every request; inject it only when a classifier or router detects a refund-related query.

A before/after example:

BEFORE (187 tokens):
You are a helpful, friendly, and knowledgeable customer support
assistant for our company. Please always be polite and professional
in your responses. Never be rude to customers. Always try your best
to help the customer with their issue. If you don't know the answer,
it's okay to say you don't know, but try to be helpful about it and
suggest next steps. Please format your responses nicely. Remember to
always be empathetic and understanding towards customer concerns.

AFTER (34 tokens):
You are a customer support assistant. If you don't know an answer,
say so and suggest a next step (escalate to a human, check the docs
link).

The trimmed version keeps the one instruction that actually changes model behavior (what to do when uncertain) and drops the rest, which a capable model already does.

Technique 2: Compress few-shot examples and retrieved context

Few-shot examples and RAG chunks are usually the biggest line items in a real production prompt, bigger than the system prompt itself. Two moves help here.

Deduplicate and rank before you stuff. If your retrieval step returns ten chunks, don't send all ten. Re-rank them and send the top three to five, and log how often the answer actually came from a chunk ranked below position five. In most systems that number is very small, and every extra chunk you were sending was pure token waste plus context dilution.

def build_context(chunks: list[dict], max_chunks: int = 4) -> str:
    # chunks already sorted by relevance score, descending
    selected = chunks[:max_chunks]
    return "\n\n".join(
        f"[{c['source']}]\n{c['text'].strip()}" for c in selected
    )

Strip formatting noise from retrieved text. Chunks pulled from HTML, PDFs, or markdown often carry leftover markup, repeated whitespace, and navigation boilerplate ("Skip to content", "Table of Contents", footer links). None of that helps the model and all of it costs tokens.

import re

def clean_chunk(text: str) -> str:
    text = re.sub(r"\s+", " ", text)               # collapse whitespace
    text = re.sub(r"\[.*?\]\(.*?\)", "", text)      # strip markdown links
    text = re.sub(r"(Skip to content|Table of Contents)", "", text, flags=re.I)
    return text.strip()

Summarize instead of paste for long source documents. If a retrieved document is 3,000 tokens but the relevant fact is one paragraph, either chunk more aggressively at ingestion time (smaller, more targeted chunks) or run a cheap summarization pass before the chunk ever reaches the expensive model call.

Technique 3: Manage conversation history instead of replaying it whole

Chat applications resend the full message history on every turn, since the API is stateless. Left unmanaged, a ten-turn conversation costs as much on turn ten as if you'd sent the entire transcript fresh, times ten. Three patterns fix this:

Sliding window. Keep only the last N turns verbatim, and drop anything older. Works well for support chat where old turns rarely matter.
Rolling summary. After every few turns, replace the oldest turns with a short model-generated summary ("User is troubleshooting a failed deploy on staging; already confirmed DNS and cert are fine; still checking env vars"). This preserves the gist at a fraction of the token cost.
Explicit memory extraction. Pull out durable facts (user's name, plan tier, the specific error code they're debugging) into a small structured memory block, and drop the rest of the raw transcript entirely.

def trim_history(messages: list[dict], keep_last: int = 6) -> list[dict]:
    if len(messages) <= keep_last:
        return messages
    older = messages[:-keep_last]
    recent = messages[-keep_last:]
    summary = summarize(older)  # your own cheap summarization call
    return [{"role": "system", "content": f"Earlier context: {summary}"}] + recent

Pick the pattern based on how much old context actually matters to your task. Debugging assistants often need the summary approach; simple FAQ bots can get away with a plain sliding window.

Technique 4: Use prompt caching for repeated context

If the same large block of context (a system prompt, a document, a tool schema list) is sent on every request, prompt caching lets the provider skip reprocessing it and bill the cached portion at a steep discount on subsequent calls. This is the single highest-leverage optimization for agentic and RAG systems, because it turns "large static context" from a cost problem into a near-free one.

The pattern with Claude's API:

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": long_static_instructions_and_docs,
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": user_query}],
)

Structure your prompt so the static, reusable part comes first and is marked for caching, and the dynamic, per-request part (the actual user question) comes last. Caches typically expire after a short idle window, so this pays off most for high-traffic endpoints, not for a script that runs once a day.

Rules of thumb for caching:

Put tool definitions, system instructions, and reference documents in the cached segment.
Put the user's current message and anything unique to this request outside the cache.
Order matters: anything you change between requests must come after the cached block, or you invalidate the cache on every call.

Technique 5: Control output tokens, not just input tokens

Output tokens are usually the more expensive side of the bill, and they're the side most engineers forget to constrain. Three levers:

Ask for structured output instead of prose. A JSON response with three fields is cheaper and easier to parse than "Sure! Here's what I found: the answer is..." Use the API's structured output or tool-calling mode rather than asking the model to format JSON inside free text, which also saves you from writing a fragile parser.

Set `max_tokens` deliberately, not to a huge default. A generous max_tokens doesn't cost you anything if the model stops early, but a runaway generation (a model that loops or over-explains) will burn through it. Set it to a realistic ceiling for the task and monitor cases that hit the limit.

Ask for terse output explicitly when the consumer is a machine, not a human. If a downstream service just needs a classification label, don't let the model narrate its reasoning in the response unless you're using that reasoning for something. Compare:

Prompt: "Classify this support ticket."
Typical output: "Looking at this ticket, it seems like the customer is
experiencing a billing issue. I would classify this as: BILLING"
(35 tokens)

Prompt: "Classify this support ticket. Respond with only the category
label, nothing else: BILLING, TECHNICAL, ACCOUNT, or OTHER."
Typical output: "BILLING"
(1 token)

That 34-token difference, multiplied across a million classification calls a month, is not a rounding error.

Technique 6: Choose the leanest data format for structured input

When you're passing structured data (a list of products, a table of user records) into a prompt, the format you choose changes the token count significantly for the same information. JSON's overhead of repeated keys and punctuation nearly always loses to a compact tabular or delimited format for large repeated records.

JSON (per row, ~28 tokens):
{"id": 1042, "name": "Wireless Mouse", "price": 24.99, "stock": 130}

CSV-style (per row, ~14 tokens):
1042,Wireless Mouse,24.99,130

For a 200-row product table, that difference is thousands of tokens. The tradeoff is that CSV-like formats are less self-describing, so send a one-line header explaining the column order once, not per row, and pick JSON when the structure is nested or when the model must return equally structured output (in that case, matching the request format to the desired response format usually improves reliability more than it costs in tokens).

Building token budgets into your workflow

Treat token count like you'd treat bundle size in frontend engineering: a metric that regresses quietly unless you watch it.

Add a token-count assertion to your prompt template tests, so a system prompt that grows past a threshold fails CI.
Log input and output token counts per request in production, broken down by endpoint or prompt template, so you can see which flows are the biggest cost drivers.
Periodically diff your system prompts against a token budget per feature. If "handle refunds" logic has crept from 40 tokens to 400 tokens over six months of patches, that's a signal to refactor, not just accept.

import logging

def call_model(system, messages, **kwargs):
    input_tokens = count_tokens(system) + sum(count_tokens(m["content"]) for m in messages)
    if input_tokens > TOKEN_BUDGET:
        logging.warning(f"Prompt exceeds budget: {input_tokens} > {TOKEN_BUDGET}")
    response = client.messages.create(system=system, messages=messages, **kwargs)
    logging.info(
        "tokens_in=%s tokens_out=%s",
        response.usage.input_tokens,
        response.usage.output_tokens,
    )
    return response

FAQ

Does prompt token optimization hurt output quality? Not if you do it correctly. Removing redundant instructions, deduplicating examples, and dropping irrelevant retrieved chunks tends to *improve* quality, because the model spends its attention on signal instead of noise. The failure mode to watch for is over-trimming instructions that actually disambiguate edge cases; test against a regression suite of real queries before and after any prompt change, not just the total token count.

Is it better to optimize input tokens or output tokens first? Check your provider's pricing: output tokens are typically several times more expensive than input tokens, so a small reduction in output length often saves more money than a larger reduction in input length. That said, input tokens usually dominate raw volume in RAG and agentic systems, so both matter; measure your actual split before deciding where to spend effort.

How much does prompt caching actually save? It depends on how much of your prompt is static versus dynamic and how often you're calling the same cached prefix within the cache's active window. Systems with a large, unchanging system prompt or document set and high call volume see the biggest gains, since the cached portion is billed at a steep discount on every cache hit after the first write.

Should I switch to a smaller model instead of optimizing prompts? These aren't mutually exclusive, and prompt optimization should usually come first. A smaller model run against a bloated, poorly structured prompt often performs worse and isn't guaranteed to be cheaper once you account for retries from lower accuracy. Trim and structure your prompts, measure quality and cost at your current model, then evaluate whether a smaller model still meets the bar on the leaner prompt.

Do longer context windows make token optimization unnecessary? No. A bigger window removes the hard ceiling but not the cost-per-token or the attention-dilution effect. Filling a million-token window because you can, instead of because the task needs it, still costs money on every call and still risks the model missing the one relevant detail buried in the middle. Treat the context window as a budget to spend deliberately, not a container to fill.

What's the single highest-leverage change to make first? Turn on prompt caching for any large static block you send repeatedly, and add token counting to your logs. Caching is close to a free win with no quality tradeoff, and visibility into your actual token spend tells you which prompt or endpoint to trim next instead of guessing.