Context Compression Techniques for Long Prompts

Context compression is the set of techniques you use to fit more useful information into a model's context window without blowing your token budget or degrading output quality. If you're building an agent that runs for hours, a RAG pipeline over a large document set, or a chat app with long conversation history, context compression is not optional. It's the difference between a system that stays fast and cheap and one that falls over at turn 40 because the prompt is now 180,000 tokens of mostly-irrelevant history.

This article walks through the compression techniques that matter in practice: summarization, selective retrieval, structured truncation, semantic deduplication, and token-level compression. Each section has runnable code so you can test the technique against your own data instead of taking my word for it.

Why context compression matters now

Long context windows (100K, 200K, even 1M tokens) solved one problem and created another. You can now stuff enormous amounts of text into a single prompt, but three costs scale with it:

• Latency. Time-to-first-token generally increases with input length, even before generation starts. A 150K-token prompt is noticeably slower to start responding than an 8K-token one.
• Cost. Most API pricing is per-token, and input tokens are billed whether the model uses them or not. Every irrelevant paragraph you send is money spent for nothing.
• Quality. This is the one people underestimate. Models attend unevenly across long contexts, a pattern often called "lost in the middle." Information buried in the center of a huge prompt gets used less reliably than information near the start or end. Dumping everything into context and hoping the model figures out what's relevant is not a strategy, it's a bet against attention mechanics.

Context compression fixes all three by reducing what you send while preserving what the model actually needs to answer correctly.

The core techniques

There are five compression strategies worth knowing. You'll usually combine two or three of them in a real pipeline.

1. Recursive summarization for long-running agents

If you're running an agent loop (an agent that calls tools, gets results, and keeps going across many turns), full conversation history grows without bound. Recursive summarization replaces older turns with a running summary, keeping only recent turns verbatim.

The pattern: once the transcript crosses a token threshold, summarize everything except the last N turns, replace the summarized portion with the summary, and continue.

def compress_history(messages, model, keep_last=6, threshold_tokens=8000):
    token_count = estimate_tokens(messages)
    if token_count < threshold_tokens:
        return messages

    to_summarize = messages[:-keep_last]
    recent = messages[-keep_last:]

    summary_prompt = (
        "Summarize the key facts, decisions, and open tasks from this "
        "conversation history. Preserve specific values (file paths, "
        "IDs, numbers, names) exactly. Drop small talk and repeated "
        "tool output.\n\n" + format_messages(to_summarize)
    )

    summary = model.generate(summary_prompt, max_tokens=500)

    return [
        {"role": "system", "content": f"Earlier context summary:\n{summary}"}
    ] + recent

The important detail is "preserve specific values exactly." A summarization pass that paraphrases a file path or an API key placeholder into something vague will break the agent two turns later. Instrument this: log what gets summarized away, and if you see the agent re-asking for information it already had, your summary prompt is too lossy.

Run this compression check on every turn, not just when you hit a hard limit. Waiting until you're at 95% of the context window means the summarization call itself has to compete for space.

2. Selective retrieval instead of full-document stuffing

If your use case is RAG (retrieval-augmented generation), the compression already happens at the point you decide what to retrieve. The mistake is retrieving too generously, then stuffing the top 20 chunks into context because "more context can't hurt." It can.

Two adjustments compress the retrieved context without touching your embedding model:

Rerank before you truncate, not after. A retriever tuned for recall will return chunks that are topically related but not actually useful. A reranker (cross-encoder models are the standard here) scores query-chunk pairs directly and is far more precise than the retriever's cosine similarity. Retrieve 40, rerank, keep the top 5-8.

def compress_retrieval(query, candidates, reranker, keep_top=6):
    scored = reranker.score(query, [c.text for c in candidates])
    ranked = sorted(zip(candidates, scored), key=lambda x: x[1], reverse=True)
    return [c for c, score in ranked[:keep_top]]

Chunk smaller, not bigger. A common instinct when context windows grew large was to grow chunk size too, "since we can afford it." That's backwards. A 2,000-token chunk where the relevant sentence sits in the middle wastes 1,800 tokens of noise per retrieved document and dilutes what the model attends to. Chunk at 200-400 tokens with overlap, and let the retrieval count (how many chunks) do the work that chunk size used to.

3. Structured truncation for tool output and logs

Agent tool calls (a shell command, an API response, a database query) often return far more than the model needs. A git diff might be 3,000 lines when the model needs the 12 lines that changed in the relevant function. Naive truncation (cut at N characters) is dangerous because it can cut off mid-structure and leave the model with malformed JSON or a diff missing its closing context.

Structured truncation means truncating at semantic boundaries and summarizing what got cut.

def compress_tool_output(output, max_tokens=1500):
    if estimate_tokens(output) <= max_tokens:
        return output

    lines = output.splitlines()
    head = lines[:40]
    tail = lines[-20:]
    omitted = len(lines) - len(head) - len(tail)

    return "\n".join(head) + f"\n\n... [{omitted} lines omitted] ...\n\n" + "\n".join(tail)

For JSON tool responses, don't line-truncate at all, parse and prune fields instead:

def compress_json_response(data, keep_fields):
    if isinstance(data, list):
        return [compress_json_response(item, keep_fields) for item in data]
    if isinstance(data, dict):
        return {k: v for k, v in data.items() if k in keep_fields}
    return data

This is the highest-leverage change for agent pipelines that call external APIs a lot. A weather API, a search API, or a database ORM will happily return 50 fields when your agent's prompt only ever references 6 of them. Define the allowlist once per tool and apply it before the response ever reaches the model.

4. Semantic deduplication across turns and documents

In multi-turn agent loops and multi-document RAG, the same fact gets restated repeatedly: a tool returns the same error three times, three retrieved chunks all restate the same product spec. Deduplication removes near-identical content before it reaches the prompt.

Exact-match dedup is trivial. Near-duplicate dedup needs embeddings:

def dedupe_chunks(chunks, embedder, similarity_threshold=0.92):
    kept = []
    kept_embeddings = []

    for chunk in chunks:
        emb = embedder.embed(chunk.text)
        is_duplicate = any(
            cosine_similarity(emb, kept_emb) > similarity_threshold
            for kept_emb in kept_embeddings
        )
        if not is_duplicate:
            kept.append(chunk)
            kept_embeddings.append(emb)

    return kept

Run this after retrieval and before reranking, since deduplication and reranking solve different problems: dedup removes redundant chunks that would otherwise all score highly and crowd out diverse but useful ones, and reranking then picks the best of what remains.

5. Token-level compression (prompt compression models)

There's a category of technique that compresses at the token level rather than the document level: a small model rewrites or scores a prompt to drop low-information tokens while keeping the ones that carry meaning. This works because natural language is redundant, function words, repeated qualifiers, and boilerplate phrasing carry little signal for the task the model is actually doing.

This technique is more brittle than the others and worth treating as an optimization, not a default. It's most useful when:

• You have a fixed, large system prompt or few-shot examples that repeat on every call, and you can afford to precompute and cache a compressed version once, then validate it doesn't change output quality.
• You're paying per-token at high volume and a measured 20-30% token reduction is worth the eval overhead.

Don't reach for token-level compression before you've applied selective retrieval and structured truncation. Those two typically remove 70-90% of the waste with far less risk of silently dropping something the model needed.

Putting it together: a compression pipeline

A realistic agent pipeline layers these in order, cheapest and safest first:

def build_context(query, conversation_history, tool_results, retriever, reranker, embedder):
    # 1. Compress conversation history (recursive summarization)
    history = compress_history(conversation_history, model=summarizer_model)

    # 2. Compress tool output (structured truncation)
    tool_context = [compress_tool_output(r) for r in tool_results]

    # 3. Retrieve, dedupe, rerank (selective retrieval)
    candidates = retriever.search(query, top_k=40)
    deduped = dedupe_chunks(candidates, embedder)
    retrieved = compress_retrieval(query, deduped, reranker, keep_top=6)

    return {
        "history": history,
        "tool_output": tool_context,
        "retrieved_docs": retrieved,
    }

Measure the pipeline, not just the individual techniques. Track total input tokens per request and, separately, an answer-quality metric (an LLM-as-judge score, a retrieval-hit-rate, or a task success rate for agents). The goal isn't the smallest possible prompt, it's the smallest prompt that doesn't move the quality metric. Cut too aggressively and you'll see quality drop quietly, often before anyone notices, because the model still produces confident-sounding wrong answers instead of erroring out.

Common mistakes

Compressing before you've measured what's actually large. Profile your prompts first. It's common to assume conversation history is the bloat when it's actually one verbose tool (a stack trace, a full HTML page dump) dominating every call. Fix the actual source before adding a generic summarization layer.

Truncating instead of summarizing when the cut content matters later. Truncation is fine for tool output you'll never reference again. It's wrong for conversation history where the user might refer back to something three turns ago ("like you said earlier"). If reference-back is possible, summarize, don't just cut.

Using one compression ratio for everything. A system prompt with your agent's tool definitions should never be compressed, it needs to be exact. A tool's raw JSON response can often be cut by 80%. Treat these as different budgets, not one global token limit split evenly.

Skipping evals on the compression step itself. Summarization and dedup are themselves LLM/embedding calls that can go wrong, a bad summarization prompt can hallucinate details that weren't in the original conversation. Build a small eval set: take real conversation transcripts, run them through compression, and manually check the summary against the source for a dozen examples before you trust it in production.

FAQ

Does a bigger context window make compression unnecessary? No. Larger windows raise the ceiling but don't fix the "lost in the middle" attention problem, and cost/latency still scale with tokens sent. Compression is about sending the right information, not just fitting under a limit.

How do I know if my compression is too aggressive? Set up a held-out eval: run the same tasks with and without compression, compare output quality with an LLM judge or task success rate. If quality drops more than a few percentage points, loosen the threshold or improve the summarization prompt before shipping.

Should I compress on every single turn or only when I hit a limit? Compress on every turn once you cross a modest threshold (well under your hard limit), not at the last moment. Compressing right before you hit the ceiling leaves no headroom for the summarization call itself, which also consumes context.

Is prompt/token-level compression worth the complexity for most teams? Usually not first. Selective retrieval, deduplication, and structured tool-output truncation solve most of the waste with much lower risk. Reach for token-level compression only after those are in place and you still need to cut cost at high volume.

How does context compression interact with prompt caching? They can conflict. Prompt caching rewards a stable prefix (same system prompt and early messages across calls) so the provider can skip reprocessing it. If your compression step rewrites history on every turn, you invalidate the cache each time. Keep a stable cached prefix (system prompt, tool definitions) and apply compression only to the mutable tail of the conversation.

What's the simplest first step if I haven't done any of this yet? Add structured truncation on tool output. It's the lowest-risk, highest-yield change: define an allowlist of fields your prompt actually references for each tool, strip everything else before the response reaches the model. You'll often see a large token drop with no measurable quality loss.