Managing the Context Window in Long Agent Runs

An agent context window is the finite slice of tokens an LLM can see at any one step: the system prompt, tool definitions, conversation history, and every tool result the agent has produced so far. In a short chat this is a non-issue, but in a long agent run, one that reads files, calls APIs, browses the web, or edits code across hundreds of steps, the window fills up fast and the agent starts forgetting its own earlier decisions, repeating failed tool calls, or getting cut off mid-task. Managing the agent context window well is the difference between an agent that finishes a two-hour job coherently and one that quietly degrades after step forty.

This piece covers the practical techniques for keeping a long-running agent inside its context budget: compaction, externalizing state to files, sub-agent delegation, tool output discipline, prompt caching, and retrieval. All of it is runnable today with common agent frameworks (Claude Agent SDK, LangChain/LangGraph, or a hand-rolled tool loop) and none of it requires a bigger context window, just better discipline about what goes into the one you have.

Why the Agent Context Window Fills Up Faster Than You Expect

Every tool call an agent makes appends at least two things to the transcript: the tool call itself (name, arguments) and the tool result. In file-editing agents, a single read_file on a 2,000-line file can add thousands of tokens. In browser agents, a page snapshot or a screenshot description can be even larger. In research agents, a single web search result page can run to several thousand tokens before the agent has even read it.

The math compounds. If an agent averages 3,000 tokens of tool output per step and runs for 100 steps, that is 300,000 tokens of raw tool output alone, before counting the system prompt, the running plan, and the model's own reasoning and replies. Most production context windows are large but not infinite, and even when the window technically fits everything, model quality tends to degrade well before the hard limit: instructions from early in the run get diluted, and the agent starts re-deriving things it already figured out, wasting steps and money.

There are three separate failure modes worth naming, because each has a different fix:

• Overflow: the transcript literally exceeds the model's context limit and the call fails or gets silently truncated.
• Dilution: the transcript fits, but the signal-to-noise ratio is so low (huge tool dumps, stale exploration) that the model loses track of the actual goal.
• Cost blowup: every step re-sends the entire growing transcript, so token spend grows roughly quadratically with the number of steps even if nothing overflows.

Good agent context window management addresses all three at once, not just the first.

Know Your Token Budget Before You Write Any Agent Code

Before optimizing anything, instrument it. You cannot manage what you do not measure, and "the agent got dumb around step 60" is not a metric.

A minimal token-tracking wrapper around any agent loop looks like this in Python:

import tiktoken

encoder = tiktoken.get_encoding("cl100k_base")

def count_tokens(messages):
    total = 0
    for m in messages:
        content = m.get("content", "")
        if isinstance(content, list):
            content = " ".join(str(c) for c in content)
        total += len(encoder.encode(str(content)))
    return total

def log_step(step_number, messages, budget_tokens):
    used = count_tokens(messages)
    pct = round(100 * used / budget_tokens, 1)
    print(f"[step {step_number}] context: {used} tokens ({pct}% of budget)")
    if pct > 70:
        print("  -> approaching budget, consider compaction")

Set budget_tokens to some fraction (say 70-80%) of the model's actual context limit, not the full limit. Leave headroom for the next tool call's output and the model's reply, both of which have to fit inside whatever is left. Wire log_step into your agent loop after every tool call and you get an early warning system instead of a surprise failure on step 90.

Two numbers matter more than the raw total: tokens added per step (is it trending up because tool outputs are getting bigger?) and tokens carried forward from turns the agent no longer needs (is old, resolved exploration still sitting in the transcript?). Track both, not just the total.

Strategy 1: Summarization and Compaction

Compaction is the most direct fix: periodically replace a chunk of old transcript with a shorter summary that preserves the facts the agent still needs. Most modern agent frameworks (including the Claude Agent SDK) support this natively, but you can implement a basic version yourself.

def compact_transcript(client, model, messages, keep_last_n=6):
    """Summarize everything except the system prompt and the last N messages."""
    system_msg = messages[0]
    old = messages[1:-keep_last_n]
    recent = messages[-keep_last_n:]

    if not old:
        return messages

    summary_prompt = (
        "Summarize the following agent transcript into a compact briefing. "
        "Preserve: the current goal, decisions already made, files already "
        "read or edited, tool calls that failed and why, and any open "
        "questions. Drop raw tool output and exploratory dead ends."
    )
    response = client.messages.create(
        model=model,
        max_tokens=800,
        messages=[
            {"role": "user", "content": summary_prompt + "\n\n" + str(old)}
        ],
    )
    summary_text = response.content[0].text

    return [
        system_msg,
        {"role": "user", "content": f"[Compacted history]\n{summary_text}"},
        *recent,
    ]

Trigger this when your token tracker crosses a threshold (70-80% of budget), not on a fixed step count, since token growth per step varies a lot by task. A few rules make compaction safer:

• Never summarize the last few turns. The model's most recent reasoning is usually load-bearing for its next action; summarizing it too early causes it to re-decide things it just decided.
• Keep tool call/result pairs together or drop them together. Summarizing a tool call without its result (or vice versa) confuses the model about what actually happened.
• Ask the summarizer to preserve failures, not just successes. An agent that forgets a tool call already failed will retry it, burning another step and another chunk of budget.
• Log the compaction event itself somewhere outside the transcript (metrics, not tokens) so you can see how often it fires and tune the threshold.

Strategy 2: Externalize State to Files and Scratchpads

The cheapest way to shrink the transcript is to stop putting things in it that do not need to be there. Instead of asking the agent to hold its plan, its findings, and its intermediate results in conversation turns, give it a scratchpad file and a small set of read/write tools, then treat the transcript as a pointer to that file rather than a copy of its contents.

import json
from pathlib import Path

SCRATCHPAD = Path("./agent_scratchpad.json")

def write_scratchpad(key, value):
    state = json.loads(SCRATCHPAD.read_text()) if SCRATCHPAD.exists() else {}
    state[key] = value
    SCRATCHPAD.write_text(json.dumps(state, indent=2))
    return f"Saved '{key}' to scratchpad ({len(json.dumps(value))} chars)."

def read_scratchpad(key=None):
    if not SCRATCHPAD.exists():
        return "Scratchpad is empty."
    state = json.loads(SCRATCHPAD.read_text())
    return state.get(key, state) if key else state

Register write_scratchpad and read_scratchpad as tools the agent can call. Tell it, in the system prompt, to write findings, plans, and long intermediate outputs to the scratchpad instead of restating them in its replies, and to read back only the specific key it needs for the current step. A research agent that found and summarized twenty sources does not need all twenty summaries in context for step twenty-one; it needs the one key fact from source fourteen, retrieved on demand.

This pattern is what file-editing coding agents already do implicitly: the codebase itself is the scratchpad. The agent does not need the full contents of every file it has touched sitting in the transcript, it needs to know which files it touched and can re-read them if needed. Explicitly encouraging the same pattern for research, planning, and multi-step task state gets you the same context savings outside of code.

Strategy 3: Sub-Agent Delegation to Isolate Context

The most powerful lever, and the one general-purpose agent frameworks lean on hardest, is delegation: spin up a sub-agent with its own fresh context window to do a bounded piece of work, and have it return only the distilled result to the parent.

def delegate_research(client, model, question, max_tool_calls=15):
    """Run an isolated sub-agent to answer one question, return only the answer."""
    sub_messages = [
        {"role": "system", "content": (
            "You are a focused research sub-agent. Answer the question below "
            "using the tools available. When done, reply with a single "
            "concise answer (under 200 words) and nothing else."
        )},
        {"role": "user", "content": question},
    ]
    # sub_messages accumulates its own tool calls/results here,
    # completely separate from the parent agent's transcript
    result = run_tool_loop(client, model, sub_messages, max_tool_calls)
    return result  # only this short string goes back to the parent

The parent agent's transcript gains one question and one answer. Everything the sub-agent did to get there, every search query, every page it read, every dead end, stays in the sub-agent's own context and is discarded when it finishes. This is why multi-agent architectures scale to tasks that a single flat agent loop cannot: the total work done can be enormous, but the parent's context window only ever holds a summary of outcomes, not a log of every action.

The tradeoff is real: sub-agents cannot see the parent's full context, so they need enough information in the delegation prompt to work independently, and coordination overhead grows with the number of sub-agents in flight. Use this for genuinely separable sub-tasks (research a specific question, review a specific file, test a specific feature), not for steps that need the full running context to make sense.

Strategy 4: Tool Output Discipline

A large share of wasted context is not the agent's reasoning, it is unfiltered tool output. Three habits fix most of it:

Truncate and paginate large outputs. If a tool can return an unbounded amount of data (a file, a search result set, a database query), cap it and tell the agent how to ask for more.

def read_file_capped(path, max_chars=4000):
    text = Path(path).read_text()
    if len(text) <= max_chars:
        return text
    return (
        text[:max_chars]
        + f"\n\n[... truncated, {len(text) - max_chars} more characters. "
          f"Call read_file_capped(path, offset=...) to continue.]"
    )

Summarize before returning, not after. If a tool wraps an expensive operation (a web page fetch, a large API response), have the tool itself extract the relevant fields or run a cheap summarization pass before handing the result to the agent, rather than dumping the raw payload into the transcript and hoping the agent skims it.

Deduplicate repeated reads. Agents commonly re-read the same file or re-fetch the same URL multiple times across a long run. A simple cache keyed on the tool call's arguments, checked before hitting the transcript, avoids paying the token cost twice for information the agent already saw.

_tool_cache = {}

def cached_tool_call(tool_fn, *args):
    key = (tool_fn.__name__, args)
    if key in _tool_cache:
        return f"[cached result, unchanged since last read]\n{_tool_cache[key]}"
    result = tool_fn(*args)
    _tool_cache[key] = result
    return result

Even a short cache TTL (a few minutes, or just "for the duration of this run") catches the common case of an agent double-checking something it already knows.

Strategy 5: Prompt Caching for Repeated Context

Prompt caching is a cost and latency optimization, not a context-size optimization, but it changes the economics of context management enough to mention here. Providers that support prompt caching let you mark a prefix of the conversation (system prompt, tool definitions, a large reference document) as cacheable, so repeated calls that share that prefix are billed and processed faster than a full re-send.

The practical implication for long agent runs: structure your context so the stable parts (system prompt, tool schemas, any large static reference material) sit at the front of the message list and change as rarely as possible, while the volatile parts (recent tool results, the current step's reasoning) sit at the end. This does not shrink the context window, but it means the compaction and delegation strategies above cost less to run, since you are not re-paying full price for the same system prompt and tool definitions on every one of a hundred steps. Check your specific SDK's documentation for how to mark cache breakpoints; the API shape differs by provider but the placement principle (stable prefix, volatile suffix) is universal.

Strategy 6: Retrieval Instead of Stuffing

If an agent's job involves a large, mostly-static body of reference material (a codebase, a knowledge base, a set of API docs), do not load it into the context window at all. Index it and let the agent retrieve only the slice it needs for the current step.

def retrieve_relevant_chunks(query, index, top_k=5):
    """Vector or keyword search over a pre-built index; returns small chunks."""
    hits = index.search(query, top_k=top_k)
    return "\n\n".join(f"[{h.source}]\n{h.text[:500]}" for h in hits)

Wire this as a tool (search_codebase, search_docs) rather than a static context block. The agent pulls in a handful of relevant chunks per query instead of carrying the entire corpus through every step of the run. This is the same principle as externalizing state to a scratchpad, applied to material the agent did not generate itself: keep the bulk of it outside the context window, and bring in only what the current step actually needs.

Putting It Together: A Context Management Loop

A long-running agent that combines these techniques typically follows a loop like this on every step:

1. Execute the next tool call and get the result.
2. Run the result through output discipline (truncate, dedupe, cache).
3. Log token usage against the budget.
4. If usage crosses the compaction threshold, compact the older portion of the transcript.
5. If the next unit of work is separable (a distinct sub-question, a distinct file review), delegate it to a sub-agent instead of doing it inline.
6. If the step needs reference material, retrieve a bounded slice rather than loading the whole source.
7. Continue.

None of these steps are exotic. What matters is that they run automatically, on every step, rather than as a one-time fix applied after an agent has already failed. A run that manages its context window continuously stays coherent at step 200 the same way it was coherent at step 10; a run that only reacts to overflow errors degrades quietly long before it actually breaks.

FAQ

What is an agent context window, exactly? It is the total set of tokens visible to the model on a single inference call: system prompt, tool definitions, conversation history, and all prior tool outputs. It resets to whatever you send on the next call, so "managing" it really means managing what you choose to send each time, not some persistent memory the model maintains on its own.

How do I know when my agent is running out of context, versus just performing badly? Instrument token usage per step (see the tracking snippet above) and watch it against your budget. If token usage is near the limit and the agent starts repeating earlier tool calls, contradicting earlier decisions, or losing track of the original goal, that is context dilution, not a reasoning failure. If usage is comfortably under budget and quality still drops, the problem is more likely prompt structure or tool design than context size.

Does a bigger context window make this problem go away? It raises the ceiling but does not fix dilution or cost blowup. Even with a very large window, stuffing hundreds of steps of raw tool output into every call degrades signal-to-noise and multiplies token spend on every subsequent turn. Compaction, externalization, and delegation are worth doing regardless of window size, they just buy more headroom before becoming mandatory.

Should I compact on a fixed step interval or based on token count? Token count. Different steps add wildly different amounts of context (a small API call versus a large file read or web page fetch), so a fixed step interval either compacts too aggressively on light steps or too late on heavy ones. Trigger compaction off a percentage-of-budget threshold instead.

Is sub-agent delegation always better than a single agent with a big context window? No. Delegation adds coordination overhead and requires the sub-task to be genuinely separable, with enough context in the delegation prompt for the sub-agent to work independently. For tightly coupled, sequential reasoning where every step depends on the full running context, a single well-compacted agent loop is usually simpler and more reliable than splitting it across sub-agents.

Where should scratchpad files live for a long agent run? Anywhere the agent's tools can read and write reliably, a temp directory, a project-scoped working directory, or a small key-value store. What matters is that the agent treats the transcript as a pointer to that state, not a copy of it, and that you clean up scratchpad files at the end of a run so stale state does not leak into the next one.