Agent State Management: Where Does Agent State Actually Live?

The question nobody asks until production breaks

Every tutorial on building AI agents shows you the happy path: a loop, a system prompt, a tool call, a nice printed response. Nobody shows you what happens when the process dies mid-task, or when a user closes their laptop between step 3 and step 4 of a 12-step agent workflow, or when two requests hit the same agent session concurrently and now your "memory" has a race condition.

Agent state management is the part of agent engineering that separates demos from products. It's unglamorous. It's also the single biggest predictor of whether your agent survives contact with real users. If you've ever asked "wait, where is the agent's state actually stored right now?" and didn't have a confident answer, this article is for you.

We're going to walk through every layer where agent state can live, why each layer exists, what breaks when you get it wrong, and how to build a state architecture that doesn't collapse the first time someone refreshes a browser tab.

What "agent state" actually means

Before we go layer by layer, let's be precise about what we're managing. "Agent state" is not one thing — it's at least five different things that people lump together and then get confused about:

• Conversation state — the message history: user turns, assistant turns, tool calls, tool results.
• Working memory / scratchpad state — intermediate reasoning, partial results, variables the agent is tracking mid-task (e.g., "step 2 of 5 done, here's what I found").
• Execution state — where exactly the agent is in a multi-step plan, including which tools have run, which are pending, and whether a step failed and needs retry.
• Long-term memory — facts, preferences, and summaries that should persist across sessions, not just within one.
• Environment state — the actual external world the agent is manipulating: a database row, a file on disk, a browser tab, a deployed container. This is state the agent affects but does not "hold."

The mistake most people make is treating all five as one blob and stuffing everything into the LLM's context window. That works for a demo. It falls apart the moment your agent needs to run for more than a few minutes, survive a restart, or be inspected by a human debugging a weird failure.

Layer 1: The context window is not a database

The most tempting place to keep state is the context window itself, because it's free and it's already there. Every message you send the model is, in a sense, "state" — it's what the model conditions its next output on. But treating the context window as your source of truth for state is a trap for three reasons.

First, it's lossy. Once you truncate or summarize old messages to fit a context budget, you've destroyed state, not stored it. If your agent's only record of "the user's account ID is 48213" was a message that got summarized away, that fact is gone.

Second, it's not queryable. You cannot ask "what tools has this agent called in the last hour across all sessions" by grepping a context window. You need structured storage for that.

Third, it's not durable. If your process crashes, restarts, or you're running serverless functions that get recycled, the context window that lived in a variable in memory is gone unless you explicitly persisted it somewhere.

Here's a minimal illustration of the naive approach that most people start with, and why it's fragile:

class NaiveAgent:
    def __init__(self):
        # State lives ONLY in this Python list, in process memory
        self.messages = []

    def run_turn(self, user_input):
        self.messages.append({"role": "user", "content": user_input})
        response = call_llm(self.messages)
        self.messages.append({"role": "assistant", "content": response})
        return response

# If this process restarts, self.messages is gone.
# If you run two instances behind a load balancer, they diverge.
# If the container is killed mid-tool-call, there's no record
# that a tool was even invoked.

This isn't wrong for a prototype. It's wrong as an architecture you ship. The context window should be treated as a *view* computed from durable state, not the state itself.

Layer 2: Session state and the persistence boundary

The first real architectural decision in agent state management is: where does state cross the boundary from "ephemeral, in-process" to "durable, survives a restart"? This is usually called the persistence boundary, and where you draw it determines your entire system's reliability characteristics.

A reasonable default is to persist after every meaningful state transition — not after every token, but after every discrete event: a user message received, a tool call started, a tool call completed, an agent turn finished. This gives you a replayable log.

import json
import time
import uuid

class SessionStore:
    """Durable event log backing an agent session.
    In production this table lives in Postgres/Neon, not in memory.
    """
    def __init__(self, db_conn):
        self.db = db_conn

    def append_event(self, session_id, event_type, payload):
        self.db.execute(
            """
            INSERT INTO agent_events (id, session_id, event_type, payload, created_at)
            VALUES (%s, %s, %s, %s, %s)
            """,
            (str(uuid.uuid4()), session_id, event_type, json.dumps(payload), time.time()),
        )

    def load_session(self, session_id):
        rows = self.db.execute(
            "SELECT event_type, payload, created_at FROM agent_events "
            "WHERE session_id = %s ORDER BY created_at ASC",
            (session_id,),
        )
        return [self._to_event(r) for r in rows]

    def _to_event(self, row):
        event_type, payload, created_at = row
        return {"type": event_type, "payload": json.loads(payload), "at": created_at}

With this in place, "agent state" for a session is a derived value — you replay the event log to reconstruct the current message history, the current plan, and the current step index. This is the same pattern event-sourced systems have used for years, and it maps onto agents extremely well because agent execution is naturally a sequence of discrete events: think, call tool, observe, think again.

The payoff is durability and debuggability. If an agent misbehaves in production, you can pull the exact event sequence and see precisely what it saw and did, in order, instead of guessing from a log line.

Layer 3: Working memory versus long-term memory

Session state answers "what happened in this conversation." It does not answer "what does this agent know about this user across every conversation they've ever had." That's a different layer, and conflating the two is one of the most common state management mistakes in agent design.

Working memory is scoped to a task or session and typically expires when the task ends — it's the scratchpad where an agent tracks "I've already checked the inventory API, now I need to check pricing." Long-term memory is scoped to an entity (a user, an account, a project) and persists indefinitely, or until explicitly forgotten.

A workable split looks like this:

class WorkingMemory:
    """Scoped to a single task run. Cheap, fast, often just Redis or in-memory."""
    def __init__(self, task_id, cache):
        self.task_id = task_id
        self.cache = cache

    def set(self, key, value):
        self.cache.set(f"task:{self.task_id}:{key}", value, ex=3600)

    def get(self, key):
        return self.cache.get(f"task:{self.task_id}:{key}")


class LongTermMemory:
    """Scoped to a user/account. Durable, queryable, survives forever."""
    def __init__(self, db_conn):
        self.db = db_conn

    def remember(self, user_id, fact, source_session_id):
        self.db.execute(
            """
            INSERT INTO agent_memory (user_id, fact, source_session_id, created_at)
            VALUES (%s, %s, %s, now())
            """,
            (user_id, fact, source_session_id),
        )

    def recall(self, user_id, limit=20):
        return self.db.execute(
            "SELECT fact, created_at FROM agent_memory "
            "WHERE user_id = %s ORDER BY created_at DESC LIMIT %s",
            (user_id, limit),
        )

The critical design decision here is *what gets promoted* from working memory to long-term memory. Not every scratchpad note deserves to live forever — you don't want your agent's long-term memory table to fill up with "step 3 of 5 complete." A common pattern is to run a lightweight extraction pass at the end of a session: summarize the session, pull out durable facts (preferences, decisions, corrections the user made), and write only those to long-term storage. Everything else in working memory is allowed to expire.

Layer 4: Execution state — the part everyone forgets

This is the layer that separates "chatbot with tools" from "agent that does multi-step work reliably." Execution state tracks where the agent is in a plan: which steps are done, which are pending, which failed, and what the retry policy is.

Without explicit execution state, a common failure mode is that an agent re-does work it already completed after a crash or timeout — it charges a customer's card twice, sends a duplicate email, or re-runs an expensive computation, because on restart it has no record of "I already did that."

from enum import Enum

class StepStatus(str, Enum):
    PENDING = "pending"
    IN_PROGRESS = "in_progress"
    DONE = "done"
    FAILED = "failed"

class ExecutionState:
    """Tracks a multi-step agent plan durably, so a crash mid-plan
    can resume instead of restarting from step one.
    """
    def __init__(self, db_conn, run_id):
        self.db = db_conn
        self.run_id = run_id

    def init_plan(self, steps):
        for i, step in enumerate(steps):
            self.db.execute(
                """
                INSERT INTO agent_steps (run_id, step_index, tool_name, args, status)
                VALUES (%s, %s, %s, %s, %s)
                """,
                (self.run_id, i, step["tool"], json.dumps(step["args"]), StepStatus.PENDING),
            )

    def mark_in_progress(self, step_index):
        self._update_status(step_index, StepStatus.IN_PROGRESS)

    def mark_done(self, step_index, result):
        self.db.execute(
            "UPDATE agent_steps SET status = %s, result = %s WHERE run_id = %s AND step_index = %s",
            (StepStatus.DONE, json.dumps(result), self.run_id, step_index),
        )

    def mark_failed(self, step_index, error):
        self.db.execute(
            "UPDATE agent_steps SET status = %s, error = %s WHERE run_id = %s AND step_index = %s",
            (StepStatus.FAILED, str(error), self.run_id, step_index),
        )

    def next_pending_step(self):
        row = self.db.execute(
            "SELECT step_index, tool_name, args FROM agent_steps "
            "WHERE run_id = %s AND status IN (%s, %s) ORDER BY step_index ASC LIMIT 1",
            (self.run_id, StepStatus.PENDING, StepStatus.IN_PROGRESS),
        )
        return row

    def _update_status(self, step_index, status):
        self.db.execute(
            "UPDATE agent_steps SET status = %s WHERE run_id = %s AND step_index = %s",
            (status, self.run_id, step_index),
        )

Notice the design here treats a resumable agent run almost like a job queue. That's not an accident — multi-step agent execution *is* a job queue problem, and idempotency matters just as much. Each step should be safe to re-check before re-running: "has this email already been sent?" should be a query against durable state, not a hope that the process didn't crash.

This is also where idempotency keys earn their keep. Before an agent calls a tool that has side effects — charging money, sending a message, writing to an external system — attach a stable idempotency key derived from the run ID and step index, so that if the step is retried after a partial failure, the downstream system can recognize and ignore the duplicate.

Layer 5: Concurrency — what happens when two things touch the same state

Single-user, single-session agent demos never hit this problem. Production systems do, constantly. A user sends a follow-up message while the agent is still processing the previous one. A scheduled background job and a live chat both try to update the same session's state. Two tool calls in a parallel tool-execution step both want to write to the same working-memory key.

The fix is boring and well-understood from regular backend engineering: optimistic concurrency control with version numbers, or pessimistic locking for the sections that truly can't tolerate a race.

class VersionedSessionState:
    """Optimistic concurrency control for session state updates."""
    def __init__(self, db_conn):
        self.db = db_conn

    def update_state(self, session_id, new_state, expected_version):
        result = self.db.execute(
            """
            UPDATE agent_sessions
            SET state = %s, version = version + 1
            WHERE session_id = %s AND version = %s
            RETURNING version
            """,
            (json.dumps(new_state), session_id, expected_version),
        )
        if result.rowcount == 0:
            raise ConcurrentModificationError(
                f"Session {session_id} was modified by another writer; "
                f"expected version {expected_version}"
            )
        return result

If you skip this, the failure mode is subtle and ugly: state updates silently overwrite each other, and you get an agent that "forgets" things a user just told it, seemingly at random, because two writers raced and the loser's update vanished. This class of bug is miserable to reproduce because it only shows up under real concurrent load, never in your local testing with one browser tab open.

Layer 6: State across agent-to-agent handoffs

Once you move from a single agent to a multi-agent system — a planner agent handing off to a specialist agent, or a supervisor delegating to sub-agents — state management gets another dimension: what state crosses the boundary, and in what shape?

The naive approach is to just pass the entire conversation history to the sub-agent. This is usually wrong. A sub-agent doing a narrow task (say, "extract structured data from this document") doesn't need — and shouldn't see — the full history of an unrelated earlier part of the conversation. It needs a scoped, purpose-built state object.

class AgentHandoff:
    """Explicit, scoped state passed between agents — not the full history."""
    def __init__(self, task, context, constraints, parent_run_id):
        self.task = task                  # what this sub-agent must do
        self.context = context             # only the relevant facts, not full history
        self.constraints = constraints      # budget, tool allowlist, deadline
        self.parent_run_id = parent_run_id  # for tracing back to the originating run

    def to_prompt_context(self):
        return {
            "task": self.task,
            "relevant_facts": self.context,
            "limits": self.constraints,
        }

def delegate_to_specialist(parent_state, sub_task_description):
    handoff = AgentHandoff(
        task=sub_task_description,
        context=extract_relevant_facts(parent_state, sub_task_description),
        constraints={"max_tool_calls": 5, "timeout_s": 30},
        parent_run_id=parent_state["run_id"],
    )
    result = run_specialist_agent(handoff)
    # Only the result, not the sub-agent's internal reasoning, flows back up
    return {"summary": result.summary, "data": result.data}

Two things matter here. First, extract_relevant_facts is doing real work — it's a deliberate filtering step, not a shortcut. Second, results flow back up as a summary, not as the sub-agent's raw internal chain of reasoning. This keeps the parent agent's context window from bloating with irrelevant sub-agent internals, and it keeps state boundaries clean enough that you can reason about, test, and replay each agent in isolation.

Where the actual bytes live: a practical stack

Pulling the layers together, here's a stack that holds up in production, roughly in order of how "hot" the data is:

• In-process memory — the current turn's variables, nothing that needs to survive past this function call.
• Redis or similar cache — working memory, active session cache, rate limit counters, anything scoped to minutes-to-hours and fine to lose occasionally.
• Postgres/Neon (or your primary OLTP store) — the event log, execution state, long-term memory, anything that must survive a restart and needs to be queried.
• Object storage (S3-compatible) — large artifacts an agent produces or consumes: documents, generated files, screenshots, anything too big to sanely put in a database row.
• Vector store — semantic recall over long-term memory or documents, when you need "find things similar to this" rather than exact lookups.

A common mistake is reaching for a vector database as the *primary* state store. Vector stores are excellent for one job — semantic similarity search — and mediocre at everything a relational store does well: transactions, exact lookups, joins, and strong consistency guarantees. Keep your execution state and event log in a proper transactional database, and use the vector store purely for the recall step when an agent needs to search unstructured memory.

Testing state management like you mean it

State bugs are the hardest bugs to catch with normal testing because they only show up under specific timing and failure conditions. A few practices that catch real issues before production does:

• Write a test that kills the agent process mid-tool-call and asserts the run resumes correctly rather than duplicating the side effect.
• Write a test that runs two "sessions" against the same session ID concurrently and asserts one of them gets a concurrency error rather than silently losing data.
• Snapshot the event log for a full run and replay it through your state-reconstruction logic; assert the reconstructed state matches what the live run produced.
• Explicitly test summarization/compaction logic against the raw event log to make sure facts you need aren't being silently dropped when the context window gets trimmed.

def test_resume_after_crash(execution_state, tool_runner):
    execution_state.init_plan(steps=THREE_STEP_PLAN)
    execution_state.mark_done(0, result={"ok": True})
    execution_state.mark_in_progress(1)
    # Simulate a crash: process dies here, nothing marks step 1 done or failed

    resumed = ExecutionState(db_conn=execution_state.db, run_id=execution_state.run_id)
    next_step = resumed.next_pending_step()

    assert next_step[0] == 1  # resumes at the interrupted step, not step 0
    # and the tool call for step 1 must be idempotent-safe to re-run

If you can't write this test, you don't actually know how your agent behaves during a crash — you're hoping.

Bringing it together

Agent state management isn't one design decision, it's five: conversation history, working memory, execution state, long-term memory, and the environment state the agent manipulates. Each has different durability, consistency, and query requirements, and treating them as one undifferentiated blob stuffed into a context window is the root cause of most "why did my agent forget/duplicate/hang" bugs you'll hit in production.

The pattern that scales is boring on purpose: an append-only event log as source of truth, derived views for what actually goes into the model's context, explicit execution-state tracking with idempotency for anything with side effects, a clear working-memory/long-term-memory split, and scoped handoffs between agents instead of dumping full history across boundaries. None of this is exotic — it's the same discipline that's kept distributed backend systems reliable for two decades, applied to a new kind of worker.

If you want to actually build this instead of just reading about it — wiring a real event-sourced session store, resumable multi-step execution, and memory promotion logic into a working agent — that's exactly the kind of hands-on system we build step by step inside 30 Days of Hermes Agent. It's designed to take you from "agent that works in a notebook" to "agent whose state architecture survives a production outage," one day at a time.