Hermes Agent Week 3 Deep Dive: Memory and Context Engineering

Why Week 3 Is the Turning Point

By the end of week 2 in 30 Days of Hermes Agent, most students have an agent that can call tools, chain a couple of steps together, and respond in a reasonable voice. It looks impressive in a demo. Then you leave the conversation open for twenty minutes, come back, ask a follow-up question, and the agent has no idea what you were talking about. Or worse — it "remembers" something you never said, because it hallucinated a summary of an earlier turn that got truncated out of the context window.

This is the exact moment week 3 is designed for. Hermes Agent's curriculum treats memory and context engineering as a distinct discipline, not a side effect of "just pass the whole conversation history to the model." Week 3 is where students stop treating the context window as an infinite scratchpad and start treating it as a scarce, expensive resource that has to be actively managed, curated, and in many cases, rewritten before every single model call.

If week 1 was about giving your agent hands (tools) and week 2 was about giving it a plan (reasoning loops), week 3 is about giving it a mind that persists. That distinction — persistence versus statelessness — is what separates a chatbot wrapper from something that can act like a genuine assistant across days, sessions, and tasks.

The Core Problem: LLMs Are Stateless by Default

Every request to a language model API is, underneath all the abstractions, a single stateless call. You send a prompt, you get a completion. The model does not remember your last request unless you send it again. This is the uncomfortable truth that week 3 opens with, because it reframes everything students thought they understood about "AI memory."

There is no persistent neuron state sitting on a server waiting for you. Memory, in every agent framework worth using, is really just re-sending the right information at the right time. That's it. The entire discipline of context engineering exists because of this one constraint.

Once students internalize that, the curriculum splits the memory problem into three layers, each with different retention windows, storage mechanisms, and retrieval strategies:

1. Working memory — the live context window for the current task
2. Short-term memory — recent conversation history and session state
3. Long-term memory — durable facts, preferences, and past interactions stored outside the context window entirely

Getting these three layers to cooperate without stepping on each other is most of what week 3 is actually about.

Short-Term Memory: Managing the Conversation Buffer

The first hands-on lab in week 3 has students build a naive agent that appends every message to a list and sends the whole list on every turn. It works for about fifteen exchanges. Then token counts spike, latency climbs, and — depending on the model's context limit — the request eventually fails outright.

The fix isn't complicated conceptually, but it's easy to get wrong in practice. Students implement a sliding window with a twist: instead of a fixed number of turns, they budget by token count, since a single tool call with a large JSON payload can eat more tokens than ten short chat turns combined.

import tiktoken

class ConversationBuffer:
    def __init__(self, max_tokens=6000, model="gpt-4o"):
        self.max_tokens = max_tokens
        self.encoder = tiktoken.encoding_for_model(model)
        self.messages = []

    def _count_tokens(self, messages):
        total = 0
        for m in messages:
            total += len(self.encoder.encode(m["content"]))
            total += 4  # role + formatting overhead per message
        return total

    def add(self, role, content):
        self.messages.append({"role": role, "content": content})
        self._trim()

    def _trim(self):
        while self._count_tokens(self.messages) > self.max_tokens and len(self.messages) > 1:
            # Always keep the system message; drop the oldest non-system turn
            for i, m in enumerate(self.messages):
                if m["role"] != "system":
                    self.messages.pop(i)
                    break

    def get_messages(self):
        return self.messages

This is deliberately simple code — Hermes Agent leans on readable Python over clever one-liners, because students need to be able to modify this trimming logic later without archaeology. The key lesson embedded here is that trimming is destructive by default. Once a message is dropped from the buffer, it's gone unless something else preserved it. That single sentence sets up the entire rest of the week.

Long-Term Memory: What Actually Needs to Survive

Not everything belongs in long-term storage. Week 3 spends real time on the *selection* problem before it ever gets to the storage mechanism, because students who skip this step tend to build memory systems that hoard everything and retrieve nothing useful.

The curriculum draws a practical line: long-term memory should hold facts, preferences, and decisions — not raw transcript. A user saying "I prefer TypeScript over JavaScript for anything production-facing" is worth storing forever. The fifteen back-and-forth messages it took to arrive at that preference are not.

Students implement an extraction step that runs periodically (not on every turn, which would be wasteful and slow) and asks the model itself to summarize what's worth keeping:

def extract_memories(conversation_chunk, existing_memories):
    prompt = f"""
You are a memory extraction system. Given the conversation below and the
list of facts already stored, output ONLY new, durable facts worth
remembering long-term (preferences, decisions, recurring constraints).
Do not repeat anything already in existing_memories. Return a JSON list
of short factual strings. If nothing new is worth storing, return [].

Existing memories:
{existing_memories}

Conversation:
{conversation_chunk}
"""
    response = llm_call(prompt, temperature=0)
    return parse_json_list(response)

Note the temperature=0 — memory extraction is not a place for creative variance. Students who leave the default temperature in place quickly discover their agent "remembers" things that were never actually said, because a higher-temperature summarization pass paraphrases too loosely and drifts from the source. This is one of the more subtle bugs the course walks through deliberately, because it teaches a broader lesson: every LLM call inside your agent's plumbing has its own settings, and copying the same config across all of them is a common source of silent failures.

Storage: Where Long-Term Memory Actually Lives

Once you've decided *what* to remember, week 3 turns to *where* it lives. Hermes Agent covers three storage patterns, in increasing order of complexity, and is explicit that most student projects only need the first two:

• Flat file / JSON store — fine for single-user prototypes, easy to inspect and debug
• Structured database rows — a memories table keyed by user ID, with a timestamp and a category column, suited to anything with more than one user
• Vector store for semantic recall — embeddings-based storage, used when memories need to be retrieved by *meaning* rather than by exact key lookup

The course intentionally delays vector stores until students have built the simpler versions first, because a common mistake among agent builders is reaching for embeddings and a vector database before establishing whether the retrieval problem actually requires semantic search at all. If a user only has thirty stored facts, a keyword filter over a database table will outperform a vector index on latency, cost, and debuggability. Week 3 makes students prove to themselves that simple retrieval is insufficient before letting them add the complexity of embeddings.

class SimpleMemoryStore:
    def __init__(self, db_connection):
        self.db = db_connection

    def save(self, user_id, fact, category="general"):
        self.db.execute(
            "INSERT INTO memories (user_id, fact, category, created_at) VALUES (?, ?, ?, ?)",
            (user_id, fact, category, current_timestamp())
        )

    def recall(self, user_id, category=None, limit=20):
        if category:
            rows = self.db.execute(
                "SELECT fact FROM memories WHERE user_id=? AND category=? ORDER BY created_at DESC LIMIT ?",
                (user_id, category, limit)
            )
        else:
            rows = self.db.execute(
                "SELECT fact FROM memories WHERE user_id=? ORDER BY created_at DESC LIMIT ?",
                (user_id, limit)
            )
        return [r[0] for r in rows]

Retrieval: Getting the Right Memory at the Right Moment

Storing memory is the easy half. Retrieval — deciding what to pull back into context for a given task — is where week 3's hardest lab lives. The failure mode students are walked through first is retrieval flooding: dumping every stored memory into the system prompt on every single turn. This bloats the context window, dilutes the model's attention on what's actually relevant to the current question, and in some cases causes the model to reference outdated preferences that no longer apply.

The Hermes Agent approach to retrieval follows a three-step pattern:

1. Classify the incoming query — what kind of information would help answer this, if any
2. Fetch a bounded, ranked set of candidate memories — never unbounded
3. Inject only the top matches into the system or user prompt, with enough structure that the model knows these are prior facts, not new instructions

For vector-backed retrieval, this looks like an embedding similarity search capped at a small k:

def retrieve_relevant_memories(query, embedder, vector_store, k=5, threshold=0.75):
    query_vector = embedder.embed(query)
    candidates = vector_store.search(query_vector, top_k=k)
    relevant = [c for c in candidates if c.score >= threshold]
    return [c.text for c in relevant]

def build_context(user_query, memories, conversation_buffer):
    memory_block = "\n".join(f"- {m}" for m in memories) if memories else "None."
    system_prompt = f"""You are a helpful assistant with access to prior context
about this user. Use these facts only if relevant to the current request:

{memory_block}

Do not mention that you are using stored memory unless asked directly."""

    return [{"role": "system", "content": system_prompt}] + conversation_buffer.get_messages()

The threshold parameter is deliberately exposed as a tunable, because students spend a full lab session tuning it against a small evaluation set of test conversations. Too low, and irrelevant memories leak in. Too high, and the agent forgets things it should know. Hermes Agent treats this threshold-tuning exercise the same way a testing course would treat writing assertions — it's not a one-time decision, it's something you validate against real examples and revisit.

Context Window Budgeting as a First-Class Skill

Everything above — conversation buffers, extracted memories, retrieved facts — has to fit into one finite context window alongside the system prompt, tool definitions, and the user's actual question. Week 3 introduces a budgeting mental model that students carry into every later week of the course: treat the context window like a fixed budget you allocate across competing sections, not a pool that fills up until it errors.

A typical allocation students practice looks like this:

• System instructions and persona: fixed, small, rarely changes
• Tool/function definitions: fixed per available toolset
• Retrieved long-term memories: capped hard limit (for example, top 5 matches)
• Recent conversation buffer: token-budgeted sliding window
• Current user turn: always included, never trimmed

Coding this budgeting explicitly, rather than letting it happen implicitly, is the difference between an agent that degrades gracefully under load and one that either crashes or silently drops the most important part of the prompt — usually the system instructions, because naive trimming logic often removes from the front of the list first.

def assemble_prompt(system_prompt, tool_defs, memories, buffer, user_query, token_budget=8000):
    encoder = tiktoken.encoding_for_model("gpt-4o")

    fixed_cost = len(encoder.encode(system_prompt)) + len(encoder.encode(str(tool_defs)))
    remaining = token_budget - fixed_cost

    memory_text = "\n".join(memories)
    memory_cost = len(encoder.encode(memory_text))
    remaining -= memory_cost

    buffer_messages = buffer.get_messages()
    while remaining < 500 and len(buffer_messages) > 1:
        buffer_messages.pop(0)
        remaining += 200  # rough re-estimate per dropped turn

    return {
        "system": system_prompt,
        "memories": memory_text,
        "history": buffer_messages,
        "query": user_query,
    }

This isn't production-grade token accounting — Hermes Agent is upfront that real systems need more precise counting per model provider — but it's exactly the right level of complexity for a student to internalize the *shape* of the problem before reaching for a heavier library.

Common Mistakes Week 3 Deliberately Surfaces

Because memory bugs are quiet by nature — an agent that forgets something doesn't throw an exception, it just gives a slightly wrong answer — the course spends time on failure patterns students would otherwise only discover in production, weeks after shipping:

• Memory contamination across users. A shared memory store without proper user-ID scoping will happily serve one person's preferences to another. Every lab enforces a user_id parameter on every read and write specifically to build this reflex early.
• Stale memory overriding fresh instructions. If a stored preference from three months ago conflicts with what the user just said, the fresh instruction should win. Students write explicit recency-weighting into their retrieval ranking to handle this.
• Summarization drift. Compressing conversation history through repeated re-summarization compounds small errors each pass. The course has students compare a single-pass summary against a chain of five sequential summaries of the same conversation — the drift is visible and often surprising.
• Treating retrieval as free. Every vector search and every database read adds latency. Week 3's later labs have students measure end-to-end response time with and without memory retrieval enabled, so the cost is felt, not just described.
• Forgetting to test the forgetting. It's easy to test that an agent remembers something. It's rarer for builders to test that an agent correctly *doesn't* over-apply an old memory to a new, unrelated context. Hermes Agent includes this as an explicit test case template.

How Week 3 Connects to the Rest of the Course

Memory and context engineering aren't an isolated module — they're infrastructure that every later week depends on. Week 4's multi-step planning relies on the agent correctly recalling intermediate results from earlier in a task. Later weeks covering multi-agent handoffs depend on shared context being passed cleanly between agents without duplicating or losing state. Students who rush through week 3's labs tend to hit confusing, hard-to-diagnose bugs two weeks later that actually trace back to a context window that was never properly budgeted, or a memory store with no user scoping.

That's part of why Hermes Agent structures week 3 with more debugging exercises than any other week in the first half of the course — not because memory is inherently more complicated than tool use or planning, but because its failures are so easy to miss until something later in the pipeline breaks in a way that seems unrelated.

Getting the Most Out of Week 3

Students who get the most value out of this week tend to do a few things consistently. They keep a running log of what their agent "remembered" incorrectly during testing, rather than fixing bugs ad hoc as they notice them. They resist the urge to add a vector database on day one, building the simpler flat-file or database version first so they actually understand what problem the vector store is solving. And they treat the token-budgeting exercise as seriously as the memory-extraction one, since an agent that forgets things gracefully under a tight budget is far more useful than one that occasionally throws a context-length error in front of a real user.

The broader theme running underneath week 3 — one that Ira Menon reinforces throughout the accompanying lecture material — is that a genuinely useful agent isn't defined by how clever its single responses are. It's defined by whether it stays coherent across the tenth message, the hundredth session, and the third day in a row a user comes back to it. Memory and context engineering are the unglamorous plumbing that makes that coherence possible, and week 3 is where that plumbing gets built by hand, once, so it's understood rather than just imported from a framework.

If this kind of hands-on, code-first treatment of the systems underneath modern AI agents is useful to you, it's exactly the approach the rest of the curriculum follows too. Week 3 is one part of a thirty-day arc — from tool use, to reasoning loops, to memory, to multi-agent coordination — built for people who want to actually understand what they're shipping. You can find the full curriculum, including this week's labs and the debugging exercises referenced above, inside 30 Days of Hermes Agent.