Agent Memory, Planning and Tools: How the Hermes Agent Handles All Three

Every AI agent demo looks the same for the first thirty seconds. The model calls a tool, gets a result, and produces a clean answer. Then somebody asks it to do something that takes four steps, touches an API that occasionally times out, and requires remembering a decision made two minutes ago. That is where most agents fall apart, and it is exactly where the interesting engineering lives. An agent is not a chatbot with function-calling bolted on. It is a system with three cooperating subsystems — tools, planning, and memory — and the quality of an agent is really the quality of how those three talk to each other. In the Hermes agent, which we use as the working reference architecture throughout this piece, every one of these subsystems is a first-class component with its own failure modes, its own schema, and its own recovery path. Let's take them apart one at a time.

Tools: teaching an agent what it's allowed to touch

A tool is the boundary between the model's reasoning and the real world. Get the boundary wrong and you get an agent that either refuses to act or acts recklessly. Hermes treats tool design as an interface contract problem, not a prompt-engineering problem.

How the agent decides when and which tool to call

The decision to call a tool happens in two stages. First, the model has to recognize that its own knowledge or reasoning is insufficient — this is the "do I need help" gate. Second, given that it needs help, it has to pick the right tool from the registered set — the "which help" gate.

The first gate is mostly a function of system prompt discipline. Hermes' system prompt explicitly enumerates categories of things the model must never answer from parametric memory: current prices, live system state, anything that depends on today's date, anything that requires a side effect (sending an email, writing a file, hitting a database). This sounds obvious, but it is the single highest-leverage lever in tool-calling reliability. Models are eager to please, and an ungated model will confidently hallucinate a plausible-looking API response rather than admit it needs to call one.

The second gate — tool selection — is where schema quality matters more than model quality. Each tool in Hermes exposes:

• A name that reads like a verb phrase (search_knowledge_base, not kb_tool_2)
• A one-line description optimized for disambiguation against neighboring tools, not for completeness
• A strict parameter schema with types, enums where possible, and required versus optional fields marked explicitly
• Usage examples embedded in the description for the tools that are commonly confused with each other

That last point matters more than people expect. If you have both search_web and search_docs, the model will misfire between them constantly unless the descriptions state the boundary condition explicitly — "use search_docs only for content already indexed in the user's workspace; use search_web for anything external or time-sensitive." Vague tool descriptions are the number one cause of wrong-tool selection, not model capability.

A minimal tool-selection pattern

Here is a simplified version of the routing logic Hermes uses before it ever reaches the LLM's own function-calling — a pre-filter that narrows the candidate tool set based on intent classification, which dramatically reduces the chance of the model picking a plausible-but-wrong tool out of a large registry:

TOOLS = {
    "get_weather": {
        "triggers": ["weather", "temperature", "forecast", "rain"],
        "schema": {"location": "string", "date": "string?"},
    },
    "search_docs": {
        "triggers": ["documentation", "internal", "workspace", "our"],
        "schema": {"query": "string", "top_k": "int?"},
    },
    "send_email": {
        "triggers": ["email", "send", "notify", "message"],
        "schema": {"to": "string", "subject": "string", "body": "string"},
    },
}

def shortlist_tools(user_message: str, tools: dict) -> list[str]:
    message = user_message.lower()
    scored = []
    for name, spec in tools.items():
        hits = sum(1 for kw in spec["triggers"] if kw in message)
        if hits > 0:
            scored.append((hits, name))
    scored.sort(reverse=True)
    # only pass the top 3 candidates into the model's own tool-calling context
    return [name for _, name in scored[:3]]

def decide_and_call(user_message: str):
    candidates = shortlist_tools(user_message, TOOLS)
    if not candidates:
        return {"action": "answer_directly"}
    # the LLM only sees these candidate schemas, not the full registry
    return {"action": "call_llm_with_tools", "tool_subset": candidates}

This is deliberately simple — a keyword shortlist, not a classifier — but the principle generalizes: narrowing the decision space before the model has to reason about it improves accuracy more than any amount of prompt tuning on the full registry. Hermes runs a lightweight embedding-similarity version of this in production, but the keyword version is a fine mental model and a fine starting point for a student project.

Schema design as a reliability lever

A tool schema is a contract, and contracts should be strict. Hermes enforces JSON Schema validation on every tool call before it is executed — not after. If the model emits a malformed argument (a string where an integer is expected, a missing required field, an enum value that doesn't exist), the call never reaches the underlying function. Instead, the validation error is fed back into the model's context as a structured correction prompt: "Your call to get_weather failed validation: date must match YYYY-MM-DD, got tomorrow. Retry with a corrected value."

This retry-on-validation-failure loop is cheap, fast, and catches the majority of tool-calling errors before they ever touch a real API, a real inbox, or a real database write. The alternative — letting malformed calls hit production systems and handling the fallout downstream — is strictly worse and considerably more expensive to debug.

Error handling when a tool actually fails

Validation failures are the easy case. The harder case is when the call is well-formed but the tool itself fails: a timeout, a 500 from a downstream API, a rate limit, a permissions error. Hermes classifies tool failures into three buckets, because each one calls for a different agent behavior:

1. Transient failures (timeouts, 429s, flaky network) — retried automatically with exponential backoff, capped at two or three attempts, invisible to the planning layer unless all retries are exhausted.
2. Deterministic failures (permission denied, resource not found, invalid state) — never retried blindly. These are surfaced immediately to the planning layer as a hard stop, because retrying an operation that will fail identically every time just burns latency and tokens.
3. Ambiguous failures (empty result set, partial data, a tool that "succeeds" but returns something useless) — the trickiest bucket. Hermes handles these by having the calling agent evaluate the tool's *output* against the *intent* of the call, not just the HTTP status. An empty search result for "find the invoice from March" is technically a successful call and a failed task — the agent needs a critic step (more on this below) to catch that distinction.

The core discipline is: never let a tool failure silently become a hallucinated answer. If search_docs returns nothing, the correct agent behavior is "I couldn't find that in the documentation," not a fabricated summary that sounds like it came from a real document. This is enforced structurally in Hermes by requiring every tool-derived claim in the final answer to be traceable back to a specific tool call's output — if it isn't there, it doesn't get said.

Planning: turning a goal into a sequence an agent can actually execute

Tools give an agent hands. Planning gives it a reason to move them in a particular order. A goal like "reconcile this month's expenses against the budget and flag anything over 10%" is not a single action — it is a sequence of actions with dependencies, and the agent needs an explicit representation of that sequence, not an implicit one buried in a single giant prompt.

Breaking a goal into a step sequence

Hermes decomposes goals using a plan-then-execute pattern rather than asking the model to interleave planning and acting turn by turn. The reasoning is subtle but important: models that plan one step at a time tend to be locally greedy — they solve the step in front of them without considering whether it sets up the next one correctly. An explicit upfront plan forces the model to reason about the whole dependency graph before committing to the first action.

A typical Hermes plan for the expense-reconciliation example looks like this internally:

{
  "goal": "Reconcile March expenses against budget, flag overages > 10%",
  "steps": [
    {"id": 1, "action": "fetch_transactions", "args": {"month": "2026-03"}, "depends_on": []},
    {"id": 2, "action": "fetch_budget", "args": {"month": "2026-03"}, "depends_on": []},
    {"id": 3, "action": "compute_category_totals", "args": {"from": 1}, "depends_on": [1]},
    {"id": 4, "action": "diff_against_budget", "args": {"from": [2, 3]}, "depends_on": [2, 3]},
    {"id": 5, "action": "flag_overages", "args": {"threshold": 0.10, "from": 4}, "depends_on": [4]},
    {"id": 6, "action": "summarize_for_user", "args": {"from": 5}, "depends_on": [5]}
  ]
}

Notice steps 1 and 2 have no dependency on each other — they can run in parallel, and Hermes' executor does exactly that, which is a meaningful latency win once agents start chaining more than three or four tool calls. The plan is a directed acyclic graph, not a list, even though it is often rendered as a numbered list for the user.

The planner, executor, critic pattern

Hermes splits agentic reasoning into three logical roles, which can be three separate model calls, three separate prompts against the same model, or in the lightest-weight configuration, three phases within one continuous reasoning trace:

• The planner takes the user's goal and produces the step graph above. It has access to the tool registry (so it knows what's achievable) but does not execute anything.
• The executor takes one step at a time, resolves its dependencies from prior step outputs, calls the appropriate tool, and records the result.
• The critic evaluates whether the *execution so far* is actually converging on the *original goal* — not just whether individual tool calls succeeded, but whether the plan is still the right plan given what's been learned.

The critic is the piece most teams skip, and it's the piece that matters most once agents run for more than a handful of steps. A plan made before any information was gathered is a plan made under uncertainty. Step 1 might reveal that "March expenses" actually spans two different ledger systems because of a mid-month migration — something the planner had no way of knowing in advance. Without a critic, the executor barrels through the original plan and produces a confidently wrong reconciliation. With a critic, that discrepancy gets caught after step 1 and triggers a replan.

Replanning when a step fails

Replanning in Hermes is scoped, not global. When a step fails or the critic flags a divergence, the system does not throw away the entire plan and start over — that's expensive and it discards perfectly good completed work. Instead:

1. The failed or flagged step and everything downstream of it (per the dependency graph) is marked stale.
2. Everything upstream — already-executed steps with valid outputs — is kept as context.
3. A new, smaller planning call is made: "Given goal G, and given that steps 1-2 produced X and Y, but step 3 failed because Z, produce a revised plan for steps 3 onward."

This local-replan approach keeps token cost bounded and keeps the agent's behavior legible — a full replan tends to produce a structurally different plan each time, which makes debugging agent behavior much harder for the person building on top of it. Scoped replanning also means the "why did the agent change its approach" question has a clean answer: point at the specific step that failed and the specific new sub-plan it produced in response.

There is a failure mode worth naming explicitly: replan thrashing, where a flaky tool causes the agent to replan, hit the same flaky tool, replan again, and loop. Hermes caps replanning at a fixed depth (typically two or three replans per goal) and, on exceeding that cap, surfaces the failure to the user rather than continuing to spin. An agent that knows when to stop and ask for help is more trustworthy than one that never gives up, because the ones that never give up are usually the ones burning your API budget on a problem they were never going to solve.

Memory: short-term, long-term, and procedural, and how they keep an agent from repeating itself

Tools and planning handle a single task well. Memory is what makes an agent good across many tasks, and across time. Without it, every conversation starts from zero, every mistake gets made again, and every user preference has to be restated. Hermes treats memory as three distinct systems with different retention, different retrieval mechanisms, and different write triggers — not one undifferentiated "context."

Short-term memory: the conversation buffer

Short-term memory is the rolling window of the current interaction — the last N turns, or everything since the goal was stated, whichever is smaller. This is the cheapest form of memory and the one every agent has by default, simply because it's the model's own context window.

The engineering problem here isn't storage, it's pruning. A naive agent that stuffs every tool call's full raw output into the conversation buffer runs out of context window within a handful of steps, especially when tool outputs include large JSON payloads or document excerpts. Hermes applies a summarization pass after each tool call: the raw output is stored in a scratch buffer (available for the current task only), and a condensed, structured summary — just the fields relevant to the goal — is what actually goes into the rolling conversation buffer that future turns see. This keeps the working context lean without losing the ability to recover raw data if a later step needs it.

Short-term memory is intentionally volatile. It does not survive past the end of the session unless something promotes specific facts out of it — which is exactly the handoff to long-term memory.

Long-term memory: durable facts in a vector store

Long-term memory in Hermes is backed by a vector store holding two kinds of entries: durable facts (a user's timezone, a project's naming convention, a recurring vendor's correct category) and episodic summaries (a compressed record of what happened in a past session and how it ended). Retrieval happens via semantic similarity search against the current goal, pulling back the top handful of relevant memories before the planner even starts building its step graph.

The write side is the part people underestimate. Not everything in a conversation deserves to become a long-term memory — if it did, the store would fill with noise and retrieval quality would degrade. Hermes gates writes to long-term memory behind an explicit "is this durable" check, run as a cheap classification pass at the end of a session: does this fact generalize beyond this one conversation? Is it something the user would be annoyed to have to repeat? If yes to either, it gets written; otherwise it's discarded with the rest of the session's short-term buffer.

A minimal memory write/retrieve pattern

Here's a simplified version of the write-then-retrieve loop, using a vector store abstraction that should map onto whatever embedding backend you're using (Hermes uses a managed pgvector setup, but the pattern is backend-agnostic):

from datetime import datetime

def write_memory(store, text: str, metadata: dict):
    embedding = embed(text)  # your embedding model of choice
    store.upsert(
        vector=embedding,
        payload={
            "text": text,
            "type": metadata.get("type", "fact"),   # "fact" | "episode" | "mistake"
            "created_at": datetime.utcnow().isoformat(),
            "tags": metadata.get("tags", []),
        },
    )

def retrieve_memory(store, query: str, k: int = 5, min_score: float = 0.75):
    query_vec = embed(query)
    results = store.search(query_vec, top_k=k)
    # filter low-similarity noise before it ever reaches the planner's context
    return [r.payload for r in results if r.score >= min_score]

# --- usage inside the agent loop ---

def before_planning(store, goal: str):
    relevant = retrieve_memory(store, goal)
    return "\n".join(f"- {m['text']}" for m in relevant)

def after_session(store, session_summary: str, was_a_mistake: bool):
    write_memory(
        store,
        text=session_summary,
        metadata={"type": "mistake" if was_a_mistake else "episode"},
    )

The detail that makes this useful rather than decorative is the type: "mistake" tag. Hermes writes a distinct memory entry whenever a plan required a correction — a tool call that failed in a non-obvious way, a critic-triggered replan, a case where the user had to intervene. These get retrieved and surfaced to the planner *before* it builds a new plan for a similar-looking goal, effectively turning "we tried this and it didn't work" into a piece of retrievable context rather than a lesson that evaporates with the session.

Procedural memory: learned skills, not just learned facts

The third layer is the one most agent frameworks skip entirely, and it's arguably the most valuable at scale. Procedural memory isn't a fact or an episode — it's a reusable *method*. If an agent has successfully executed a five-step plan for "reconcile expenses" three times, with only minor variations, that plan itself becomes worth storing as a template: a named, parameterized procedure that the planner can retrieve and instantiate directly, skipping the full from-scratch decomposition.

Hermes builds procedural memory bottom-up: successful plans that hit a repetition threshold (the same goal shape solved cleanly more than twice) get promoted into a procedure library, stored alongside the tool registry itself. The planner checks this library before doing full decomposition — if a matching procedure exists, it's used as a starting template and only the leaf-level arguments get re-planned, which is both faster and more reliable than re-deriving the whole graph every time.

How the three memories interact so mistakes don't repeat

The real payoff shows up in the interaction between the layers, not any single one of them. A concrete sequence:

1. Short-term memory notices, mid-session, that a tool call to fetch_transactions returned a permissions error for a specific ledger.
2. The critic flags this as a plan-invalidating failure and triggers a scoped replan.
3. At session end, this gets written to long-term memory as a mistake-tagged entry: "ledger X requires a different auth scope than ledger Y."
4. Next time a similar goal is planned — even weeks later, even in a different conversation — retrieval surfaces that fact before the planner builds its graph, so the plan accounts for the auth difference from step one instead of rediscovering it the hard way.
5. If this exact planning correction recurs across enough sessions, it eventually gets folded into the procedural template itself, so it stops being a retrieved fact and becomes a baked-in step of the default plan.

That progression — short-term catches it, long-term remembers it, procedural memory eventually absorbs it into the default behavior — is the actual mechanism by which an agent gets better over time instead of just getting longer transcripts. Without this pipeline, every session is opening night. With it, mistake number one becomes the last time that mistake happens.

Putting it together

None of these three pillars is impressive in isolation. A tool-calling agent with no planning is a chatbot with extra steps. A planner with no memory replans the same mistake every single session. Memory with no critic just accumulates noise nobody ever retrieves correctly. The reason Hermes behaves like a competent junior engineer instead of a very fast but forgetful intern is that tools, planning, and memory are wired into each other as a loop: plans decide which tools get called, tool failures trigger replans, and replans get written back into memory so the next plan starts smarter than the last one did.

If you want to build this kind of system yourself rather than just read about it, 30 Days of Hermes Agent walks through the entire architecture end to end — tool schema design, the planner/executor/critic loop, and all three memory layers — with a working agent you build incrementally, day by day, until it's doing exactly what's described above.