Building an Autonomous Agent from Scratch: The Hermes Agent Architecture

Most people who say they've "built an agent" have actually just imported one. They pip installed a framework, copy-pasted a quickstart, swapped in their own system prompt, and shipped it. That's fine for a demo. It's a liability the moment something breaks in production and you're staring at a stack trace three abstraction layers deep with no idea what the framework is actually doing under the hood. The fastest way to actually understand agents — not just operate them — is to build one from scratch, with no framework, using nothing but a language model API and a while loop. That's what this article walks through: the Hermes Agent architecture, a minimal but complete blueprint for an autonomous agent, built up piece by piece so you can see exactly what every "batteries included" framework is hiding from you.

Why skip the framework

LangChain, AutoGPT-style wrappers, and the dozen agent SDKs that showed up in the last two years all do the same handful of things: they call an LLM, parse its output, decide whether to call a tool, execute the tool, feed the result back in, and repeat until some condition says stop. That's the entire trick. Everything else — the retry logic, the callback managers, the tracing decorators, the chain-of-chains abstractions — is convenience scaffolding on top of that one idea.

The problem with learning agents through a framework first is that the framework's abstractions become your mental model of what an agent *is*. You start thinking in terms of "AgentExecutor" and "Runnable" instead of "the model produced text, I parsed it, I ran something, I fed the result back." When the abstraction leaks — and it always leaks, usually at 2am before a demo — you have no first-principles understanding to fall back on. You just have Stack Overflow.

Building from scratch inverts this. Once you've written the loop yourself, every framework you touch afterward reads like a variation on a theme you already know cold. You'll recognize LangChain's AgentExecutor as "my while loop, but with more indirection." You'll see CrewAI's task delegation as "multiple copies of my loop talking to each other." Nothing is magic anymore, which means nothing is scary anymore either. That's the actual goal: not anti-framework dogma, but framework-independence.

The core loop: perceive, reason, act, observe

Strip away every fancy term and an autonomous agent is a cycle with four steps repeated until a goal is satisfied:

• Perceive: gather the current state — the user's request, the results of previous actions, any new information from the environment.
• Reason: ask the model what to do next, given everything perceived so far.
• Act: execute whatever the model decided — call a function, run a query, write a file, or simply respond.
• Observe: capture the result of that action and feed it back into the next perception step.

This is not a novel idea. It's the classic PEAS framework from control theory and robotics (perceive-act loops go back to cybernetics in the 1940s), repackaged for LLMs. What changed with large language models is that the "reason" step, which used to require a hand-built decision tree or a symbolic planner, can now be handled by a general-purpose model that reads text and produces text. The loop is old. The reasoning engine inside it is new.

Here's the important part that most tutorials gloss over: an LLM by itself is not an agent. A single call to chat.completions.create is a function — stateless, one-shot, no memory of what it did a moment ago. An agent only exists once you wrap that call in a loop that persists state across iterations and lets the model's own output determine what happens next. The "agentic" part isn't the model. It's the loop, plus the decision that the model's output controls the loop's next action rather than a human controlling it turn by turn.

That distinction matters because it tells you exactly what you need to build: a state object, a call to the model, a router that inspects the model's output, and a termination check. Four things. No proprietary framework required.

Tool use: function calling from scratch

The first capability that turns a chatbot into an agent is the ability to act on the world instead of just describing what it would do. This is "tool use," and every framework builds an elaborate abstraction around it, but the mechanism is simple: you give the model a list of functions it can request (name, description, and parameter schema), the model responds with structured intent to call one, you actually call it in your own code, and you feed the return value back in as context.

Modern LLM APIs support this natively via a tools parameter, but you can build the same thing manually even against a model with no native tool-calling support — you just instruct it to emit a specific JSON shape when it wants to act, and you parse that JSON yourself. That's worth doing once, purely so you understand what "function calling" is actually buying you when a provider does support it natively: structured output guarantees, not new capability.

import json

def get_weather(city: str) -> str:
    # pretend this hits a real API
    return f"{city}: 22C, clear skies"

TOOLS = {
    "get_weather": get_weather,
}

TOOL_SCHEMA = [
    {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    }
]

def call_model(messages):
    # stand-in for a real API call — returns either
    # {"tool_call": {"name": ..., "arguments": {...}}} or {"final": "..."}
    ...

def execute_tool_call(tool_call: dict) -> str:
    name = tool_call["name"]
    args = tool_call["arguments"]
    if name not in TOOLS:
        return f"error: unknown tool {name}"
    try:
        return str(TOOLS[name](**args))
    except Exception as e:
        return f"error: {e}"

Notice the try/except in execute_tool_call. This is not defensive boilerplate — it's load-bearing. The model will, with total confidence, call your function with the wrong argument types, missing required fields, or a city that doesn't exist. If your tool execution layer throws an unhandled exception, your entire agent process dies. If it instead catches the error and returns a string describing what went wrong, the model gets a chance to self-correct on the next turn — "oh, that city name was misspelled, let me try again." Error handling at the tool boundary is what separates an agent that recovers from a fragile script that crashes on the first malformed call.

The other detail worth internalizing: the tool schema you send the model is the interface contract. A vague description ("gets data") produces vague, wrong calls. A precise description with explicit parameter types and a couple of example values in the description string produces reliable calls. Prompt engineering doesn't stop at the system prompt — it extends into every tool description you write.

Memory: from a conversation buffer to a durable store

The simplest form of agent memory is just the message list you're already passing to the API on every call — a running transcript of user turns, assistant turns, and tool results. This is enough for a single session and is genuinely all you need for most agents. Don't reach for a vector database because a blog post told you memory means embeddings. Start with a list.

messages = [{"role": "system", "content": SYSTEM_PROMPT}]

def remember(role, content):
    messages.append({"role": role, "content": content})

The limitation shows up fast: context windows are finite, and every tool call result you append makes the next API call more expensive and slower. Two problems emerge — cost/latency, and the model's attention getting diluted across an ever-growing transcript. The fix isn't a new architecture, it's summarization: periodically compress older turns into a shorter summary and replace them in the buffer, keeping only recent turns verbatim.

def compact_if_needed(messages, max_turns=20):
    if len(messages) <= max_turns:
        return messages
    system = messages[0]
    old = messages[1:-10]
    recent = messages[-10:]
    summary = summarize(old)  # one more model call, condensing `old`
    return [system, {"role": "system", "content": f"Earlier context: {summary}"}] + recent

That covers memory *within* a run. Durable memory — remembering facts across sessions, days later — is a different problem: persistence. The minimal version is a JSON file or a SQLite table keyed by user or session ID, storing facts extracted from past runs ("user prefers Python," "user's project uses Postgres"). You read relevant facts back in as part of the system prompt on the next run. This is the same idea as a vector store with semantic retrieval, just without the vector part — and for most agents, exact or keyword-based retrieval over a small structured store outperforms embedding search anyway, because it's debuggable. You can look at the file and see exactly what the agent "remembers." A vector index is a black box until it isn't retrieving the right thing, and then it's a very hard black box to debug. Reach for embeddings only once keyword/structured retrieval demonstrably fails — not by default.

Planning: breaking a goal into steps

Give a model a vague, multi-step goal — "research competitor pricing and draft a summary" — and ask it to just go, and you'll often get a shallow, one-shot attempt that stops after the first sub-task looks done. Planning is the fix: an explicit step where the model decomposes the goal into an ordered list of sub-tasks *before* any action is taken, and that plan becomes part of the state the agent tracks across iterations.

The mechanism is unremarkable and that's the point — it's just another model call, with a prompt engineered to produce a structured list instead of a direct answer:

PLAN_PROMPT = """Break the following goal into a numbered list of concrete,
independently-verifiable steps. Return only the numbered list.

Goal: {goal}"""

def make_plan(goal: str) -> list[str]:
    raw = call_model([{"role": "user", "content": PLAN_PROMPT.format(goal=goal)}])
    return [line.strip() for line in raw.splitlines() if line.strip()]

Once you have a plan, the outer loop changes shape slightly: instead of looping on "keep going until the model says it's done," you loop over the plan steps, and for each step you run the full perceive-reason-act-observe cycle until that specific step is satisfied, then advance to the next one. This is the difference between a reactive agent (respond to whatever's in front of you) and a planning agent (maintain a goal structure and work through it deliberately). Planning agents are more predictable and more debuggable, because at any point you can print the plan and the current step index and know exactly where the agent is and why it did what it just did.

Re-planning matters too. Plans made from an initial, incomplete understanding of a problem are often wrong by step three. A mature planning loop checks, after each step, whether the remaining plan still makes sense given what was just learned, and re-generates it if not — rather than stubbornly executing a stale plan.

Self-correction: checking your own output against the goal

This is the step that separates an agent from a script that calls an LLM in a loop. Self-correction means the agent evaluates its own intermediate output against the original goal *before* declaring success, and loops back to fix it if the check fails.

The cheapest version of this is a second model call — a critic pass — where you show the model its own draft output next to the original goal and ask it to judge whether the goal is actually satisfied, in a structured, parseable format.

CRITIC_PROMPT = """Goal: {goal}
Proposed output: {output}

Does this output fully satisfy the goal? Answer with exactly one line:
PASS
or
FAIL: <specific reason>"""

def self_check(goal: str, output: str) -> tuple[bool, str]:
    verdict = call_model([{"role": "user", "content": CRITIC_PROMPT.format(goal=goal, output=output)}])
    verdict = verdict.strip()
    if verdict.startswith("PASS"):
        return True, ""
    return False, verdict.removeprefix("FAIL:").strip()

Why does this work at all, if it's the same model checking its own homework? Two reasons. First, generation and evaluation are different tasks even when performed by the same model — producing an answer under the pressure of "solve this now" engages different failure modes than calmly comparing a finished answer against a checklist. Second, and more practically, the critic prompt can be far more specific than the original task prompt — you can hand it a rubric, ask it to check for missing sub-goals, factual inconsistency with earlier tool outputs, or unaddressed constraints from the original request. A generic "are you done?" self-check is weak. A specific, itemized rubric check is genuinely useful, and it's nearly free compared to the cost of shipping a wrong answer.

The failure reason string matters as much as the pass/fail verdict. "FAIL: doesn't include Q3 numbers" is something the next generation pass can act on directly. A bare boolean gives the agent nothing to correct toward — it'll just try again and likely produce the same gap. Always make your self-correction step return an actionable reason, not just a verdict.

Stopping conditions: the detail everyone skips

This is the section that separates agents that work in production from agents that rack up a four-figure API bill overnight because they got stuck reasoning in circles. It is also, reliably, the part every "build an agent in 20 minutes" tutorial skips, because it doesn't demo well — a stopping condition only proves its worth when something has already gone wrong.

An LLM has no innate sense of "I've been trying this for too long." Left unchecked, an agent loop can fail in at least four distinct ways, and you need a distinct guard for each:

• Max iteration count: the simplest and most essential guard — a hard ceiling on how many times the loop can run, full stop, regardless of what the model claims. This alone prevents runaway cost.
• No-progress detection: the loop is under the iteration cap but the agent is repeating the same tool call with the same arguments, or oscillating between two states. Cap-based limits don't catch this; you need to compare each new action against recent history.
• Goal-satisfaction check: the positive stopping condition — the self-correction pass from the previous section returning PASS. This is the "happy path" exit and should be checked before the negative guards, since most runs should end here.
• Wall-clock / cost budget: independent of iteration count, because a single iteration can itself be expensive (a large tool call, a long generation). Track elapsed time or cumulative token spend and stop when a budget is exhausted.

import time

def should_stop(state: dict) -> tuple[bool, str]:
    if state["iterations"] >= state["max_iterations"]:
        return True, "max_iterations_reached"

    if state["goal_satisfied"]:
        return True, "goal_satisfied"

    if time.time() - state["start_time"] > state["max_seconds"]:
        return True, "time_budget_exceeded"

    recent_actions = state["action_history"][-4:]
    if len(recent_actions) == 4 and len(set(recent_actions)) == 1:
        return True, "no_progress_repeated_action"

    if state["total_tokens_used"] > state["max_tokens"]:
        return True, "token_budget_exceeded"

    return False, ""

Every one of these checks is cheap and every one of them has, in practice, saved a real agent from a real production incident. The iteration cap catches the model stuck in a "let me try one more thing" spiral. The no-progress check catches the more insidious case where the model *thinks* it's making progress but is actually calling the same broken tool the same broken way on repeat. The time and token budgets catch cases where individual steps are each reasonable but the aggregate has quietly become unaffordable.

Put the should_stop check at the very top of every loop iteration, before you even call the model again — not as an afterthought at the bottom. And always log the stop reason. When something looks wrong in production, "stopped: no_progress_repeated_action after 6 iterations" tells you exactly where to look. A silent timeout tells you nothing.

Wiring it together: the minimal agent loop

Here's the entire agent, all four pillars — tool use, memory, and a stopping condition — collapsed into one runnable shape. This is deliberately close to the floor of what "an agent" can mean while still being a real one:

def run_agent(goal: str, max_iterations: int = 10) -> str:
    messages = [
        {"role": "system", "content": "You solve tasks using tools. Call a tool or give a final answer."},
        {"role": "user", "content": goal},
    ]
    iterations = 0

    while iterations < max_iterations:
        response = call_model(messages)          # reason
        iterations += 1

        if "final" in response:
            return response["final"]              # goal_satisfied exit

        tool_call = response["tool_call"]
        result = execute_tool_call(tool_call)      # act
        messages.append({"role": "assistant", "content": json.dumps(tool_call)})
        messages.append({"role": "tool", "content": result})  # observe -> perceive

    return "stopped: max_iterations_reached"

Every framework you'll ever adopt is, structurally, a more elaborate version of this same thirty lines. messages is memory. call_model is reasoning. execute_tool_call is action. The while condition is your stopping guard. Planning and self-correction slot in as additional model calls layered around this same skeleton — a planning call before the loop starts, a critic call before returning the final answer. Nothing about scaling this up to multi-agent systems, longer-running workflows, or production reliability changes the shape. It adds guards, retries, observability, and orchestration around a loop that looks almost exactly like the one above.

What breaks first when you take this to production

Building the loop is the easy 20%. The failure modes that show up once real users hit a real agent are where the actual engineering lives, and they cluster around a few predictable spots.

• Tool call hallucination: the model invents a tool that doesn't exist, or invents parameters your schema never defined. Your dispatcher needs to fail loudly to logs but gracefully to the model — return a clear error string, never crash.
• Context window exhaustion: without the compaction step covered earlier, long-running agents silently truncate their own history and start "forgetting" the original goal midway through a task.
• Cost runaway: this is the stopping-condition section again, but it deserves repeating because it's the single most common production incident with home-built agents — no framework saves you from this, only your own guards do.
• Silent partial success: an agent that completes 80% of a task and reports full success because nothing in the loop ever checked. This is exactly what the self-correction pass exists to catch, and it's the step people cut first when they're in a hurry.

None of these are exotic. They're the direct, predictable consequences of skipping one of the six pillars covered above. If you've built the loop with all six in place — perceive/reason/act/observe, tools, memory, planning, self-correction, and stopping conditions — you've already inoculated yourself against the failure modes that take down most first agents.

Where to go from here

Everything above is deliberately minimal — enough to see the whole shape of an autonomous agent without a framework hiding any of the moving parts. Real production agents add plenty on top of this: parallel tool execution, retry-with-backoff, structured tracing, multi-agent handoffs, sandboxed code execution. But every one of those additions is comprehensible, and buildable, once this base loop is solid — and none of it is comprehensible if the base loop is a black box you copy-pasted from a template.

If you want to go deeper than a single blog post can take you — actually building out durable memory stores, multi-step planning with re-planning, tool sandboxing, and a production-grade self-correction pipeline, with real projects and real code review — that's exactly what 30 Days of Hermes Agent is built for. It's the guided, in-depth version of everything covered here: the same first-principles, no-framework philosophy, taken all the way from a thirty-line loop to a deployable autonomous agent, with Pramod Dutta and Ira Menon walking through the architecture decisions step by step.