Hermes Agent Week 4 Deep Dive: Shipping to Production

The gap between "it works on my laptop" and "it works"

There is a very specific kind of silence that happens in a cohort call around day 22 of any serious agent-building course. Students have spent three weeks building something genuinely impressive: a tool-using agent that plans, calls functions, retries on failure, and remembers context across a session. It runs beautifully in a notebook. Then someone asks the obvious question: "So how do I actually put this in front of users?" And the room goes quiet, because building an agent and operating an agent are two completely different disciplines.

Week 4 of 30 Days of Hermes Agent exists to close exactly that gap. The first three weeks of the course are about capability — reasoning loops, tool calling, memory, retrieval, multi-step planning. Week 4 is about custody. It's where we stop asking "can the agent do the task" and start asking "what happens when it does the task wrong, at 2 a.m., for a paying customer, while the upstream API is down." This article walks through what we actually cover during that week: the deployment pipeline, the monitoring stack, the guardrail layers, and the capstone demo that closes out the course. If you've been through the first three weeks of Hermes or you're evaluating whether the course is worth your time, this is the week that separates a portfolio project from a production system.

Day 22-23: Packaging the agent for deployment

The first two days of Week 4 are deliberately unglamorous. Before anyone talks about scaling or observability, we force students to answer a boring but load-bearing question: how does this agent actually run outside of a Jupyter cell?

We start by wrapping the Hermes agent loop in a proper service boundary. Students take the agent core they built in Week 2 — the planner, the tool router, the memory store — and put a thin API layer in front of it. We use FastAPI for this because it gives students async support out of the box, which matters once tool calls start hitting external APIs with real latency.

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from hermes.agent import HermesAgent
from hermes.config import load_settings

app = FastAPI(title="Hermes Agent Service")
settings = load_settings()
agent = HermesAgent(settings=settings)

class AgentRequest(BaseModel):
    session_id: str
    message: str

class AgentResponse(BaseModel):
    session_id: str
    reply: str
    tool_calls: list[str]

@app.post("/v1/agent/invoke", response_model=AgentResponse)
async def invoke_agent(payload: AgentRequest) -> AgentResponse:
    try:
        result = await agent.run(
            session_id=payload.session_id,
            user_message=payload.message,
        )
    except TimeoutError:
        raise HTTPException(status_code=504, detail="agent timed out")
    return AgentResponse(
        session_id=payload.session_id,
        reply=result.final_message,
        tool_calls=[t.name for t in result.tool_trace],
    )

That snippet looks simple, and it's meant to. The teaching point isn't the FastAPI wiring — it's everything students are forced to decide around it: what's the timeout budget for a full agent turn, what status code do you return when a tool call fails mid-plan, how do you keep session_id from leaking across users. We spend a full lecture just on session isolation, because it's the single most common mistake we see in student submissions from earlier cohorts — agents that accidentally shared conversation state across concurrent requests under load.

From there we containerize. Every student builds a Dockerfile for their agent service, and we're strict about multi-stage builds so the shipped image doesn't carry build tooling or dev dependencies. We also introduce environment-based configuration at this stage — API keys, model selection, tool allowlists all move out of code and into environment variables validated at startup with Pydantic settings. An agent that crashes on a missing environment variable at 3 a.m. in production is a much better failure mode than one that silently falls back to a default and misbehaves for six hours before anyone notices.

Day 24: Deployment pipelines and rollout strategy

Day 24 is where we get into actual deployment mechanics. We don't lock students into one cloud provider — Hermes agents get deployed to whatever the student is already comfortable with, whether that's a container platform, a serverless function runtime, or a small VM behind a reverse proxy — but we do standardize the deployment pattern regardless of target.

The pattern is: build, tag, stage, canary, promote. Every agent change goes through a CI pipeline that runs the eval suite from Week 3 (more on that below) before it's allowed to build an image. The image gets tagged with the git commit SHA, never latest, so rollback is a one-line operation. We deploy to a staging environment first, run a smoke-test script against it, then roll a small percentage of production traffic to the new version before promoting fully.

name: deploy-hermes-agent

on:
  push:
    branches: [main]

jobs:
  test-and-eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install dependencies
        run: pip install -r requirements.txt
      - name: Run unit tests
        run: pytest tests/unit -q
      - name: Run agent eval suite
        run: python -m hermes.evals.run --suite regression --fail-under 0.9

  build-and-push:
    needs: test-and-eval
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build image
        run: docker build -t hermes-agent:${{ github.sha }} .
      - name: Push image
        run: docker push hermes-agent:${{ github.sha }}

  deploy-canary:
    needs: build-and-push
    runs-on: ubuntu-latest
    steps:
      - name: Roll 10% traffic to new version
        run: ./scripts/canary_rollout.sh hermes-agent:${{ github.sha }} 10
      - name: Watch error budget for 15 minutes
        run: ./scripts/watch_slo.sh --window 15m --max-error-rate 0.02
      - name: Promote to 100%
        run: ./scripts/canary_rollout.sh hermes-agent:${{ github.sha }} 100

The --fail-under 0.9 flag on the eval suite step is intentional and it's a rule we hold students to firmly: if your regression eval score drops below the threshold you set for your agent, the pipeline blocks the deploy. This is the first time in the course students experience an automated gate that can say no to their own code, and it's an important professional habit to build early. An agent is not "done" because it compiles and passes a manual smoke test you ran once — it's done when it clears a bar you defined in advance and can't quietly lower under deadline pressure.

We also cover blue-green deployment as an alternative for students whose infrastructure doesn't support fine-grained canary percentages, and we talk honestly about the tradeoffs: canary gives you a smaller blast radius but takes longer to reach full confidence, blue-green gives you instant full rollback but exposes 100% of new traffic to any regression that your canary window would have caught.

Day 25: Structured logging and traceability

You cannot operate what you cannot see, and agents are unusually opaque compared to normal software. A REST endpoint either returns the right JSON or it doesn't. An agent might return a perfectly formatted, grammatically correct, entirely wrong answer — and nothing about the HTTP response will tell you that happened.

Day 25 is dedicated to instrumentation. We require every tool call, every planning step, and every model invocation to emit a structured log event with a shared trace ID. Students implement this as middleware around their tool router so they don't have to remember to add logging calls scattered through business logic.

import time
import uuid
import structlog

logger = structlog.get_logger()

class TracedToolRouter:
    def __init__(self, tools: dict):
        self.tools = tools

    async def call(self, tool_name: str, args: dict, trace_id: str) -> dict:
        call_id = str(uuid.uuid4())
        start = time.perf_counter()
        logger.info(
            "tool_call_started",
            trace_id=trace_id,
            call_id=call_id,
            tool=tool_name,
            args=args,
        )
        try:
            result = await self.tools[tool_name].run(**args)
            logger.info(
                "tool_call_succeeded",
                trace_id=trace_id,
                call_id=call_id,
                tool=tool_name,
                latency_ms=round((time.perf_counter() - start) * 1000, 2),
            )
            return result
        except Exception as exc:
            logger.error(
                "tool_call_failed",
                trace_id=trace_id,
                call_id=call_id,
                tool=tool_name,
                error=str(exc),
                latency_ms=round((time.perf_counter() - start) * 1000, 2),
            )
            raise

We push students toward structlog or an equivalent structured logging library specifically so the output is machine-parseable JSON rather than free-form strings. The reason is downstream: once logs are structured, they can be shipped straight into a dashboard or alerting system without a fragile regex layer trying to parse human-readable log lines. Students wire this trace ID all the way from the incoming HTTP request through every tool call and back out to the final response, so when something goes wrong, they can pull every event for a single user turn with one query instead of grepping through a firehose of interleaved logs from concurrent sessions.

We also cover what NOT to log — full prompt contents including anything a user typed that might contain personal data get redacted or hashed before they hit persistent storage, and we treat this as non-negotiable rather than an optional nice-to-have. Several students in past cohorts have gone on to handle real user data in production, and this is the point in the course where we make clear that logging discipline is a compliance issue, not just a debugging convenience.

Day 26: Monitoring, dashboards, and alerting

With structured logs flowing, Day 26 turns them into something a human can actually act on. We have students stand up a basic observability stack — metrics for latency percentiles, tool-call error rates, token usage per session, and cost per conversation — and put it behind a dashboard.

The specific tools vary by student preference (some use a hosted APM, some build on open-source Prometheus and Grafana), but the metric taxonomy is fixed across the cohort because it maps to the failure modes agents actually have:

• Latency percentiles (p50/p95/p99) per tool, not just for the whole request. A slow vector DB call buried inside a fast overall response still needs to be visible before it becomes the bottleneck at scale.
• Tool error rate, broken out by tool name, so a single flaky integration doesn't get averaged away into an overall healthy-looking number.
• Plan length distribution — how many steps the agent takes to complete a task. A sudden shift toward longer plans is often the earliest signal that a prompt change or a model upgrade has degraded reasoning quality, well before users start complaining.
• Token spend per session, alerted on a rolling window, because a bug that causes the agent to loop is both a UX failure and a budget failure simultaneously.
• Fallback / refusal rate — how often the agent declines to answer or falls back to a canned response, tracked over time to catch guardrails becoming too aggressive or too permissive.

We ask students to write at least three alert rules against these metrics with concrete thresholds, then intentionally break their own agent in staging — inject a slow tool, feed it a malformed response, force a rate limit — and confirm the alert actually fires and routes somewhere a human will see it. This "break it yourself first" exercise consistently produces the most feedback in course surveys, because it's the first time many students have closed the loop between "I added monitoring" and "monitoring actually caught something."

Day 27: Guardrails — input, output, and behavioral

Day 27 is the philosophical heart of Week 4. Guardrails are not a single feature you bolt on; they're layered checks at every boundary the agent touches, and we teach them as three distinct categories.

Input guardrails run before the agent sees a user message. This includes basic content moderation, prompt-injection pattern detection, and rate limiting per user or session. We're explicit with students that input filtering catches the obvious cases and nothing more — a determined adversary will get past keyword-based filters, so this layer is about raising the cost of attack, not eliminating it.

Output guardrails run after the agent produces a response but before it reaches the user. This is where we spend the most implementation time, because it's where real incidents happen — an agent confidently hallucinating a refund policy, or a tool call returning data the agent shouldn't be allowed to surface verbatim.

class OutputGuardrail:
    def __init__(self, banned_patterns: list[str], max_length: int = 4000):
        self.banned_patterns = banned_patterns
        self.max_length = max_length

    def check(self, response_text: str) -> tuple[bool, str | None]:
        if len(response_text) > self.max_length:
            return False, "response_too_long"

        lowered = response_text.lower()
        for pattern in self.banned_patterns:
            if pattern in lowered:
                return False, f"banned_pattern:{pattern}"

        return True, None

class GuardedAgent:
    def __init__(self, agent, guardrail: OutputGuardrail, fallback_message: str):
        self.agent = agent
        self.guardrail = guardrail
        self.fallback_message = fallback_message

    async def run(self, session_id: str, user_message: str):
        result = await self.agent.run(session_id, user_message)
        passed, reason = self.guardrail.check(result.final_message)
        if not passed:
            logger.warning("output_guardrail_blocked", reason=reason, session_id=session_id)
            result.final_message = self.fallback_message
        return result

Behavioral guardrails are the hardest category and the one students find most eye-opening. These constrain what the agent is allowed to *do*, not just what it's allowed to *say*. A tool-calling agent that can issue refunds, send emails, or modify a database needs hard limits independent of the model's judgment — a maximum refund amount enforced in code, a require-human-approval step for any destructive action, an allowlist of tools available in a given conversation context. We tell students plainly: never trust the model to self-limit an action that has real-world consequences. The limit has to live in code that the model cannot talk its way around.

We close Day 27 with a red-teaming exercise. Students trade agents with a partner in the cohort and spend an hour trying to break each other's guardrails — get the agent to reveal its system prompt, get it to call a tool outside its intended scope, get it to agree to something a policy should have blocked. Every cohort finds gaps. That's the point.

Day 28: Handling failure gracefully

Production agents fail constantly, and Day 28 is about designing for that reality rather than pretending it won't happen. We cover retry strategy with exponential backoff and jitter for transient tool failures, circuit breakers for tools that are down long enough that retrying is just wasted latency, and graceful degradation — what does the agent say when a tool it needs is unavailable, versus silently guessing.

import asyncio
import random

async def call_with_backoff(tool_fn, *args, max_retries=3, base_delay=0.5, **kwargs):
    last_exc = None
    for attempt in range(max_retries):
        try:
            return await tool_fn(*args, **kwargs)
        except (TimeoutError, ConnectionError) as exc:
            last_exc = exc
            delay = base_delay * (2 ** attempt) + random.uniform(0, 0.25)
            logger.warning("tool_retry", attempt=attempt, delay=delay)
            await asyncio.sleep(delay)
    raise last_exc

We also make students write an explicit degradation policy for their capstone agent: for each tool the agent depends on, what should the agent do if that tool is unreachable? Sometimes the right answer is "tell the user honestly that this feature is temporarily unavailable." Sometimes it's "fall back to a cached or approximate answer and say so." The wrong answer, which we call out directly, is an agent that fabricates a plausible-sounding result when a tool fails silently — this is the single most common cause of user trust breaking, and it's entirely preventable with an explicit failure path.

Day 29-30: The capstone — demo and defense

The last two days are the capstone. Every student deploys a complete Hermes-style agent to a live environment, and it has to survive a scripted chaos exercise in front of the cohort: an instructor deliberately kills a tool integration, injects a malformed input, and runs a burst of concurrent requests, all while the student's dashboards are on screen for everyone to watch.

The rubric doesn't reward students for having zero failures — it rewards them for the failures being visible, logged, and handled the way their guardrail policy said they would be. An agent that returns a clean "I couldn't complete that, here's why" during a chaos test scores higher than one that happens to dodge every failure but has no monitoring to prove it. That's a deliberate choice on our part: we are teaching operational maturity, not luck.

Students then give a short defense — walking the room through their architecture diagram, their deployment pipeline, their guardrail layers, and one incident from their own build-and-break process, explaining what they'd change if they were doing it again with more time. Past cohorts have shipped agents for customer support triage, internal documentation search, expense report validation, and a genuinely well-built travel itinerary planner that handled a live flight-API outage mid-demo without losing its composure.

What Week 4 actually teaches you

By the end of Week 4, the technical surface area students have covered — FastAPI service wrapping, containerized deployment, canary rollouts, structured tracing, dashboards, layered guardrails, retry and degradation logic — is substantial. But the real shift is in mindset. Weeks 1 through 3 teach you to make an agent smart. Week 4 teaches you to make an agent trustworthy, which is a different and, frankly, harder problem. Trustworthy means predictable under failure, observable when something goes wrong, and bounded so that a bad decision by the model can't turn into a bad outcome for a user.

That's the material covered in Week 4 of 30 Days of Hermes Agent — deployment, monitoring, guardrails, and the capstone demo that puts all of it under real pressure. If the first three weeks got you an agent that works, this week is what gets you an agent you'd actually be willing to put your name on in production.