The Hermes Agent Checkpoint System: Learning by Milestones

Why "Day 14" Means Nothing Without a Checkpoint

Most course curricula are organized around time. Day 1, Day 2, Week 3. That structure is easy to build a syllabus around, but it's a terrible way to measure whether someone actually learned anything. You can sit through fourteen days of lectures on agent architecture and still not be able to answer a basic question: does the agent you built actually recover from a failed tool call, or does it just look like it does in the one demo you ran?

This is the problem the checkpoint system in 30 Days of Hermes Agent was built to solve. Instead of treating the thirty days as a straight timeline, we split the course into a series of checkpoints — hard, testable gates that sit between phases of the build. You don't advance past a checkpoint by watching a video. You advance by producing an artifact — a script, a log, a passing test — that proves your agent does the thing the checkpoint claims it does.

We called it the Hermes Agent Checkpoint System, and after running multiple cohorts through it, it's become the backbone of how the course is taught. This article walks through what it is, why milestone-based learning beats calendar-based learning for anything as slippery as "agent behavior," and how the checkpoints are actually structured — with real examples from the course, not hand-wavy descriptions.

The Problem With Linear, Time-Boxed Curricula

Agents are not like typical software features. A CRUD endpoint either returns the right JSON or it doesn't — you can eyeball it in a minute. An agent, on the other hand, can appear to work across five test prompts and then quietly fall apart on the sixth because it mismanaged its own memory, or because a tool call failed silently and the agent hallucinated a plausible-sounding result instead of surfacing the error.

That means a purely linear curriculum — where each day builds on the last and you just keep moving forward — has a specific failure mode: students accumulate invisible debt. You build a planning loop on Day 5 that has a subtle bug in how it handles empty tool results. Nobody catches it because the demo on Day 5 doesn't happen to trigger that path. By Day 20 you're debugging a multi-agent handoff and the actual root cause is still that Day 5 bug, three layers down, and now it's tangled up with a dozen other assumptions you've built on top of it.

We've watched this happen enough times in early cohorts to know it's not a hypothetical. Students would report "my agent is acting weird" in week three, and root-causing it meant unwinding two weeks of code to find a problem that a five-minute check on Day 5 would have caught immediately.

Checkpoints exist to stop debt from accumulating. A checkpoint is a forced stop where you cannot rationalize "it mostly works." Either the artifact exists and passes the check, or it doesn't, and you don't move on.

What a Checkpoint Actually Is

A checkpoint in the Hermes course has four parts, and all four are non-negotiable:

• A trigger condition — the specific point in the build where the checkpoint fires (e.g., "after you've wired your first tool call")
• A required artifact — something concrete you produce: a log file, a test script output, a committed diff
• A pass/fail rubric — an explicit, binary list of what counts as passing (not vibes, not "looks good")
• A rollback instruction — exactly what to do if you fail, so failing isn't a dead end

That last point matters more than people expect. A checkpoint that just says "you failed, go back and try again" is demoralizing and vague. A good checkpoint tells you precisely which of the last N steps to revisit, because the rubric already isolated where the failure lives.

Here's a simplified version of an actual Day 6 checkpoint from the course — the "Tool Call Resilience Checkpoint," which comes right after students wire up their agent's first external tool (usually a weather API or a simple database lookup):

# checkpoint_06_tool_resilience.py
# Run this AFTER wiring your first tool call.
# Pass condition: all four scenarios must exit 0.

import subprocess
import sys

SCENARIOS = [
    ("normal_call", "Tool returns valid data"),
    ("timeout", "Tool call times out after 5s"),
    ("malformed_response", "Tool returns unexpected JSON shape"),
    ("tool_unavailable", "Tool endpoint returns 503"),
]

def run_scenario(name, description):
    print(f"Checking: {description}")
    result = subprocess.run(
        ["python", "agent_runner.py", "--scenario", name],
        capture_output=True, text=True, timeout=15
    )
    # The rubric: the agent must NEVER crash, and must NEVER
    # fabricate a tool result when the tool fails.
    crashed = result.returncode != 0
    fabricated = "posing as real data" in result.stdout.lower()
    hallucinated = "assuming the value is" in result.stdout.lower()

    if crashed or fabricated or hallucinated:
        print(f"  FAIL: {name}")
        return False
    print(f"  PASS: {name}")
    return True

if __name__ == "__main__":
    results = [run_scenario(n, d) for n, d in SCENARIOS]
    if all(results):
        print("\nCheckpoint 06 PASSED. Proceed to Day 7.")
        sys.exit(0)
    else:
        print("\nCheckpoint 06 FAILED. Do not proceed.")
        print("Revisit: your tool-call wrapper's error handling.")
        print("Specifically: does a failed tool call raise a")
        print("structured exception, or does it return a string")
        print("that gets fed straight back into the prompt?")
        sys.exit(1)

Notice what this script does not check: it does not check whether the agent gives a "good" answer. It checks whether the agent behaves correctly under failure conditions that are guaranteed to happen in production and almost never show up in a happy-path demo. That's deliberate. Most agent bugs students ship to production aren't reasoning bugs — they're failure-handling bugs.

Milestone One: The Single-Tool Loop

The first real checkpoint in the course, arriving around Day 4-5, is deliberately unglamorous. Students aren't asked to build a multi-agent system or anything with memory yet. The milestone is: build an agent that can call exactly one tool, handle exactly one failure mode of that tool, and log what it did in a way a human can audit afterward.

The reason this is the first checkpoint and not, say, "build a working agent that answers questions" is that logging and failure handling are the two things students most reliably skip when they're excited to get to the "cool" parts of agent building. If you don't force the habit here, at the ten-tool stage, you'll have ten different ad-hoc error handling styles scattered across the codebase, none of which are debuggable.

The artifact required for this milestone is a transcript log — literally a JSON lines file where every tool call, its input, its output (or its failure), and the agent's next action are recorded. Students submit this log, and the rubric checks for exactly three things: every tool call has a corresponding result entry, every failure is tagged as a failure (not silently swallowed), and the agent's subsequent behavior after a failure is visibly different from its behavior after a success.

Milestone Two: Memory Without Amnesia or Hoarding

The second big checkpoint, around Day 10, is about state. This is where a lot of the actual conceptual difficulty of agent-building lives, and it's also where the checkpoint format earns its keep the most.

Students are asked to build an agent that holds a multi-turn conversation across at least eight turns, referencing something said in turn two by turn eight, without re-sending the entire raw conversation history as context on every single call. In other words: some form of summarization, retrieval, or structured memory has to exist.

The failure modes here are specific and the checkpoint tests for them directly:

• Amnesia: the agent forgets something said three turns ago that it should still know
• Hoarding: the agent just stuffs the entire raw history into every prompt, which "works" in a toy example but blows past context limits and cost budgets the moment a real user has a long session
• Corruption: the agent's summarized memory drifts from what was actually said — it "remembers" something that didn't happen

The checkpoint script runs a scripted eight-turn conversation with a known ground truth (a specific preference stated in turn two, a correction issued in turn five that reverses something said in turn three) and then asks the agent, in turn eight, to recall the current state of that preference. It's graded against the ground truth, not against vibes. Either the agent gets it right or it doesn't.

Here's a stripped-down version of that ground-truth check:

# checkpoint_10_memory_integrity.py

GROUND_TRUTH = {
    "turn_2_preference": "prefers dark mode",
    "turn_5_correction": "actually prefers light mode",
    "expected_final_state": "light mode",
}

def check_memory_integrity(agent_response_turn_8: str) -> bool:
    final = GROUND_TRUTH["expected_final_state"]
    stale = GROUND_TRUTH["turn_2_preference"].split()[-2:]  # ["dark", "mode"]

    mentions_correct_state = final in agent_response_turn_8.lower()
    mentions_stale_state = " ".join(stale) in agent_response_turn_8.lower()

    if mentions_stale_state and not mentions_correct_state:
        print("FAIL: agent is citing stale preference (turn 2, overwritten by turn 5)")
        return False
    if not mentions_correct_state:
        print("FAIL: agent lost the preference entirely (amnesia)")
        return False
    return True

Students who pass this checkpoint on the first try are rare, and that's fine — that's the point. The checkpoint exists precisely because "my agent remembers things" is a claim almost every student makes confidently and almost none of them have actually verified before this gate forces them to.

Milestone Three: The Handoff Checkpoint

By Day 18-ish, students are working with more than one agent role — a planner and an executor, or a researcher and a writer, depending on which track of the course they're following. This is where the Handoff Checkpoint comes in, and it's the one that trips up the most people, because multi-agent handoff bugs are genuinely subtle.

The milestone requires demonstrating a clean handoff: agent A completes its portion of a task, packages up exactly the context agent B needs (not the entire conversation, not nothing), and agent B can pick up the task without needing to ask a clarifying question that agent A already answered.

The rubric measures this by literally counting redundant clarifying questions. If agent B asks the user something agent A already knew and should have passed along, that's a checkpoint failure, full stop. It's an unambiguous, countable signal, which is exactly what you want a rubric to be.

This checkpoint also introduces the idea of a handoff contract — a fixed schema for what gets passed between agents, so the interface is explicit rather than "whatever felt convenient in the moment." Students who skip this and just pass around a blob of freeform text almost always fail the checkpoint on the first attempt, because freeform handoffs are exactly where information silently gets dropped.

Milestone Four: The Production Readiness Gate

The last major checkpoint, near the end of the thirty days, is the widest one. It's not testing a single capability — it's testing whether the agent, as a whole system, is something you'd actually be comfortable putting in front of a real user.

This checkpoint bundles several sub-checks:

1. Cost bound — the agent must complete a defined benchmark task within a token/cost ceiling, so students confront the economics of their design choices instead of ignoring them
2. Timeout behavior — every external call has an enforced timeout, and the agent has a defined fallback for each one
3. Idempotency — rerunning the same task twice doesn't duplicate side effects (this catches a huge class of bugs in agents that call write-actions like sending emails or creating records)
4. Observability — there's a way to reconstruct, after the fact, exactly what the agent did and why, from logs alone, without re-running it

The idempotency check is the one that catches people most often, and it's worth dwelling on because it's a class of bug that's easy to never notice until it's expensive. An agent that retries a failed step by just re-running it, without checking whether the first attempt's side effect already landed, will happily send a duplicate email, create two calendar events, or charge a card twice. The checkpoint script deliberately simulates a network blip mid-task and checks whether the agent's retry logic re-executes the side-effecting step or recognizes it already happened.

# checkpoint_final_idempotency.py

def simulate_retry_after_partial_failure(agent, task):
    # First attempt: side effect fires, then connection drops
    # before the agent receives confirmation.
    result_1 = agent.run(task, inject_failure="post_side_effect_disconnect")

    # Agent believes the call failed and retries.
    result_2 = agent.run(task, resume_from=result_1.state)

    side_effect_count = count_side_effects(task.target_system)
    if side_effect_count > 1:
        print(f"FAIL: side effect fired {side_effect_count} times, expected 1")
        return False
    print("PASS: idempotency held under simulated disconnect")
    return True

What Failing a Checkpoint Actually Looks Like

It's worth being explicit that failing checkpoints is normal, expected, and by design not punished. The course tracks checkpoint attempts, not first-attempt pass rates, and Ira Menon, who built most of the checkpoint rubrics, has said repeatedly in cohort office hours that a checkpoint passed on the third attempt after real debugging teaches more than one passed cleanly on the first try by luck.

The rollback instructions attached to each checkpoint are intentionally specific rather than generic. "Go back and fix your code" is useless advice. "Your tool-call wrapper is catching the exception but not tagging it as a failure before it re-enters the prompt loop — check the except block in your call_tool function" is advice someone can act on in ten minutes. Every checkpoint in the course is written with that second style of feedback, because the entire point of gating progress is to make the feedback loop tight, not to make people feel bad about restarting.

Why Milestones Beat the Calendar

The deeper argument for structuring a course this way is that competence in agent-building isn't a smooth curve — it's a series of step functions. You either understand how to structure a tool-call failure path, or you don't, and once you do, it's mostly permanent. Time spent watching lectures doesn't reliably produce that kind of step-change; producing an artifact and having it tested against a rubric does.

This is also why the checkpoint system doubles as a portfolio. By the time a student finishes 30 Days of Hermes Agent, they don't just have a certificate — they have a stack of checkpoint artifacts: resilience logs, memory-integrity transcripts, handoff contracts, an idempotency test suite. Those are the exact things a hiring manager or a client actually wants to see, far more than a "completed the course" badge. A calendar tells you someone showed up for thirty days. A checkpoint stack tells you what they can actually build.

Bringing Checkpoints Into Your Own Agent Projects

Even outside the course, the checkpoint pattern is worth stealing wholesale for any agent project you're building on your own. The four-part structure — trigger, artifact, rubric, rollback — doesn't require our specific curriculum to be useful. If you're building an agent right now, you can retrofit this today: pick the riskiest assumption in your current build (usually it's "my agent handles failures gracefully" or "my agent's memory is accurate"), write a five-line script that tests it against a scripted failure case, and refuse to let yourself add the next feature until that script passes.

That discipline is the entire reason the checkpoint system exists in 30 Days of Hermes Agent, and it's the reason students come out the other side with agents that hold up outside the demo, not just inside it. If you've been putting off learning agent development because the space feels shapeless and hard to measure progress in, the checkpoint structure is exactly the fix — it turns "I think I understand agents" into a stack of tested, dated proof that you do.