Agent Testing Strategies: Unit, Integration and End-to-End

Why testing an agent breaks your old testing brain

If you have shipped backend services before, you already have instincts for testing. You mock the database, you assert on return values, you check status codes. Those instincts do not fully transfer to agents, and pretending they do is how teams end up "testing" an agent by chatting with it in a notebook and eyeballing the output.

An agent is not a pure function. Give it the same prompt twice and you can get two different tool call sequences, two different phrasings, sometimes two different final answers. The model is non-deterministic by design, the tools it calls have their own failure modes, and the "correctness" of a response is often fuzzy rather than exact-match. A customer support agent that says "I've refunded your order" in one run and "Your refund has been processed" in another is not broken — but an agent that calls refund() twice because it got confused about whether the first call succeeded absolutely is.

This is the core problem this article works through: how do you build a testing strategy that catches the failures that matter (wrong tool calls, broken loops, unsafe actions, regressions after a prompt change) without drowning in flaky assertions on natural language output that was never going to be identical twice.

The answer, as with traditional software, is layers. You still want a testing pyramid — a lot of fast, cheap, deterministic unit tests at the bottom, a smaller number of integration tests that exercise real tool wiring, and a thin top layer of end-to-end conversation tests that prove the whole system behaves under realistic conditions. What changes is *what* each layer tests and *how* you assert on results. Let's go layer by layer.

Unit testing: isolate the parts that are actually deterministic

The trick to unit testing an agent is to stop trying to unit test "the agent" as a whole and instead unit test the pieces that are not the LLM. Your tools, your parsers, your state machine, your routing logic, your guardrails — all of that is regular code, and regular code deserves regular unit tests.

Consider a tool definition for a support agent that looks up an order:

# tools/orders.py
from dataclasses import dataclass

class OrderNotFoundError(Exception):
    pass

@dataclass
class Order:
    order_id: str
    status: str
    total_cents: int
    refundable: bool

def get_order(order_id: str, db) -> Order:
    row = db.query_one(
        "SELECT id, status, total_cents FROM orders WHERE id = %s",
        (order_id,),
    )
    if row is None:
        raise OrderNotFoundError(f"No order with id {order_id}")

    refundable = row["status"] in ("delivered", "shipped") \
        and row["total_cents"] > 0

    return Order(
        order_id=row["id"],
        status=row["status"],
        total_cents=row["total_cents"],
        refundable=refundable,
    )

This function has nothing to do with the LLM. It is deterministic, it has clear inputs and outputs, and it deserves the exact same unit test treatment you would give any other data-access function:

# tests/unit/test_orders.py
import pytest
from tools.orders import get_order, OrderNotFoundError

class FakeDB:
    def __init__(self, rows):
        self.rows = rows

    def query_one(self, sql, params):
        order_id = params[0]
        return self.rows.get(order_id)

def test_get_order_marks_delivered_order_as_refundable():
    db = FakeDB({"o1": {"id": "o1", "status": "delivered", "total_cents": 4200}})
    order = get_order("o1", db)
    assert order.refundable is True
    assert order.total_cents == 4200

def test_get_order_raises_when_missing():
    db = FakeDB({})
    with pytest.raises(OrderNotFoundError):
        get_order("does-not-exist", db)

def test_cancelled_order_is_not_refundable():
    db = FakeDB({"o2": {"id": "o2", "status": "cancelled", "total_cents": 1500}})
    order = get_order("o2", db)
    assert order.refundable is False

No LLM involved, no flakiness, runs in milliseconds. This is where the bulk of your test count should live. Every tool your agent can call — search, refund, send-email, create-ticket, write-to-database — should have this kind of coverage independent of the agent that calls it.

The second thing worth unit testing is your structured-output parsing and validation logic. If your agent asks the model to emit JSON matching a schema, and you have a step that validates and coerces that JSON before acting on it, that validation step is pure code and should be tested with a battery of malformed inputs: missing fields, wrong types, extra fields, empty strings where you expect an enum. This is exactly the kind of code where LLMs occasionally go off-script, and you want your parser to fail loudly and safely rather than silently accept garbage.

The third candidate is your routing and control-flow logic — the code that decides which tool to call based on the model's output, or which sub-agent to hand off to. If you have a router function that inspects an intent classification and dispatches accordingly, test every branch:

def route_intent(intent: str):
    if intent == "refund_request":
        return "refund_agent"
    elif intent == "shipping_question":
        return "shipping_agent"
    elif intent == "general":
        return "chat_agent"
    else:
        return "fallback_agent"

A one-line change to this function can silently misroute an entire category of user requests, and a unit test suite that pins down every branch will catch that in under a second, long before it reaches a customer.

Mocking the model without lying to yourself

Once you unit test tools and routing, the next question is how to test the code that actually orchestrates calls to the LLM — the agent loop itself. You do not want to hit a real model API in a unit test: it is slow, it costs money, and it is non-deterministic, which defeats the purpose of a unit test.

The answer is to mock the LLM client at the boundary, and assert on what your orchestration code *does* with a given model response, not on what the model would say in the wild.

# tests/unit/test_agent_loop.py
from unittest.mock import MagicMock
from agent.loop import run_agent_step

def test_agent_calls_refund_tool_when_model_requests_it():
    fake_llm = MagicMock()
    fake_llm.complete.return_value = {
        "tool_calls": [
            {"name": "refund_order", "arguments": {"order_id": "o1", "amount_cents": 4200}}
        ],
        "content": None,
    }

    fake_tools = {"refund_order": MagicMock(return_value={"status": "refunded"})}

    result = run_agent_step(fake_llm, fake_tools, history=[])

    fake_tools["refund_order"].assert_called_once_with(order_id="o1", amount_cents=4200)
    assert result["tool_results"][0]["status"] == "refunded"

def test_agent_stops_loop_when_model_returns_final_answer():
    fake_llm = MagicMock()
    fake_llm.complete.return_value = {"tool_calls": [], "content": "Your refund is done."}

    result = run_agent_step(fake_llm, tools={}, history=[])

    assert result["done"] is True
    assert "refund" in result["content"].lower()

Notice what is being tested here: not "does the model give a good answer," but "given this specific model response, does the loop correctly extract the tool call, invoke the right function with the right arguments, and correctly decide whether to keep looping or stop." That is completely deterministic logic sitting between you and the model, and it is exactly the kind of thing that breaks silently when someone refactors the loop or upgrades an SDK version.

This same technique extends to testing failure handling — what happens when a tool throws an exception, when the model requests a tool that does not exist, when the model returns malformed JSON in a tool-call argument, or when you hit a maximum iteration count. Each of these is a scenario you can construct by hand with a mock, and each one represents a real production failure mode you want covered before it happens to a real user.

Integration testing: prove the wiring actually works

Unit tests prove your tools work in isolation and your loop handles mocked model responses correctly. Integration tests prove that the real pieces connect — the actual tool functions, the actual database (or a realistic test database), and, usually, the actual model, at least for a curated set of scenarios.

The key design decision at this layer is: how much do you let the real LLM into the test, and how do you keep it from making your CI pipeline flaky and expensive?

A common and effective pattern is to run integration tests against a real model but with a fixed, small set of scripted scenarios, and to assert on behavior classes rather than exact text. You are not checking that the agent says "I've processed your refund of $42.00." You are checking that, given this input, the agent ends up calling the refund_order tool with the correct order ID, and that it does not call it more than once.

# tests/integration/test_refund_flow.py
import pytest
from agent.runtime import AgentRuntime
from tools.orders import Order

@pytest.fixture
def runtime(test_db):
    test_db.insert_order(Order(order_id="o1", status="delivered",
                                total_cents=4200, refundable=True))
    return AgentRuntime(db=test_db, llm_model="claude-sonnet-4-5")

def test_agent_refunds_eligible_order_exactly_once(runtime):
    transcript = runtime.run_conversation([
        {"role": "user", "content": "I want a refund for order o1, it arrived broken."}
    ])

    refund_calls = [c for c in transcript.tool_calls if c.name == "refund_order"]
    assert len(refund_calls) == 1
    assert refund_calls[0].arguments["order_id"] == "o1"
    assert transcript.final_status == "completed"

def test_agent_refuses_refund_for_ineligible_order(runtime, test_db):
    test_db.insert_order(Order(order_id="o2", status="cancelled",
                                total_cents=1500, refundable=False))

    transcript = runtime.run_conversation([
        {"role": "user", "content": "Refund order o2 please."}
    ])

    refund_calls = [c for c in transcript.tool_calls if c.name == "refund_order"]
    assert len(refund_calls) == 0

A few things matter here. First, this test hits a real database (even if it is a disposable test container or an in-memory substitute with the same schema) so that you actually exercise SQL, constraints, and transactions rather than a hand-wavy mock. Second, it may hit a real model, which means these tests are slower and cost tokens — so you keep this suite small and targeted at the scenarios that matter most: the "happy path," the most common failure path, and one or two edge cases that have bitten you before.

Third — and this is the part teams skip — integration tests are where you catch tool contract drift. If someone changes the signature of refund_order from (order_id, amount_cents) to (order_id, amount_dollars), your unit tests for the tool itself might still pass because they were updated in the same PR, but if the agent's tool schema (the JSON schema you hand to the model describing the tool) was not updated to match, the model will keep calling it with the old argument shape and it will start silently failing or refunding the wrong amount. An integration test that runs the real tool schema against the real tool function catches this immediately; a unit test with a mock tool will not.

A second category of integration test worth calling out is multi-tool sequencing. Many real agent tasks require calling tool A, using its output to decide whether to call tool B, and so on. Test that the sequence actually happens in the right order and that state is correctly threaded through:

def test_agent_checks_inventory_before_confirming_order(runtime):
    transcript = runtime.run_conversation([
        {"role": "user", "content": "I'd like to order 3 units of SKU-882."}
    ])

    tool_names = [c.name for c in transcript.tool_calls]
    assert "check_inventory" in tool_names
    assert tool_names.index("check_inventory") < tool_names.index("create_order")

This is a small but powerful assertion: it does not care what the agent said in natural language, only that it checked inventory before committing to an order. That is the kind of test that survives a prompt rewrite next month, because it is anchored to behavior, not to phrasing.

End-to-end testing: does the whole thing survive contact with reality

End-to-end tests run the full stack — real model, real tools (or close-to-production sandboxes), real memory/session storage, and ideally the same entry point your users hit, whether that's an API endpoint, a chat widget, or a Slack bot. This is the layer that answers "does this actually work" rather than "does this component work."

Because full conversations with a live model are slow, costly, and non-deterministic in wording, end-to-end suites should be small, curated, and focused on scenarios that represent real business risk: the core happy path for your main use case, the most damaging failure mode you can imagine (an agent that takes an irreversible action it should not have), and any bug that has actually happened in production and that you never want to see again.

A useful pattern here is the scripted multi-turn conversation test with an LLM-as-judge for the fuzzy parts, combined with hard assertions for the parts that must be exact:

# tests/e2e/test_support_conversation.py
from agent.runtime import AgentRuntime
from eval.judge import judge_response

def test_multi_turn_refund_conversation():
    runtime = AgentRuntime(env="e2e-sandbox", llm_model="claude-sonnet-4-5")

    turn1 = runtime.send("Hi, my order o1 arrived damaged.")
    turn2 = runtime.send("Yes please, I'd like a refund.")

    # Hard assertions on things that must never vary
    refund_calls = [c for c in runtime.transcript.tool_calls if c.name == "refund_order"]
    assert len(refund_calls) == 1
    assert runtime.transcript.tool_calls[-1].result["status"] == "refunded"

    # Fuzzy assertion on tone/content, delegated to a judge model
    verdict = judge_response(
        conversation=runtime.transcript,
        rubric=(
            "The assistant should confirm the refund was processed, "
            "apologize for the damaged item, and avoid promising a "
            "specific delivery date for any replacement."
        ),
    )
    assert verdict.passed, verdict.reasoning

The judge_response helper is typically another LLM call, given the transcript and a rubric, asked to return a pass/fail with reasoning. This is standard practice for evaluating open-ended text output — you are not trying to string-match, you are asking a model to check whether the response satisfies a small number of concrete criteria. Keep rubrics narrow and concrete ("must not promise a delivery date," "must mention the refund amount") rather than vague ("must sound helpful"), because vague rubrics produce flaky judge verdicts just like vague assertions produce flaky test failures.

End-to-end tests are also where you should test the things that only show up under real-world conditions: session persistence across turns, concurrent conversations from the same user, tool timeouts and retries, and rate-limit handling. These are exactly the failure modes that unit and integration tests, by construction, cannot see, because they involve the full runtime under realistic timing and load.

The special case of agent loops: testing for runaway behavior

Agents that can call tools repeatedly introduce a failure mode that traditional software mostly does not have: the infinite or near-infinite loop, where the agent keeps calling a tool, gets a result it does not like, calls it again, and never converges on an answer. This deserves its own explicit test category, at both the unit and integration level.

At the unit level, test your loop's termination conditions directly:

def test_agent_loop_stops_after_max_iterations():
    fake_llm = MagicMock()
    # Model that always wants to call another tool, never finishes
    fake_llm.complete.return_value = {
        "tool_calls": [{"name": "search", "arguments": {"query": "x"}}],
        "content": None,
    }
    fake_tools = {"search": MagicMock(return_value={"results": []})}

    result = run_agent_loop(fake_llm, fake_tools, history=[], max_iterations=5)

    assert result["stopped_reason"] == "max_iterations_reached"
    assert fake_tools["search"].call_count == 5

At the integration level, construct a scenario that is genuinely hard to resolve — a search query that returns no useful results no matter how it is rephrased — and assert that the agent gracefully gives up and tells the user it could not find an answer, rather than burning through the iteration budget silently and returning an empty or truncated response.

This category of test matters disproportionately because runaway loops are also a cost and safety problem, not just a correctness one: an agent stuck in a loop calling a paid API or, worse, an action tool like "send email" or "create ticket," can cause real damage in the time it takes someone to notice a spike in a dashboard.

Building a regression harness so prompt changes stop being scary

The single biggest practical improvement most teams can make is turning their scattered manual "let me just try a few prompts" habit into a versioned regression suite that runs automatically whenever the system prompt, model version, or tool set changes. Think of this as a golden dataset of representative conversations, each with expected tool-call sequences and/or judge rubrics, that you replay on every change.

# eval/regression_suite.py
import json
from agent.runtime import AgentRuntime
from eval.judge import judge_response

def load_cases(path="eval/cases.jsonl"):
    with open(path) as f:
        return [json.loads(line) for line in f]

def run_regression_suite():
    runtime = AgentRuntime(env="eval-sandbox")
    failures = []

    for case in load_cases():
        transcript = runtime.run_conversation(case["turns"])
        expected_tools = case.get("expected_tool_calls", [])
        actual_tools = [c.name for c in transcript.tool_calls]

        if expected_tools and actual_tools != expected_tools:
            failures.append((case["id"], "tool_sequence_mismatch", actual_tools))
            continue

        if "rubric" in case:
            verdict = judge_response(transcript, rubric=case["rubric"])
            if not verdict.passed:
                failures.append((case["id"], "rubric_failed", verdict.reasoning))

    return failures

if __name__ == "__main__":
    failures = run_regression_suite()
    if failures:
        for case_id, reason, detail in failures:
            print(f"FAIL {case_id}: {reason} -> {detail}")
        raise SystemExit(1)
    print("All regression cases passed.")

This is the piece that turns agent development from "ship and pray" into something closer to normal software engineering discipline. Every time someone tweaks the system prompt to fix one bug, this suite tells you within minutes whether that tweak broke five other things — which, with LLM agents, it frequently does, because prompts are coupled in non-obvious ways. Start this file on day one with even five cases. It is far easier to grow a regression suite incrementally than to retrofit one after your agent has been in production for six months and nobody remembers what "correct" behavior looked like for that one edge case from March.

Putting it together: a pyramid that actually holds weight

Here is roughly how the layers should balance out in practice, ordered by how much of your total test count should live there:

• Unit tests (the bulk of your suite): every tool function, every parser, every router, and the agent loop's control flow against mocked model responses. Fast, free, deterministic, run on every commit.
• Integration tests (a meaningful minority): real tools plus real or realistic databases, a small set of scripted scenarios against a real model, asserting on tool-call sequences and side effects rather than exact wording. Run on every PR.
• End-to-end tests (a thin top layer): full-stack multi-turn conversations through the real entry point, mixing hard assertions with LLM-as-judge rubrics for open-ended output. Run before releases and on a schedule, not necessarily on every commit.
• Regression harness (cuts across all three): a growing, versioned set of golden cases that gets replayed whenever the prompt, model, or tools change, catching the coupling problems that make agent development feel unpredictable.

None of this requires exotic tooling. Everything shown above is plain pytest, a mock library, a database fixture, and a thin custom runtime wrapper — the same muscles you already have from testing normal backend systems, redirected at the specific failure modes agents introduce: non-determinism in wording, runaway tool loops, contract drift between tool schemas and tool implementations, and the fuzzy correctness of natural language output.

If you want to go deeper on building and testing production agents — including the orchestration patterns, tool design, and evaluation pipelines that make this kind of testing strategy sustainable at scale — that is exactly what we cover, end to end, inside 30 Days of Hermes Agent, our flagship agent-engineering course. It walks through building a real multi-tool agent from scratch and hardening it with the same unit, integration, and end-to-end testing layers described here, so you are not just reading about the strategy, you are shipping it.