Agent Retry Logic: Handling Transient Tool Failures Gracefully

Why your agent crashes on a hiccup it should have shrugged off

Somewhere between "the demo worked" and "customers depend on this," every agent builder hits the same wall. A tool call times out. An API returns a 503. A rate limiter kicks in for two seconds. And the agent, which handled a genuinely hard reasoning task a moment ago, falls over because of a problem that had nothing to do with intelligence at all.

This is the part of agent engineering that doesn't show up in demo videos. Nobody posts a screen recording of their agent gracefully recovering from a dropped connection. But in production, this is most of the job. Language models are the least reliable part of an agent's failure surface — network blips, provider-side rate limits, database connection pool exhaustion, and third-party API hiccups happen far more often than the model saying something wrong. If your agent treats every failure as fatal, you've built something that looks impressive in a sandbox and falls apart the moment real traffic hits it.

Retry logic sounds like a solved problem — "just wrap it in a try/except and loop" — until you actually build it. Retry too aggressively and you turn a two-second blip into a self-inflicted denial-of-service against your own downstream API. Retry too passively and users see failures that would have resolved themselves half a second later. Retry the wrong kind of error and you silently repeat a mistake five times, burning latency and tokens on something that was never going to succeed. This article walks through how to build retry logic that actually holds up: what to retry, what never to retry, how exponential backoff and jitter work under the hood, and how to wire circuit breakers and idempotency into an agent's tool-calling loop so failures degrade gracefully instead of cascading.

Transient vs. permanent: the classification problem nobody skips correctly

Before writing a single line of backoff code, you need a classifier: is this failure worth retrying at all? Get this wrong and retry logic becomes actively harmful.

Transient failures are conditions that are likely to resolve on their own within seconds:

• Network timeouts and connection resets
• HTTP 429 (rate limited) and 503 (service unavailable)
• Database connection pool exhaustion
• DNS resolution hiccups
• Provider-side "overloaded, try again" responses from LLM APIs

Permanent failures are conditions that will not change no matter how many times you retry:

• HTTP 400 (bad request) — your payload is malformed
• HTTP 401/403 — you're not authenticated or authorized
• HTTP 404 — the resource doesn't exist
• Schema validation errors in a tool call's arguments
• Business logic rejections ("insufficient balance", "invalid coupon code")

The mistake I see most often in agent codebases is a blanket except Exception: retry() that doesn't distinguish between these. That's how you end up retrying a malformed JSON payload eight times, each attempt failing identically, while the actual root cause — a bug in the agent's argument construction — goes unnoticed until someone reads the logs three days later.

A good rule of thumb: retry on the transport layer, not the application layer. If the request never really landed, or if the server is telling you it's temporarily overwhelmed, retry. If the server understood your request and rejected it on its merits, stop and surface the error instead.

class RetryableError(Exception):
    """Raised for conditions that are likely to resolve if retried."""
    pass

class PermanentError(Exception):
    """Raised for conditions that will not change on retry."""
    pass

def classify_http_error(status_code: int, exc: Exception | None = None) -> Exception:
    if exc and isinstance(exc, (TimeoutError, ConnectionError)):
        return RetryableError(f"transport failure: {exc}")

    if status_code in (429, 502, 503, 504):
        return RetryableError(f"transient server error: {status_code}")

    if status_code in (400, 401, 403, 404, 409, 422):
        return PermanentError(f"non-retryable client error: {status_code}")

    if 500 <= status_code < 600:
        # Unclassified 5xx — treat as transient but log distinctly,
        # since a persistent 500 usually indicates a real bug.
        return RetryableError(f"unclassified server error: {status_code}")

    return PermanentError(f"unexpected status: {status_code}")

This classifier is the single most important piece of the whole system. Everything downstream — backoff schedules, retry budgets, circuit breakers — assumes you've already separated "try again" from "stop and report."

Exponential backoff: the mechanism, not just the buzzword

"Exponential backoff" gets thrown around as if writing time.sleep(2 ** attempt) is the whole story. It isn't. A correct implementation needs a base delay, a growth factor, a maximum cap, and — critically — jitter.

Here's why each piece matters:

• Base delay sets how long you wait after the first failure. Too short, and you're hammering a server that just told you it's overloaded.
• Growth factor (usually 2x) means each successive failure waits longer, giving the downstream system real room to recover.
• Max delay cap prevents the wait from growing unboundedly — nobody wants a retry scheduled for six minutes from now when the whole request should timeout in thirty seconds.
• Jitter — randomizing the delay — exists to prevent the "thundering herd" problem. If a thousand agent instances all fail at the same moment because a shared dependency went down, and they all back off with the exact same deterministic schedule, they'll all retry at the exact same moment too, hitting the recovering service with a synchronized wave that can knock it back down. Jitter staggers the retries so they spread out.

Here's a full implementation with decorrelated jitter, which AWS's architecture team popularized specifically because it outperforms naive "full jitter" in reducing contention:

import random
import time
from functools import wraps

def retry_with_backoff(
    max_attempts: int = 5,
    base_delay: float = 0.5,
    max_delay: float = 30.0,
    retryable_exceptions: tuple = (RetryableError,),
):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            delay = base_delay
            last_exception = None

            for attempt in range(1, max_attempts + 1):
                try:
                    return func(*args, **kwargs)
                except retryable_exceptions as exc:
                    last_exception = exc
                    if attempt == max_attempts:
                        break

                    # Decorrelated jitter: next delay is a random value
                    # between base_delay and 3x the previous delay,
                    # capped at max_delay. This spreads retries out
                    # more evenly than simple exponential + random jitter.
                    delay = min(max_delay, random.uniform(base_delay, delay * 3))

                    print(
                        f"[retry] attempt {attempt}/{max_attempts} failed "
                        f"({exc}); sleeping {delay:.2f}s"
                    )
                    time.sleep(delay)
                except PermanentError:
                    # Never retry these — fail fast and surface immediately.
                    raise

            raise last_exception

        return wrapper

    return decorator


@retry_with_backoff(max_attempts=4, base_delay=0.5, max_delay=10.0)
def call_search_tool(query: str) -> dict:
    response = http_client.post("/tools/search", json={"query": query})
    if response.status_code >= 400:
        raise classify_http_error(response.status_code)
    return response.json()

Notice the PermanentError branch re-raises immediately without consuming a retry attempt. That's the classification logic from the previous section doing its job — it keeps the retry loop honest instead of blindly retrying everything that comes through.

Retry budgets: because unlimited retries are their own outage

Exponential backoff controls the *pace* of retries. It says nothing about whether you should keep retrying at all. This is where a lot of agent systems go wrong — they set max_attempts=5 per tool call and think they're done. But an agent that calls five tools in a chain, each with five retries, each waiting up to ten seconds, can turn a single user request into a two-minute ordeal that eventually fails anyway.

The fix is a retry budget: a ceiling on total retry time or total retry count across an entire agent run, not just per call. Think of it as a token bucket that every tool invocation draws from.

import time


class RetryBudget:
    """
    Tracks a shared retry budget across an entire agent session.
    Once the budget is exhausted, no further retries are allowed
    even if individual tool calls would otherwise qualify.
    """

    def __init__(self, max_retries: int = 12, max_total_seconds: float = 45.0):
        self.max_retries = max_retries
        self.max_total_seconds = max_total_seconds
        self.retries_used = 0
        self.seconds_spent = 0.0
        self.start_time = time.monotonic()

    def can_retry(self) -> bool:
        elapsed = time.monotonic() - self.start_time
        return (
            self.retries_used < self.max_retries
            and elapsed < self.max_total_seconds
        )

    def record_retry(self, delay: float) -> None:
        self.retries_used += 1
        self.seconds_spent += delay

    def remaining(self) -> tuple[int, float]:
        elapsed = time.monotonic() - self.start_time
        return (
            self.max_retries - self.retries_used,
            self.max_total_seconds - elapsed,
        )

Wire this into the agent's tool-execution loop, and every tool call checks the shared budget before attempting another retry, not just its own local counter. This is what stops a single flaky dependency from silently eating your entire request timeout budget. It also gives you a clean, honest failure mode: when the budget runs out, the agent should stop, report what happened, and — where possible — hand the user a partial result rather than a generic error.

A retry budget also does something subtle but important for cost control. LLM agent loops often re-invoke the model after a tool failure to decide "what do I do now?" If retries are unbounded, a persistent outage in one tool can trigger dozens of unnecessary model calls, each one burning tokens to essentially say "try again." Bounding retries at the session level bounds that cost too.

Idempotency: the precondition retries assume and rarely check

Here's the assumption baked into every retry loop: retrying is safe because the failed attempt didn't actually do anything. That assumption is true for a GET request. It is often false for anything that writes.

If your agent calls a "charge the customer" tool, gets a timeout, and retries, you have to know whether the first attempt's request actually reached the payment processor before the timeout fired. If it did, retrying blindly could double-charge the customer. This is not a hypothetical edge case — it's one of the most common real-world bugs in agent systems that touch payments, bookings, or any external side effect.

The fix is idempotency keys: a unique identifier attached to the operation so that retries of the same logical request are deduplicated on the server side, regardless of how many times the client sends them.

import uuid


def make_idempotent_request(tool_name: str, payload: dict, idempotency_key: str | None = None):
    """
    Wraps a side-effecting tool call with an idempotency key so that
    retries are safe even if a prior attempt partially succeeded.
    """
    key = idempotency_key or f"{tool_name}-{uuid.uuid4()}"

    @retry_with_backoff(max_attempts=4, base_delay=0.5, max_delay=8.0)
    def _send():
        response = http_client.post(
            f"/tools/{tool_name}",
            json=payload,
            headers={"Idempotency-Key": key},
        )
        if response.status_code >= 400:
            raise classify_http_error(response.status_code)
        return response.json()

    return _send()

The key insight: generate the idempotency key once, before the first attempt, and reuse the *same* key across every retry of that logical operation. If you generate a new key on each retry, you've defeated the entire purpose — the downstream service will see each retry as a brand-new request and happily execute it again.

Not every tool your agent calls will support idempotency keys natively. Stripe, for example, has first-class support for this. Plenty of internal or third-party APIs don't. When a tool lacks native support, the agent-side mitigation is to make the operation itself idempotent where possible ("set status to X" instead of "increment counter by 1") or to check-before-act ("does an order with this reference number already exist?") before retrying a write.

Circuit breakers: stopping the bleeding when a tool is actually down

Retry budgets protect a single agent run. But if a downstream tool is genuinely down — not flaky, actually down — every concurrent agent session hammering it with retries makes the outage worse and delays its recovery. This is where the circuit breaker pattern earns its keep.

A circuit breaker sits in front of a tool and tracks its recent failure rate. When failures cross a threshold, the circuit "opens" and short-circuits further calls immediately — no network round-trip, no wasted retry cycles — for a cooldown period. After the cooldown, it lets a small number of "trial" requests through in a half-open state to check if the dependency has recovered.

import time
from enum import Enum


class CircuitState(Enum):
    CLOSED = "closed"       # normal operation
    OPEN = "open"           # failing fast, no calls allowed through
    HALF_OPEN = "half_open" # testing recovery with limited calls


class CircuitBreaker:
    def __init__(
        self,
        failure_threshold: int = 5,
        recovery_timeout: float = 20.0,
        half_open_trial_calls: int = 2,
    ):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.half_open_trial_calls = half_open_trial_calls

        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.opened_at = None
        self.half_open_successes = 0

    def _maybe_transition_to_half_open(self):
        if self.state == CircuitState.OPEN:
            if time.monotonic() - self.opened_at >= self.recovery_timeout:
                self.state = CircuitState.HALF_OPEN
                self.half_open_successes = 0

    def allow_request(self) -> bool:
        self._maybe_transition_to_half_open()
        return self.state != CircuitState.OPEN

    def record_success(self):
        if self.state == CircuitState.HALF_OPEN:
            self.half_open_successes += 1
            if self.half_open_successes >= self.half_open_trial_calls:
                self.state = CircuitState.CLOSED
                self.failure_count = 0
        else:
            self.failure_count = 0

    def record_failure(self):
        self.failure_count += 1
        if self.state == CircuitState.HALF_OPEN:
            # Failed during trial period — reopen immediately.
            self.state = CircuitState.OPEN
            self.opened_at = time.monotonic()
        elif self.failure_count >= self.failure_threshold:
            self.state = CircuitState.OPEN
            self.opened_at = time.monotonic()


def call_tool_with_circuit_breaker(breaker: CircuitBreaker, func, *args, **kwargs):
    if not breaker.allow_request():
        raise RetryableError("circuit open — tool is currently unavailable")

    try:
        result = func(*args, **kwargs)
        breaker.record_success()
        return result
    except RetryableError:
        breaker.record_failure()
        raise

The behavioral difference from plain retries is important: when the circuit is open, the agent doesn't even attempt the network call. It fails immediately with a clear "this tool is currently unavailable" signal, which the agent's planning layer can act on — falling back to a different tool, queuing the action for later, or telling the user directly instead of spending five seconds discovering what a shared circuit breaker already knows.

In multi-tenant agent platforms, circuit breaker state should generally be shared across sessions per tool (not per-request), since the whole point is protecting a downstream dependency from an aggregate load spike, not just managing one conversation's politeness.

Wiring it into the agent's tool-calling loop

All of this matters only if it's actually integrated into how the agent decides what to do after a tool call fails. A common mistake is bolting retry logic onto the HTTP client layer and stopping there — which handles transient network issues but leaves the agent's reasoning loop blind to what happened. The model doesn't know a retry occurred, doesn't know a circuit is open, and will often just try the same tool call again on its own, unaware that it's about to walk into the same wall.

A cleaner architecture surfaces retry outcomes back into the agent's context so the planning step can react intelligently:

def execute_tool_call(agent_context, tool_name: str, arguments: dict, budget: RetryBudget, breaker: CircuitBreaker):
    if not breaker.allow_request():
        agent_context.record_tool_result(
            tool_name,
            status="unavailable",
            message=f"{tool_name} is temporarily unavailable (circuit open). "
                    f"Consider an alternate approach or informing the user.",
        )
        return None

    attempt = 0
    delay = 0.5

    while True:
        try:
            result = dispatch_tool(tool_name, arguments)
            breaker.record_success()
            agent_context.record_tool_result(tool_name, status="success", data=result)
            return result

        except PermanentError as exc:
            breaker.record_success()  # not the tool's fault — don't penalize it
            agent_context.record_tool_result(
                tool_name, status="failed", message=str(exc), retryable=False
            )
            return None

        except RetryableError as exc:
            breaker.record_failure()
            attempt += 1

            if not budget.can_retry():
                agent_context.record_tool_result(
                    tool_name,
                    status="failed",
                    message=f"Retry budget exhausted after {attempt} attempts: {exc}",
                    retryable=False,
                )
                return None

            delay = min(10.0, random.uniform(0.5, delay * 3))
            budget.record_retry(delay)
            time.sleep(delay)

The key design choice here: every outcome — success, permanent failure, exhausted retries, or an open circuit — gets written back into agent_context in a form the model can reason about on its next turn. That's what turns retry logic from an invisible infrastructure detail into something the agent's decision-making actually accounts for. An agent that knows "the payment tool failed permanently, don't retry it" behaves very differently — and much more usefully — than one that just sees a stack trace and tries the exact same call again.

Observability: retries you can't see are retries you can't trust

None of the above is complete without logging and metrics, because retry logic that fails silently is worse than no retry logic at all — it hides real outages behind a veneer of "eventually it worked." At minimum, instrument:

• Retry counts per tool, per session, and in aggregate
• Time spent in backoff, separated from time spent doing actual work
• Circuit breaker state transitions, with timestamps
• The final outcome of every retried call (succeeded after N attempts / exhausted budget / permanent failure)

If you're using structured logging, tag every retry attempt with a correlation ID tied to the original tool call so you can reconstruct the full sequence later:

import logging

logger = logging.getLogger("agent.retry")

def log_retry_attempt(tool_name, attempt, max_attempts, delay, error, correlation_id):
    logger.warning(
        "tool_retry",
        extra={
            "tool_name": tool_name,
            "attempt": attempt,
            "max_attempts": max_attempts,
            "delay_seconds": round(delay, 2),
            "error": str(error),
            "correlation_id": correlation_id,
        },
    )

Without this, the first sign of trouble is usually a user complaint about slowness, and by the time you've grepped through logs to figure out why, the underlying dependency issue may already be gone — leaving you no way to confirm what actually happened. Good retry observability turns "the agent felt slow yesterday" into "tool X had an 8% retry rate between 2:14 and 2:19 PM, all recovered within budget."

Building this muscle for real

Retry logic looks like a small implementation detail until you're on call for an agent that's silently retrying a permanent 400 error twelve times a minute, or double-charging a customer because two retries both landed. The patterns here — classifying errors correctly, backing off with jitter instead of a fixed delay, capping retries with a shared budget, protecting writes with idempotency keys, and using circuit breakers to stop hammering a dead dependency — are not exotic. They're the same resilience patterns distributed systems engineers have used for two decades, applied to the newer problem of agents that call tools autonomously and can't always tell the difference between "try again" and "stop."

The hard part isn't writing the backoff function. It's deciding, tool by tool, what's safe to retry, what needs an idempotency key, and what should fail loud and fast instead of quietly consuming your retry budget. That judgment only comes from building agents that actually run against real, unreliable infrastructure — not toy demos where every API call succeeds on the first try.

If you want to build this judgment hands-on rather than just read about it, 30 Days of Hermes Agent walks through building a production-grade agent from scratch — including the retry, backoff, and failure-handling architecture covered in this article — so you come out the other side with code you've actually debugged under real failure conditions, not just a checklist you've read once.