Retry and Backoff Patterns for AI Agents

Agent retry logic is the difference between an agent that quietly recovers from a rate limit and one that crashes a multi-step workflow because a single tool call timed out. Unlike a simple API client, an AI agent retries inside a loop that already has state: partial plans, tool outputs, conversation history, and sometimes side effects that already happened. Get agent retry wrong and you either burn tokens hammering a dead endpoint or you silently double-charge a customer because a "failed" payment call actually succeeded. This article covers the patterns that make agent retry safe: backoff with jitter, error classification, idempotency keys, retry budgets, circuit breakers, and how to wire all of it into an agent loop without turning your code into a tangle of try/except blocks.

Why agent retry is harder than API retry

A typical HTTP client retry wraps one request. An agent retry has to account for a few things a plain API client doesn't:

• Multi-step state. An agent might be three tool calls into a plan when call two fails. Retrying call two in isolation is fine. Retrying the whole plan from scratch wastes tokens and can re-trigger side effects from call one.
• Non-idempotent tools. Sending an email, creating a database row, or charging a card is not safe to retry blindly. A network timeout doesn't tell you whether the action ran on the server before the connection dropped.
• Mixed failure sources. A single agent turn can fail because the LLM provider rate-limited you, a downstream tool API timed out, the model returned malformed JSON for a tool call, or your own code threw a bug. Each of these needs a different retry response.
• Cost per retry. Every retried LLM call re-sends context and burns tokens. A naive retry loop on a large context window is not free the way a retried GET request is.

Because of this, agent retry design starts with error classification, not with a backoff formula.

Classify errors before you retry anything

The first rule of agent retry: never retry blind. Split every failure into retryable and non-retryable, and be conservative about which bucket a new error type goes into.

Retryable (transient, likely to succeed on a retry):

• HTTP 429 (rate limited)
• HTTP 500, 502, 503, 504 from the LLM provider or a tool API
• Connection reset, DNS failure, read timeout
• Provider-specific "overloaded" errors

Not retryable (retrying won't help, and might hurt):

• HTTP 400 (malformed request, bad schema)
• HTTP 401/403 (auth is broken, retrying just repeats the failure)
• HTTP 404 (the resource doesn't exist)
• Content policy or safety refusals from the model
• Tool-call validation errors caused by the model producing bad arguments (this needs a repair step, not a raw retry)

That last one matters for agents specifically. If the model calls a tool with {"amount": "fifty dollars"} instead of a number, retrying the exact same call will fail the exact same way. What you actually want is to feed the validation error back to the model and let it self-correct, which is a different code path than a network retry.

Here's a minimal classifier in Python:

class RetryDecision:
    RETRY = "retry"
    REPAIR = "repair"       # feed the error back to the model
    FAIL = "fail"           # give up, surface to caller

def classify_error(exc, response=None):
    status = getattr(response, "status_code", None)

    if status in (429, 500, 502, 503, 504):
        return RetryDecision.RETRY

    if isinstance(exc, (ConnectionError, TimeoutError)):
        return RetryDecision.RETRY

    if status == 400 and getattr(exc, "kind", None) == "tool_argument_error":
        return RetryDecision.REPAIR

    if status in (401, 403, 404):
        return RetryDecision.FAIL

    # unknown error: fail closed, don't assume it's transient
    return RetryDecision.FAIL

Fail closed on unknown errors. An agent retry loop that retries everything by default will happily loop on a 401 for five attempts before giving up, wasting time and, if it's inside a rate-limited quota, burning your remaining budget on a request that was never going to succeed.

Exponential backoff with jitter

Once you know an error is retryable, don't retry immediately. Fixed-delay retries synchronize badly when you have multiple agent workers hitting the same rate-limited endpoint: they all back off for exactly 1 second and then all retry at the same instant, recreating the spike. Exponential backoff with jitter spreads retries out.

import random
import time

def backoff_delay(attempt, base=1.0, cap=30.0, jitter=True):
    """attempt starts at 0"""
    delay = min(cap, base * (2 ** attempt))
    if jitter:
        delay = random.uniform(0, delay)  # full jitter
    return delay

def call_with_retry(fn, max_attempts=5):
    last_exc = None
    for attempt in range(max_attempts):
        try:
            return fn()
        except Exception as exc:
            decision = classify_error(exc)
            if decision != "retry" or attempt == max_attempts - 1:
                raise
            delay = backoff_delay(attempt)
            time.sleep(delay)
            last_exc = exc
    raise last_exc

Two details worth calling out:

1. Full jitter, not additive jitter. random.uniform(0, delay) (full jitter) spreads retries better than delay + random.uniform(0, small_offset) (additive jitter). AWS's original backoff writeup covers why full jitter reduces contention more effectively, and the same logic applies to LLM provider rate limits.
2. Respect `Retry-After`. If the provider or tool API returns a Retry-After header, use it instead of your own backoff formula. The server is telling you exactly when it'll accept traffic again; overriding that with your own exponential curve just means you retry too early and get rate-limited again.

def backoff_delay_with_retry_after(attempt, response=None, base=1.0, cap=30.0):
    retry_after = None
    if response is not None:
        retry_after = response.headers.get("Retry-After")
    if retry_after:
        return float(retry_after)
    return backoff_delay(attempt, base=base, cap=cap)

Idempotency keys for tool calls

This is the part most agent retry write-ups skip, and it's the one that actually causes production incidents. If a tool call times out, you don't know if the server processed it. Retrying a "create order" call without an idempotency key can create two orders.

The fix: every tool that has a side effect (write, charge, send, delete) should accept an idempotency key, and your agent runtime should generate one deterministically per tool-call attempt, not per retry.

import hashlib
import json

def idempotency_key(tool_name, arguments, agent_run_id, step_index):
    """Same call across retries produces the same key.
    A genuinely new call (new step_index) produces a new key."""
    payload = json.dumps(
        {"tool": tool_name, "args": arguments, "run": agent_run_id, "step": step_index},
        sort_keys=True,
    )
    return hashlib.sha256(payload.encode()).hexdigest()

def call_tool_with_idempotency(tool_fn, tool_name, arguments, agent_run_id, step_index):
    key = idempotency_key(tool_name, arguments, agent_run_id, step_index)
    return tool_fn(arguments, idempotency_key=key)

On the receiving end, the tool (your own API, Stripe, Razorpay, whatever) stores the idempotency key against the result of the first successful call and returns that cached result if the same key shows up again. Most payment APIs support this natively. For your own internal tools, a simple table with a unique constraint on the key column does the job:

create table tool_call_results (
    idempotency_key text primary key,
    result jsonb not null,
    created_at timestamptz not null default now()
);

Without this, agent retry on write operations is not safe to automate. You either need idempotency keys or you need the tool itself to be naturally idempotent (an upsert keyed on a stable external ID is a common shortcut).

Retry budgets, not just retry counts

A per-call max_attempts of 3 sounds safe until you realize an agent might call ten tools across a plan, and each one independently retries three times. In the worst case that's 30 attempts for one user request, at three times the latency and cost you budgeted for.

Track a retry budget at the agent-run level, not just per call:

class RetryBudget:
    def __init__(self, max_total_retries=8, max_time_seconds=120):
        self.max_total_retries = max_total_retries
        self.max_time_seconds = max_time_seconds
        self.used = 0
        self.start = time.monotonic()

    def can_retry(self):
        elapsed = time.monotonic() - self.start
        return self.used < self.max_total_retries and elapsed < self.max_time_seconds

    def consume(self):
        self.used += 1

Pass the same RetryBudget instance through every tool call in a run. Once it's exhausted, stop retrying and surface a clear failure to the caller (or to the model, so it can decide to try a different approach) instead of continuing to loop. This turns a runaway agent into a bounded-cost operation, which matters a lot once you're paying per token and per API call at scale.

Circuit breakers for flaky dependencies

Backoff handles a single call. A circuit breaker handles a dependency that's down for a while. If a tool API is returning 503s on every request, retrying each individual agent's calls with backoff still means every agent run pays the full retry cost before failing. A circuit breaker short-circuits that: after enough consecutive failures, it stops sending traffic for a cooldown period and fails fast instead.

import time

class CircuitBreaker:
    def __init__(self, failure_threshold=5, cooldown_seconds=30):
        self.failure_threshold = failure_threshold
        self.cooldown_seconds = cooldown_seconds
        self.failures = 0
        self.opened_at = None

    def is_open(self):
        if self.opened_at is None:
            return False
        if time.monotonic() - self.opened_at > self.cooldown_seconds:
            # half-open: allow one probe request through
            return False
        return True

    def record_success(self):
        self.failures = 0
        self.opened_at = None

    def record_failure(self):
        self.failures += 1
        if self.failures >= self.failure_threshold:
            self.opened_at = time.monotonic()

Wire it in front of the retry loop, keyed per tool or per downstream host:

breakers = {}  # tool_name -> CircuitBreaker

def call_tool(tool_name, fn):
    breaker = breakers.setdefault(tool_name, CircuitBreaker())
    if breaker.is_open():
        raise RuntimeError(f"{tool_name} circuit open, failing fast")
    try:
        result = call_with_retry(fn)
        breaker.record_success()
        return result
    except Exception:
        breaker.record_failure()
        raise

This matters more for agents than for typical services because agent runs are often long and stacked: a research agent might call the same search tool 20 times in one session. Without a circuit breaker, a dependency outage means 20 separate full retry cycles instead of one fast failure after the first few.

Timeouts: set them before you set retries

Retry and timeout are a pair, and getting the timeout wrong makes retry logic pointless. If your tool call timeout is 60 seconds and you retry 5 times with backoff, a single failing call can block an agent step for several minutes. Set timeouts based on what "too slow to be useful" means for that specific call, not a global default:

• LLM completion calls: scale timeout with expected output length; a short classification call should time out in seconds, a long generation call needs more room.
• Tool/API calls: use the tool's documented p99 latency plus a margin, not an arbitrary round number.
• Overall agent step budget: cap the sum of (timeout + backoff delays) per step so one flaky call can't dominate a multi-step plan.

import concurrent.futures

def call_with_timeout(fn, timeout_seconds):
    with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool:
        future = pool.submit(fn)
        try:
            return future.result(timeout=timeout_seconds)
        except concurrent.futures.TimeoutError:
            raise TimeoutError(f"call exceeded {timeout_seconds}s")

Putting it together in an agent loop

Here's how the pieces combine in a step of an agent's tool-calling loop:

def execute_tool_step(tool_name, arguments, agent_run_id, step_index, budget):
    breaker = breakers.setdefault(tool_name, CircuitBreaker())
    if breaker.is_open():
        return {"error": "dependency_unavailable", "tool": tool_name}

    key = idempotency_key(tool_name, arguments, agent_run_id, step_index)

    attempt = 0
    while True:
        try:
            result = call_with_timeout(
                lambda: tools[tool_name](arguments, idempotency_key=key),
                timeout_seconds=15,
            )
            breaker.record_success()
            return {"result": result}
        except Exception as exc:
            decision = classify_error(exc)
            breaker.record_failure()

            if decision == "repair":
                # hand the error back to the model instead of retrying blindly
                return {"error": "needs_repair", "detail": str(exc)}

            if decision != "retry" or not budget.can_retry():
                return {"error": "failed", "detail": str(exc)}

            budget.consume()
            delay = backoff_delay(attempt)
            time.sleep(delay)
            attempt += 1

The important structural choice here: retry, repair, and hard failure are three distinct return paths, not one exception handler. The calling agent loop can react differently to each: retry is invisible to the model, repair goes back into the conversation as a tool error message so the model can adjust its arguments, and a hard failure gets surfaced as a terminal state for that step.

Logging and observability for agent retry

Retry logic that isn't observable will silently degrade. At minimum, log per attempt:

• tool name, attempt number, error classification, delay applied
• whether the circuit breaker was open
• retry budget remaining at the time of the call

import logging

logger = logging.getLogger("agent.retry")

def log_attempt(tool_name, attempt, decision, delay, budget):
    logger.info(
        "retry_attempt",
        extra={
            "tool": tool_name,
            "attempt": attempt,
            "decision": decision,
            "delay_seconds": round(delay, 2) if delay else 0,
            "budget_remaining": budget.max_total_retries - budget.used,
        },
    )

Feed this into whatever metrics system you already use and alert on retry rate per tool, not just error rate. A tool with a 2% raw error rate but a 40% retry-success rate is fine. A tool where retries almost never succeed is a sign the "retryable" classification for that error type is wrong, and you're just adding latency for nothing.

Testing agent retry logic

Don't test retry logic by hoping a real API fails at the right moment. Inject failures deterministically:

class FlakyStub:
    """Fails N times, then succeeds. Use to test backoff and budget logic."""
    def __init__(self, fail_times, exception_factory):
        self.fail_times = fail_times
        self.calls = 0
        self.exception_factory = exception_factory

    def __call__(self, *args, **kwargs):
        self.calls += 1
        if self.calls <= self.fail_times:
            raise self.exception_factory()
        return {"ok": True}

Use it to assert three things: the call eventually succeeds within the retry budget, the number of attempts matches what the budget allows, and non-retryable errors fail on the first attempt without any delay. That third case is easy to get backwards during development, where it's tempting to test only the happy path of "retry until success" and never check that a 401 fails fast.

FAQ

Should every tool call in an agent have retry logic, or only some? Only calls that can fail transiently and are safe to repeat, meaning read-only calls or writes with an idempotency key. Wrapping every tool in retry logic without checking idempotency is how duplicate side effects happen. If a tool has a side effect and you can't add an idempotency key to it, treat a timeout as unknown state and fail rather than retry.

How many retry attempts is reasonable for an LLM API call? Three to five with exponential backoff and jitter is a common range, capped by an overall run-level retry budget rather than an unlimited per-call count. More than that rarely helps for a genuinely transient error and just adds latency for a call that's going to fail anyway.

What's the difference between retry and repair in an agent loop? Retry re-sends the exact same request after a delay, for transient infrastructure errors. Repair sends the error back into the model's context so it can produce a corrected tool call, for cases where the model itself made a mistake, like a malformed argument or a schema violation. Treating a repair case as a retry just repeats the same bad call.

Does exponential backoff need jitter, or is the exponential curve enough? It needs jitter. Without it, multiple agents or workers that fail at the same time end up retrying at the same time too, recreating the load spike that caused the rate limit in the first place. Full jitter (a random delay between 0 and the computed backoff ceiling) spreads retries out more effectively than adding a small random offset to a fixed delay.

How do circuit breakers interact with retry budgets? They operate at different levels. A retry budget limits how many attempts a single agent run makes across all its tool calls. A circuit breaker tracks the health of a dependency across many runs and stops sending traffic to it once it's clearly down, so new runs fail fast instead of each independently burning their retry budget against a dead endpoint.

Should I retry on a tool call that already partially succeeded? Only if the tool call is idempotent or you have an idempotency key that lets the server return the cached result instead of repeating the effect. If you can't guarantee that, treat a timeout on that call as an unknown outcome: check the actual state (did the order get created, did the email send) before deciding whether to retry, repair, or move on.