Production AIreliabilityerror handlingdistributed systemsapi design

Retries and Idempotency for LLM Calls

Pramod Dutta · Jun 28, 2026 · 15 min read

LLM retry idempotency is the discipline of retrying failed model calls safely so a single logical request never produces duplicate side effects, even when the network, the provider, or your own worker crashes mid-flight. In practice that means two things working together: a retry policy that knows which errors are worth retrying and backs off politely, and an idempotency layer that guarantees a retried call reuses the first result instead of running the whole pipeline again. Get one without the other and you either drop requests on transient failures or you charge a customer twice, send two emails, or write two rows for what the user thinks was one action.

This article is for engineers wiring LLM calls into real backends, jobs, and agents. Every example is runnable, uses the plain HTTP and SDK patterns you already have, and avoids invented numbers. We will build up from a naive call to a retry-safe, idempotent client, then extend the same idea to multi-step pipelines and background workers.

Why LLM calls fail more than normal APIs

A typical LLM request is a long, expensive, stateful-feeling HTTP call over a connection that can stay open for many seconds or minutes with streaming. That long window is exactly when things break. The failure modes you will actually see:

Rate limits (HTTP 429) when you burst past your tokens-per-minute or requests-per-minute quota.
Overloaded or server errors (HTTP 500, 502, 503, 529) when the provider is under load.
Timeouts, both client-side (your read timeout fires) and network-side (a proxy or load balancer kills an idle connection during a long generation).
Connection resets mid-stream, where you have received part of a response and then the socket dies.
Content and validation errors (HTTP 400, 422) when your prompt is malformed or exceeds the context window.

The first three categories are transient: the same request retried a moment later will probably succeed. The last category is not: retrying a malformed request just burns money and quota. A good llm retry idempotency setup treats these two groups completely differently. Retry the transient ones with backoff; fail fast and surface the permanent ones.

The subtle trap is the timeout. When your client times out, you do not know whether the server finished the work. It may have generated a full completion, billed you for it, and been unable to deliver it because your side hung up. If that call had a side effect (writing to a database, calling a tool, charging a card), a blind retry doubles it. This is why retries alone are dangerous and why idempotency is the other half of the story.

Retry policy: classify, back off, cap

Start with the classification. Do not retry on blanket "any exception." Decide per error class.

RETRYABLE_STATUS = {408, 409, 429, 500, 502, 503, 504, 529}

def is_retryable(status_code, exception):
    if exception is not None:
        # network-level: timeouts, connection resets, DNS
        return isinstance(exception, (TimeoutError, ConnectionError))
    return status_code in RETRYABLE_STATUS

Note that 400 and 422 are absent. A 400 means your request is wrong; retrying it unchanged is pointless. A 401 or 403 is an auth problem, also not retryable. Being explicit here saves you from silent money leaks.

Next, back off with jitter. Fixed-interval retries create thundering herds: every worker that hit the rate limit at the same instant retries at the same instant and hits it again. Exponential backoff with full jitter spreads them out.

import random
import time

def backoff_delay(attempt, base=0.5, cap=30.0):
    # attempt starts at 0
    ceiling = min(cap, base * (2 ** attempt))
    return random.uniform(0, ceiling)

Full jitter (a uniform pick between zero and the ceiling) is the version worth using. It de-correlates retries across workers far better than a fixed multiplier does.

If the provider sends a Retry-After header on a 429 or 503, respect it. It is the provider telling you exactly when to come back, and it beats your guess.

def resolve_delay(attempt, headers):
    ra = headers.get("retry-after")
    if ra is not None:
        try:
            return float(ra)
        except ValueError:
            pass
    return backoff_delay(attempt)

Finally, cap the attempts and the total time. Infinite retries turn a provider outage into your outage as jobs pile up. A common shape is up to five or six attempts with an overall deadline, after which you give up and let the caller decide (queue for later, degrade, or surface an error).

Here is the retry loop assembled, using a synchronous HTTP client so the mechanics are visible.

import httpx

def call_with_retries(client, request_fn, max_attempts=6, deadline_s=120):
    start = time.monotonic()
    last_error = None
    for attempt in range(max_attempts):
        if time.monotonic() - start > deadline_s:
            break
        try:
            resp = request_fn(client)
            if resp.status_code < 400:
                return resp
            if not is_retryable(resp.status_code, None):
                resp.raise_for_status()
            delay = resolve_delay(attempt, resp.headers)
        except (httpx.TimeoutException, httpx.ConnectError) as exc:
            last_error = exc
            delay = backoff_delay(attempt)
        else:
            last_error = httpx.HTTPStatusError(
                "retryable status", request=resp.request, response=resp
            )
        time.sleep(delay)
    raise RuntimeError(f"exhausted retries: {last_error}")

Most official SDKs (the Anthropic and OpenAI Python and TypeScript clients among them) ship a built-in retry with backoff and let you set max_retries and per-request timeouts. Use those defaults as a floor, not a ceiling. They handle the transient-error backoff for you, but they do not know about your side effects, so they cannot make your pipeline idempotent. That part is on you.

Idempotency: make a retry a no-op

Idempotency means running the same operation twice has the same effect as running it once. For LLM work there are two distinct places you need it, and people constantly conflate them.

The first is the provider call itself. Some LLM APIs accept an idempotency key (often an Idempotency-Key header or an idempotency_key field). You generate a unique key per logical request and send the same key on every retry. If the provider already processed a request with that key, it returns the original result instead of generating (and billing) again. This directly defends against the timeout trap: your client timed out, the server finished, and your retry with the same key gets the cached completion back rather than a second generation.

import uuid

def new_idempotency_key():
    return f"req-{uuid.uuid4()}"

def request_fn(key, payload):
    def _do(client):
        return client.post(
            "https://api.provider.example/v1/messages",
            headers={
                "authorization": "Bearer ...",
                "idempotency-key": key,
                "content-type": "application/json",
            },
            json=payload,
            timeout=httpx.Timeout(connect=5, read=120, write=10, pool=5),
        )
    return _do

key = new_idempotency_key()
resp = call_with_retries(client, request_fn(key, payload))

The critical rule: generate the key once, outside the retry loop, and reuse it across every attempt. A fresh key per attempt defeats the entire purpose. Providers typically scope these keys to a time window (a day is common), so treat them as short-lived, not permanent.

Check your provider's docs for whether idempotency keys are supported and how long they are honored; do not assume. Where the API does not offer them, you fall back to the second layer, which you should build anyway.

The second place is your own pipeline. The provider's idempotency key only covers the provider's side. It does nothing for the database row you write, the email you send, or the payment you capture after the model responds. You need your own idempotency store keyed on the logical unit of work.

An application-level idempotency store

The pattern is a small table (or Redis key) that records, per idempotency key, the state and eventual result of an operation. Before doing expensive or side-effecting work, you claim the key. If it is already completed, you return the stored result. If it is in progress, you wait or reject. If it is new, you do the work and record the outcome.

create table idempotency_records (
    idem_key text primary key,
    status text not null,          -- 'in_progress' | 'done' | 'failed'
    result jsonb,
    created_at timestamptz not null default now(),
    updated_at timestamptz not null default now()
);

The claim uses an atomic insert so two concurrent workers cannot both start the same operation.

import json
import psycopg

def claim_or_get(conn, idem_key):
    with conn.cursor() as cur:
        cur.execute(
            """
            insert into idempotency_records (idem_key, status)
            values (%s, 'in_progress')
            on conflict (idem_key) do nothing
            returning idem_key
            """,
            (idem_key,),
        )
        if cur.fetchone() is not None:
            return ("claimed", None)
        cur.execute(
            "select status, result from idempotency_records where idem_key = %s",
            (idem_key,),
        )
        status, result = cur.fetchone()
        return (status, result)

The on conflict do nothing ... returning trick is the load-bearing part. Exactly one caller gets a row back from the insert and owns the work. Everyone else falls through to the select and sees either the finished result or an in-progress marker. This is a database-enforced lock, not an application-level one, so it survives a race between two workers on two machines.

Wrapping the LLM call and its side effects:

def run_idempotent(conn, idem_key, do_work):
    state, result = claim_or_get(conn, idem_key)
    if state == "done":
        return result
    if state == "in_progress":
        raise InProgressError(idem_key)   # caller can retry later
    if state == "failed":
        # decide policy: reopen for retry, or return the failure
        pass

    try:
        output = do_work()               # LLM call + DB writes + tool calls
    except Exception:
        _mark(conn, idem_key, "failed", None)
        raise
    _mark(conn, idem_key, "done", output)
    return output

def _mark(conn, idem_key, status, result):
    with conn.cursor() as cur:
        cur.execute(
            """
            update idempotency_records
            set status = %s, result = %s, updated_at = now()
            where idem_key = %s
            """,
            (status, json.dumps(result) if result is not None else None, idem_key),
        )
    conn.commit()

Now a retried job with the same idem_key short-circuits to the stored result. The expensive generation and its side effects run once. This is the heart of llm retry idempotency at the application layer, and it works whether or not the provider offers its own keys.

Choosing the idempotency key

The key must be derived from the logical request, not generated randomly per call, otherwise a retry produces a new key and reruns everything. Good sources, in rough order of preference:

A client-supplied request id passed in from the caller (best: the caller controls dedup).
The upstream event id if you are processing a queue or webhook (for example the message id from your broker).
A deterministic hash of the meaningful inputs: user id, action, and a content hash of the prompt.

import hashlib

def derive_key(user_id, action, prompt):
    h = hashlib.sha256(prompt.encode("utf-8")).hexdigest()[:16]
    return f"{user_id}:{action}:{h}"

Hashing the prompt is convenient but has a sharp edge: if the same user legitimately asks the same question twice and you want two separate answers, a content hash collapses them into one. Include a caller-provided nonce or a coarse time bucket when repeat requests are meant to be distinct. Match the key's identity to the business meaning of "the same request."

Streaming, partial output, and non-determinism

Streaming complicates retries because a failure can arrive after you have already emitted tokens to the user. If the stream dies at token 400 of an expected 800, you cannot naively restart and re-stream from zero without the user seeing a jarring restart or duplicated text.

Practical handling:

Buffer server-side and only commit the completion (and its side effects) once the stream finishes cleanly. Treat a broken stream as a failed attempt and retry the whole call under the same idempotency key, discarding the partial.
If you must show tokens live, render them as provisional and reconcile against the final committed text, or accept that a mid-stream failure means a visible restart.
Never persist side effects from a partial stream. Tool calls, database writes, and downstream messages fire only after a clean finish.

Non-determinism is the other reason to store results rather than recompute. Even at temperature zero, model outputs are not guaranteed bit-identical across calls or model versions. If your system's correctness depends on a specific output (a parsed JSON that a downstream step consumes), recomputing on retry can yield a different structure and break the pipeline. Storing the first successful result and replaying it on retry sidesteps this entirely. That is a second, independent reason idempotency matters here that pure REST APIs do not face.

Retries inside multi-step agents and pipelines

A single LLM call is the easy case. Agents and chains that make several calls, invoke tools, and branch need idempotency at each step, not just at the top.

The reliable pattern is to give every step its own idempotency key derived from the run id plus the step id, and to make each step's side effects individually replayable. Structure it as a durable state machine:

def run_step(conn, run_id, step_id, do_step):
    step_key = f"{run_id}:{step_id}"
    return run_idempotent(conn, step_key, do_step)

def agent_run(conn, run_id, user_input):
    plan = run_step(conn, run_id, "plan",
                    lambda: llm_plan(user_input))
    facts = run_step(conn, run_id, "retrieve",
                     lambda: run_tool("search", plan["query"]))
    answer = run_step(conn, run_id, "synthesize",
                      lambda: llm_answer(plan, facts))
    return answer

If the whole run crashes after the retrieve step, restarting it with the same run_id replays plan and retrieve from the store and only executes synthesize fresh. You do not repeat the expensive retrieval or re-bill the planning call. This is how durable execution frameworks (Temporal, and queue-plus-state designs you build yourself) keep long agent runs safe: every side-effecting activity is idempotent and its result is checkpointed.

A few rules that keep this honest:

Make tool calls idempotent too. A tool that sends an email must dedupe on the step key, or you email twice on replay.
Keep steps small and side-effect-focused so a replay boundary is meaningful.
Store the result of each step, not just a "done" flag, so replay returns real data to the next step.

Interaction with rate limits and queues

Retries and rate limits fight each other if you are careless. Aggressive retries during a 429 storm add load exactly when the provider is telling you to back off. Coordinate them:

Honor Retry-After before your own backoff math.
Add a concurrency limit or token-bucket in front of the provider so you never exceed your quota in the first place, turning most 429s into local queueing instead of failed calls.
Use a dead-letter queue for requests that exhaust retries, so an outage parks work for later instead of dropping it. Because the work is idempotent, replaying the dead-letter queue after recovery is safe.

The combination (a limiter to avoid limits, retries with jitter for the ones you still hit, and idempotency so replays are free) is what keeps a system stable when the provider has a bad hour.

A checklist you can apply today

Classify errors: retry 408, 409, 429, 5xx, timeouts, and connection resets; fail fast on 400, 401, 403, 422.
Exponential backoff with full jitter, honoring Retry-After, capped attempts, and an overall deadline.
Send a provider idempotency key generated once per logical request and reused across every retry, where the API supports it.
Build an application-level idempotency store with an atomic claim, so retries replay the stored result instead of rerunning side effects.
Derive keys from the logical request (caller id, event id, or input hash), never randomly per attempt.
Commit side effects only after a clean, complete response; discard partial streams.
Give every step in an agent its own key and checkpoint its result for safe replay.
Put a limiter and a dead-letter queue around the provider so retries never become a self-inflicted outage.

FAQ

Do I still need my own idempotency if the provider supports idempotency keys? Yes. The provider's key only makes the provider's call idempotent. It does nothing for the database rows, emails, payments, or tool calls your code performs after the model responds. Use the provider key to avoid double generation and double billing, and use your own store to make the whole pipeline replay-safe. They cover different blast radii and you want both.

What errors should I never retry? Anything that says your request is wrong or unauthorized: 400 bad request, 401 and 403 auth failures, 404, and 422 validation errors. Retrying these unchanged wastes quota and money because the same input will fail the same way. Content-policy refusals are also not retryable as-is; you have to change the prompt. Fix the input or the credentials, then resubmit as a new request.

How many retries and how long should backoff run? There is no universal number, so pick from your latency budget rather than a myth. A common shape is up to five or six attempts with exponential backoff, full jitter, a per-delay cap of tens of seconds, and an overall deadline so a provider outage does not make jobs pile up forever. If you have a hard user-facing latency limit, a smaller attempt count with a shorter deadline and a fallback path is better than retrying past the point the user has left.

Where should the idempotency key come from? From the logical request, so a retry reproduces the same key. Best is a client-supplied request id. Next best is the upstream event or message id when processing a queue or webhook. If you have neither, hash the meaningful inputs (user id, action, prompt content), but add a nonce or time bucket when repeat identical requests are supposed to produce separate results, otherwise a content hash will incorrectly collapse them.

How do retries work with streaming responses? Treat a broken stream as a failed attempt. Buffer server-side and only commit the completion and any side effects once the stream finishes cleanly, then retry the whole call under the same idempotency key if it broke. Do not persist partial output or fire tool calls from an incomplete stream. If you are rendering tokens live, mark them provisional and reconcile against the final committed text.

Will retrying give me a different answer because the model is non-deterministic? It can, which is exactly why you store the first successful result and replay it on retry instead of recomputing. Even at temperature zero, outputs are not guaranteed identical across calls or model versions, so a downstream step that parses a specific structure can break on a recompute. The idempotency store returns the original result, keeping the rest of your pipeline stable.

Does this apply to background jobs and agents, or only synchronous requests? It applies everywhere, and it matters most in background jobs and agents because those are the systems that crash and get replayed. Give each job or each agent step an idempotency key derived from a run id and step id, checkpoint each step's result, and make every side effect dedupe on that key. Then a worker crash halfway through replays completed steps for free and only runs the unfinished work.