Error Recovery for LLM Applications: A Production Playbook

LLM error recovery means designing your application so that a failed model call, a malformed response, a rate limit, or a dropped connection never takes down the user-facing feature it powers. Unlike a typical REST API, an LLM call can fail in ways that are silent: the request succeeds with a 200, but the JSON is truncated, the model refuses the task, or it hallucinates a tool call that does not exist. If you have shipped an LLM feature and watched it work perfectly in a demo, then fall over in production under real traffic, this article is the fix. We will go through every layer where recovery needs to live: transport, validation, business logic, and observability, with code you can drop into a FastAPI or worker service today.

Why LLM Calls Fail Differently Than Normal API Calls

A normal API call fails in a handful of well-known ways: connection refused, timeout, 4xx, 5xx. You wrap it in a retry with backoff and move on. LLM error recovery has to handle a wider surface because the "response" is generated text, not a fixed schema pulled from a database.

Here is the failure taxonomy that matters in practice:

• Transport failures: connection resets, timeouts, DNS hiccups, TLS handshake failures. Same as any HTTP call.
• Rate limit and capacity errors: 429s from the provider, or 529 "overloaded" responses during peak traffic.
• Context length errors: your prompt plus history exceeds the model's context window.
• Output validation failures: the model returns text that does not parse as JSON, or JSON that does not match your schema.
• Tool call failures: the model calls a tool with the wrong arguments, calls a tool that does not exist, or calls no tool when one was required.
• Content policy refusals: the model declines to answer, sometimes mid-generation.
• Silent degradation: the call succeeds, the output validates, but the content is wrong, truncated, or repetitive (a loop).
• Streaming failures: the connection drops halfway through a stream, leaving a partial response.

Each of these needs a different recovery strategy. Retrying a context-length error with the same prompt will fail again forever; you need to truncate or summarize instead. Retrying a rate limit with the same request immediately just adds to the queue; you need exponential backoff with jitter. Building a single generic "try/except, retry 3 times" wrapper handles maybe a third of what actually goes wrong.

Building a Retry Layer That Knows What It Is Retrying

The first mistake most teams make is retrying everything the same way. A well-built retry layer classifies the error first, then decides whether to retry, how long to wait, and whether to change the request before retrying.

import random
import time
from dataclasses import dataclass
from enum import Enum


class ErrorClass(Enum):
    TRANSIENT = "transient"        # retry as-is
    RATE_LIMITED = "rate_limited"  # retry with backoff
    CONTEXT_TOO_LONG = "context_too_long"  # must shrink prompt first
    INVALID_OUTPUT = "invalid_output"      # must repair prompt first
    FATAL = "fatal"                # do not retry


@dataclass
class RetryDecision:
    should_retry: bool
    wait_seconds: float
    mutate_request: bool


def classify_error(exc: Exception) -> ErrorClass:
    name = type(exc).__name__
    message = str(exc).lower()

    if "ratelimit" in name.lower() or "429" in message:
        return ErrorClass.RATE_LIMITED
    if "overloaded" in message or "529" in message:
        return ErrorClass.RATE_LIMITED
    if "context_length" in message or "maximum context" in message:
        return ErrorClass.CONTEXT_TOO_LONG
    if "timeout" in name.lower() or "connection" in name.lower():
        return ErrorClass.TRANSIENT
    if "authenticationerror" in name.lower() or "permissiondenied" in name.lower():
        return ErrorClass.FATAL
    return ErrorClass.TRANSIENT


def decide_retry(error_class: ErrorClass, attempt: int, max_attempts: int = 5) -> RetryDecision:
    if attempt >= max_attempts:
        return RetryDecision(False, 0, False)

    if error_class == ErrorClass.FATAL:
        return RetryDecision(False, 0, False)

    if error_class == ErrorClass.RATE_LIMITED:
        base = min(2 ** attempt, 30)
        jitter = random.uniform(0, base * 0.3)
        return RetryDecision(True, base + jitter, False)

    if error_class == ErrorClass.CONTEXT_TOO_LONG:
        return RetryDecision(True, 0, True)

    if error_class == ErrorClass.TRANSIENT:
        base = min(1.5 ** attempt, 10)
        return RetryDecision(True, base, False)

    return RetryDecision(False, 0, False)

The mutate_request flag is the key idea: some errors need the request itself changed before a retry has any chance of succeeding. A context-length error will not go away by waiting five seconds, it needs the prompt trimmed. Wire this into the actual call loop:

def call_with_recovery(client, request, mutator, max_attempts=5):
    attempt = 0
    current_request = request

    while attempt < max_attempts:
        try:
            return client.messages.create(**current_request)
        except Exception as exc:
            error_class = classify_error(exc)
            decision = decide_retry(error_class, attempt, max_attempts)

            if not decision.should_retry:
                raise

            if decision.mutate_request:
                current_request = mutator(current_request)

            if decision.wait_seconds:
                time.sleep(decision.wait_seconds)

            attempt += 1

    raise RuntimeError("exhausted retries")

The mutator function is where you implement your context-shrinking strategy: drop the oldest turns from chat history, summarize earlier turns into a shorter block, or switch to a model with a larger context window as a fallback. Do not hardcode this into the retry loop itself; keep it swappable per endpoint since a chat feature and a document-summarization feature will want different truncation strategies.

Validating and Repairing Structured Output

If your application asks the model for JSON, structured output validation failures are the most common recovery scenario you will hit, more common than transport errors in most codebases. The fix is a validate-then-repair loop rather than a blind retry.

from pydantic import BaseModel, ValidationError
import json


class ExtractedInvoice(BaseModel):
    vendor: str
    total_amount: float
    currency: str
    line_items: list[str]


def extract_invoice(client, raw_text: str, max_repairs: int = 2) -> ExtractedInvoice:
    prompt = build_extraction_prompt(raw_text)
    last_error = None

    for attempt in range(max_repairs + 1):
        response = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=1024,
            messages=[{"role": "user", "content": prompt}],
        )
        raw = response.content[0].text

        try:
            parsed = json.loads(raw)
            return ExtractedInvoice.model_validate(parsed)
        except (json.JSONDecodeError, ValidationError) as exc:
            last_error = exc
            prompt = build_repair_prompt(raw, str(exc))

    raise ValueError(f"could not extract valid invoice after repairs: {last_error}")


def build_repair_prompt(bad_output: str, error_message: str) -> str:
    return (
        "Your previous response was not valid JSON matching the required schema.\n\n"
        f"Previous response:\n{bad_output}\n\n"
        f"Validation error:\n{error_message}\n\n"
        "Return ONLY the corrected JSON object, no explanation, no markdown fences."
    )

Feeding the exact validation error back to the model is the difference between a repair loop that converges in one or two turns and one that flails randomly. Pydantic's ValidationError messages are specific enough ("field total_amount: expected float, got str") that the model can usually fix its own mistake immediately.

If you are doing this frequently across many endpoints, look at libraries built specifically for this loop, like instructor for Python, which wraps the client and handles the validate-repair cycle for you, including capping retries and raising a typed exception when it gives up. Building your own is fine for one or two extraction paths; past that, adopt the library so the retry logic is not duplicated across your codebase.

Tool Call Recovery for Agents

Agentic workflows add a failure mode that plain text generation does not have: the model can call a tool with arguments that do not match the tool's schema, call a tool name that does not exist in the current toolset, or return a tool call when you expected a final answer. Recovery here happens at the tool-execution boundary, not the API-call boundary.

def execute_tool_call(tool_call, registry: dict):
    tool_name = tool_call.name
    if tool_name not in registry:
        return {
            "tool_use_id": tool_call.id,
            "content": f"Error: tool '{tool_name}' does not exist. "
                       f"Available tools: {', '.join(registry.keys())}",
            "is_error": True,
        }

    tool_fn, schema = registry[tool_name]

    try:
        validated_args = schema.model_validate(tool_call.input)
    except ValidationError as exc:
        return {
            "tool_use_id": tool_call.id,
            "content": f"Error: invalid arguments for '{tool_name}': {exc}",
            "is_error": True,
        }

    try:
        result = tool_fn(**validated_args.model_dump())
        return {"tool_use_id": tool_call.id, "content": json.dumps(result), "is_error": False}
    except Exception as exc:
        return {
            "tool_use_id": tool_call.id,
            "content": f"Error executing '{tool_name}': {exc}",
            "is_error": True,
        }

The pattern to notice: every failure path returns a tool result with is_error: True and a human-readable message, rather than raising an exception up the call stack. Feed that error result back into the conversation as the next turn and let the model see it. In practice, models correct their own tool-calling mistakes when shown the actual error, in the same way they correct malformed JSON when shown a validation error. Treat the agent loop itself as the recovery mechanism, and only escalate to a hard failure after a capped number of consecutive tool errors (three is a reasonable default before you give up and hand off to a human or a fallback path).

Circuit Breakers and Fallback Models

Retries handle isolated failures. They do not handle sustained outages, where a provider is down for ten minutes and every retry just burns latency and money before failing anyway. That is what a circuit breaker is for: after a threshold of consecutive failures, stop calling the primary path entirely for a cooldown window, and route to a fallback.

import time
from enum import Enum


class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


class CircuitBreaker:
    def __init__(self, failure_threshold=5, cooldown_seconds=30):
        self.failure_threshold = failure_threshold
        self.cooldown_seconds = cooldown_seconds
        self.failure_count = 0
        self.state = CircuitState.CLOSED
        self.opened_at = None

    def call(self, primary_fn, fallback_fn, *args, **kwargs):
        if self.state == CircuitState.OPEN:
            if time.time() - self.opened_at > self.cooldown_seconds:
                self.state = CircuitState.HALF_OPEN
            else:
                return fallback_fn(*args, **kwargs)

        try:
            result = primary_fn(*args, **kwargs)
            self._on_success()
            return result
        except Exception:
            self._on_failure()
            return fallback_fn(*args, **kwargs)

    def _on_success(self):
        self.failure_count = 0
        self.state = CircuitState.CLOSED

    def _on_failure(self):
        self.failure_count += 1
        if self.failure_count >= self.failure_threshold:
            self.state = CircuitState.OPEN
            self.opened_at = time.time()

The fallback function is where design judgment matters most. Common fallback tiers, roughly in order of preference:

1. Same model family, different region or endpoint, if your provider offers it, since this recovers from a regional outage without changing behavior.
2. A smaller or older model in the same family, accepting a quality drop to keep the feature functioning rather than showing an error to the user.
3. A cached or templated response, appropriate for features like summarization where a slightly stale cached summary beats no summary.
4. A queued retry, where you accept the request, tell the user it is processing, and complete it asynchronously once the primary path recovers.
5. A graceful denial, the last resort, where the UI clearly explains the feature is temporarily unavailable rather than hanging or returning a broken result.

Wire the circuit breaker per provider and per model, not globally. A rate limit on your embeddings calls should not open the circuit for your chat completions calls; they are different quotas even if they hit the same underlying provider.

Idempotency for Retried Side Effects

The recovery patterns above assume retrying a call is safe. That is true for read-only generation, but not for anything that triggers a side effect: sending an email, charging a customer, writing to a database, calling another API. If a request times out after the model already triggered a tool that sent an email, and your retry logic calls it again, the user gets two emails.

The fix is an idempotency key attached to every request that can trigger a side effect, checked before the side effect runs:

import hashlib


def idempotency_key(user_id: str, action: str, payload: dict) -> str:
    raw = f"{user_id}:{action}:{json.dumps(payload, sort_keys=True)}"
    return hashlib.sha256(raw.encode()).hexdigest()


def send_email_tool(user_id: str, recipient: str, subject: str, body: str, seen_keys: set):
    key = idempotency_key(user_id, "send_email", {
        "recipient": recipient, "subject": subject, "body": body
    })
    if key in seen_keys:
        return {"status": "skipped_duplicate"}

    seen_keys.add(key)
    # actual send logic here
    return {"status": "sent"}

In production, seen_keys is a Redis set with a TTL matching your retry window, not an in-memory set. The key point for LLM error recovery specifically: any tool the model can call that mutates state needs this guard, because the model itself may re-issue the same tool call across a retried agent turn, independent of your own retry logic.

Streaming Recovery

Streaming responses fail differently: the connection can drop after the client has already rendered half a response to the user. Two things need to happen. First, buffer enough of the stream that you can detect a clean end-of-stream signal versus a dropped connection. Second, decide whether to resume or restart, since most providers do not support resuming a stream mid-generation.

def stream_with_recovery(client, request, on_chunk, max_resume_attempts=2):
    accumulated = ""
    attempts = 0

    while attempts <= max_resume_attempts:
        try:
            with client.messages.stream(**request) as stream:
                for chunk in stream.text_stream:
                    accumulated += chunk
                    on_chunk(chunk)
                final = stream.get_final_message()
                if final.stop_reason in ("end_turn", "stop_sequence"):
                    return accumulated
                # stop_reason of "max_tokens" or an unexpected stop needs handling
                break
        except Exception:
            attempts += 1
            if attempts > max_resume_attempts:
                raise
            on_chunk("\n\n[reconnecting...]\n\n")
            request = build_continuation_request(request, accumulated)

    return accumulated


def build_continuation_request(original_request, partial_text):
    messages = list(original_request["messages"])
    messages.append({"role": "assistant", "content": partial_text})
    messages.append({"role": "user", "content": "Continue exactly where you left off."})
    new_request = dict(original_request)
    new_request["messages"] = messages
    return new_request

For the UI side, always render a visible indicator when you are recovering from a dropped stream rather than silently splicing text together; users notice a seam in the response even if the words are correct, and an honest "reconnecting" message reads better than a confusing jump in tone.

Observability: Making Failures Visible Before Users Report Them

None of the above matters if you cannot see it happening. LLM error recovery needs its own logging dimension separate from generic application logs, because the interesting signal is not "did the HTTP call succeed" but "did the recovery path fire, and how often."

Track these counters per feature and per model:

• Retry count by error class (transient, rate limited, context too long, invalid output)
• Repair loop iterations for structured output, and the final success/failure rate
• Circuit breaker state transitions, with timestamps
• Fallback tier usage, so you know when users are silently getting a degraded model
• Tool call error rate, broken down by tool name

import logging

logger = logging.getLogger("llm_recovery")


def log_recovery_event(event_type: str, feature: str, model: str, **details):
    logger.info(
        "llm_recovery_event",
        extra={
            "event_type": event_type,
            "feature": feature,
            "model": model,
            **details,
        },
    )

Feed this into whatever metrics stack you already run, Datadog, Grafana, or plain structured logs into your log aggregator. Set an alert on fallback-tier usage crossing a threshold; a spike there means your primary provider is degraded even if every individual user request still eventually succeeds, and you want to know that before it turns into an outage.

Putting It Together

A production LLM feature needs recovery at every layer: classify the error, decide whether retrying helps, mutate the request when a blind retry cannot succeed, validate and repair structured output with the actual error fed back to the model, guard side-effecting tools with idempotency keys, open a circuit breaker and fall back to a degraded path during sustained outages, and log every recovery event so you catch degradation before your users file a ticket about it. None of these patterns are exotic engineering; they are the same reliability discipline you would apply to any external dependency, adapted to the specific ways an LLM call can go wrong that a database call cannot.

Start with the retry classifier and the structured-output repair loop, since those two cover the majority of real-world failures. Add the circuit breaker and idempotency guard once you have tool-calling agents in production, since that is where an unguarded retry turns into a user-visible bug rather than a quiet log entry.

FAQ

What is the difference between a retry and a repair loop in LLM error recovery? A retry re-sends the same request, useful for transient failures like a dropped connection or a rate limit. A repair loop changes the request based on what went wrong, typically by feeding the validation error back to the model, and is what you need for malformed structured output that a plain retry would reproduce.

Should I retry every LLM API error automatically? No. Authentication errors, permission errors, and content policy refusals will not resolve with a retry and should fail fast. Classify the error first, then decide, rather than wrapping every call in a blanket retry.

How many retries is reasonable for an LLM call? Three to five attempts with exponential backoff and jitter is a common range. Beyond that you are usually masking a real outage rather than recovering from a blip, and the circuit breaker should take over instead of continuing to retry.

Do I need an idempotency key if my LLM feature only generates text? No. Idempotency keys matter specifically for tool calls or downstream actions that have a side effect, such as sending a notification or writing to a database. Pure text generation is safe to retry freely.

What is the best fallback when my primary model provider is down? It depends on the feature. For latency-sensitive chat, a smaller model in the same family keeps the feature usable. For batch or async work, a queued retry that completes once the provider recovers is usually better than serving a degraded response.

How do I test error recovery paths before they happen in production? Wrap your client in a fault-injection layer during testing that can force specific error classes, timeouts, malformed JSON, simulated rate limits, and run your recovery code against each one explicitly rather than waiting for a real outage to exercise the path for the first time.