LangChain

LangChain Error Handling: Retries, Fallbacks and Timeouts

Ira Menon · Jun 30, 2026 · 14 min read

Why Your LangChain App Breaks in Production (But Not in the Demo)

Every LangChain tutorial ends the same way: you run the chain in a notebook, the model responds beautifully, and you ship it. Then three days later, at 2 a.m., your on-call phone buzzes because a customer-facing chatbot has been throwing 500s for twenty minutes. You check the logs. It is not a bug in your prompt. It is not a hallucination problem. It is a RateLimitError from your provider, or a socket timeout, or a transient 503 that would have resolved itself if the request had just been retried once.

This is the part of building with LLMs that nobody puts in the getting-started guide, and it is also the part that separates a weekend project from something you can actually put in front of paying users. Language model APIs are remote services, and remote services fail. Not occasionally — constantly, in small, statistically predictable ways. Rate limits get hit during traffic spikes. Context windows overflow when a user pastes in a huge document. Providers have regional outages. Response bodies occasionally arrive malformed. None of these are edge cases; they are Tuesday.

The good news is that LangChain ships with first-class primitives for exactly this problem: .with_retry() for transient failures, .with_fallbacks() for provider outages and model-specific errors, and configurable timeouts so a hung request doesn't hold your whole application hostage. Used together, these three tools turn a fragile chain into something that degrades gracefully instead of falling over. This article walks through all three, with working code, and explains the reasoning behind the patterns so you can adapt them instead of just copy-pasting.

The Anatomy of an LLM Call Failure

Before reaching for retry logic, it helps to categorize what actually goes wrong, because the right fix depends on the failure type.

Transient/network errors — connection resets, DNS hiccups, brief 5xx responses from the provider's load balancer. These are almost always worth retrying immediately, because the failure has nothing to do with your request.
Rate limit errors (429) — the provider is telling you to slow down. Retrying immediately makes this worse; retrying with exponential backoff is the correct response.
Timeouts — the request hangs longer than acceptable, often because of provider-side load or an unusually long generation. You need a hard ceiling on wait time, independent of retries.
Model/provider-specific failures — an entire provider is down, a specific model has been deprecated, or a request violates that provider's content policy in a way another provider wouldn't. No amount of retrying fixes this — you need a fallback to a different model or provider entirely.
Malformed output — the model returns text that fails your output parser (bad JSON, a missing field). This is a retry-with-correction problem, often solved differently from network-level retries.

Conflating these categories is the most common mistake. Retrying a malformed-output error ten times against the same model with the same prompt just burns tokens and produces the same bad output ten times. Falling back to a different provider when you actually just got rate-limited is unnecessarily expensive. The rest of this article treats each category with the tool actually built for it.

Retries with `.with_retry()`: Handling Transient Failures

Every Runnable in LangChain — which includes chat models, prompts, output parsers, and full chains built with LCEL (the LangChain Expression Language) — exposes a .with_retry() method. It wraps the runnable so that on failure, it automatically re-executes according to a backoff policy you configure.

Here is the baseline pattern:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

resilient_llm = llm.with_retry(
    retry_if_exception_type=(Exception,),
    wait_exponential_jitter=True,
    stop_after_attempt=4,
)

response = resilient_llm.invoke("Summarize the concept of exponential backoff in two sentences.")
print(response.content)

A few details matter here:

wait_exponential_jitter=True adds randomized jitter on top of exponential backoff. Without jitter, if you have many concurrent requests that all fail at the same moment (say, a provider-wide blip), they will all retry at the same intervals and hammer the API in synchronized waves. Jitter spreads retries out so they don't stampede.
stop_after_attempt=4 caps the total number of attempts, including the original call. This is non-negotiable — retrying forever on a permanently broken request will silently hang your application or your user's request.
retry_if_exception_type lets you scope retries to specific exception classes. In production, you almost never want (Exception,) — you want to retry network and rate-limit errors but *not* retry, say, an authentication failure, because retrying bad credentials four times just wastes four round trips before you get the same failure.

A more targeted version, scoped to the exceptions that are actually worth retrying:

import openai
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

resilient_llm = llm.with_retry(
    retry_if_exception_type=(
        openai.APIConnectionError,
        openai.RateLimitError,
        openai.APITimeoutError,
    ),
    wait_exponential_jitter=True,
    stop_after_attempt=5,
)

This is a meaningfully better default than retrying on any exception. openai.AuthenticationError and openai.BadRequestError (malformed request, invalid model name, content policy rejection) will never succeed on retry — they need a code fix, not a retry loop. Only retry the exception types that represent transient conditions.

Because .with_retry() returns a Runnable, it composes naturally into a full LCEL chain:

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_template(
    "Explain {topic} to a junior engineer in three sentences."
)

chain = prompt | resilient_llm | StrOutputParser()

result = chain.invoke({"topic": "idempotency in distributed systems"})
print(result)

You can also apply .with_retry() to the *whole chain* rather than just the model call, which is useful when a downstream step (like a custom output parser or a tool call) is the thing that occasionally fails:

resilient_chain = chain.with_retry(
    stop_after_attempt=3,
    wait_exponential_jitter=True,
)

One nuance worth internalizing: retrying a whole chain re-runs every step in that chain, including the LLM call, even if the LLM call succeeded and the failure happened in a later parsing step. If your chain has expensive or non-idempotent steps (writing to a database, calling a paid third-party API), be deliberate about what you wrap in .with_retry(). Wrap the smallest unit that actually fails, not the largest unit that's convenient to wrap.

Fallbacks with `.with_fallbacks()`: Surviving Provider Outages

Retries solve "this will probably work if I wait a moment and try again." They do nothing for "this provider is down," "this model has been sunset," or "this specific request triggers a content filter on Provider A but not Provider B." For those situations, you need a fallback — a different runnable that takes over when the primary one exhausts its attempts and still fails.

.with_fallbacks() takes a list of alternative runnables and tries them in order after the primary fails:

from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic

primary_llm = ChatOpenAI(model="gpt-4o", temperature=0).with_retry(
    stop_after_attempt=3,
    wait_exponential_jitter=True,
)

backup_llm = ChatAnthropic(model="claude-sonnet-4-5", temperature=0).with_retry(
    stop_after_attempt=3,
    wait_exponential_jitter=True,
)

llm_with_fallback = primary_llm.with_fallbacks([backup_llm])

response = llm_with_fallback.invoke("Draft a one-paragraph incident summary for a database failover.")
print(response.content)

Notice the retry-then-fallback layering: each model in the fallback chain gets its own retry policy, so a single transient blip on the primary doesn't immediately punt to the backup. Only after the primary has genuinely exhausted its retries does the request fail over. This ordering matters — without retries on the primary, you'd fall back to your (probably more expensive, or lower-quality-for-your-use-case) backup model on every minor hiccup, which is both wasteful and can quietly degrade output quality without anyone noticing.

You can chain multiple fallbacks, and they're tried in sequence:

llm_with_fallbacks = primary_llm.with_fallbacks(
    [backup_llm, tertiary_llm],
    exceptions_to_handle=(Exception,),
)

exceptions_to_handle scopes which exception types trigger a fallback, similar to retry_if_exception_type on .with_retry(). You can also pass exception_key to have the exception injected into the input of the fallback runnable, which is handy if your fallback logic wants to log or react to *why* the primary failed:

from langchain_core.runnables import RunnableLambda

def log_and_pass_through(inputs: dict):
    if "exception" in inputs:
        print(f"Primary model failed with: {inputs['exception']}")
    return inputs["input"]

fallback_chain = RunnableLambda(log_and_pass_through) | backup_llm

llm_with_logged_fallback = primary_llm.with_fallbacks(
    [fallback_chain],
    exception_key="exception",
)

Fallbacks compose at the chain level too, not just the model level. This is useful if the failure mode you're worried about is a full pipeline behavior — for instance, a RAG chain that fails if the vector store times out, where the fallback is a simpler chain that answers from the model's parametric knowledge without retrieval:

full_rag_chain = retrieval_prompt | primary_llm | StrOutputParser()
no_retrieval_chain = simple_prompt | backup_llm | StrOutputParser()

safe_chain = full_rag_chain.with_fallbacks([no_retrieval_chain])

This pattern — fall back to a degraded-but-functional chain rather than a full outage — is often more valuable to end users than falling back to a different model of equivalent capability. A slightly less accurate answer beats a spinning loader or an error page every time.

Timeouts: Don't Let One Slow Request Take Down Everything

Retries and fallbacks both assume the failure happens *quickly*. But sometimes a request doesn't fail — it just hangs. A provider under load might take 90 seconds to respond to something that normally takes 3. Without a timeout, your retry logic never even triggers, because from the code's perspective, the call hasn't failed yet — it's just still running. Meanwhile, a user is staring at a blank screen, and if you're running this inside a web request handler, you may be holding a worker thread or a connection slot the entire time.

LangChain chat models accept a request_timeout (or timeout, depending on the integration) parameter directly:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-4o-mini",
    temperature=0,
    timeout=15,
    max_retries=0,
)

Note max_retries=0 here — this disables the underlying SDK's own built-in retry logic so that LangChain's .with_retry() is the single source of truth for retry behavior. Layering the provider SDK's retries underneath LangChain's .with_retry() leads to a multiplicative, hard-to-reason-about retry count (SDK retries × LangChain retries), and it makes your backoff timing unpredictable. Pick one layer to own retries — for most teams that's LangChain, since it's provider-agnostic and lets you tune policy consistently.

Combine timeout with retry and fallback for the full picture:

from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic

primary_llm = ChatOpenAI(
    model="gpt-4o",
    temperature=0,
    timeout=20,
    max_retries=0,
).with_retry(
    stop_after_attempt=3,
    wait_exponential_jitter=True,
)

backup_llm = ChatAnthropic(
    model="claude-sonnet-4-5",
    temperature=0,
    timeout=20,
    max_retries=0,
).with_retry(
    stop_after_attempt=3,
    wait_exponential_jitter=True,
)

production_llm = primary_llm.with_fallbacks([backup_llm])

You can also apply a timeout at the LCEL chain level using RunnableConfig, which is useful when the concern isn't a single model call but the entire chain running too long end-to-end (retrieval plus generation plus parsing):

chain = prompt | production_llm | StrOutputParser()

result = chain.invoke(
    {"topic": "circuit breakers"},
    config={"max_concurrency": 5},
)

For a hard wall-clock ceiling around an entire invocation, wrapping the call with Python's standard concurrent.futures timeout (or an async equivalent with asyncio.wait_for) gives you a backstop that isn't dependent on the underlying HTTP client honoring its own timeout correctly:

import asyncio

async def invoke_with_deadline(chain, inputs, deadline_seconds=30):
    try:
        return await asyncio.wait_for(chain.ainvoke(inputs), timeout=deadline_seconds)
    except asyncio.TimeoutError:
        return {"error": "Request exceeded maximum allowed time. Please try again."}

This belt-and-suspenders approach — a per-call timeout on the model client, plus an outer deadline on the whole operation — catches the cases where a single component's timeout configuration doesn't behave the way you expect, which happens more often than you'd hope across different provider SDKs and streaming response modes.

Putting It Together: A Production-Grade Resilient Chain

Here's a complete pattern combining timeout, retry, and fallback into a single resilient chain suitable for a customer-facing feature:

from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnableLambda

# Primary model: fast, cheap, good default
primary = ChatOpenAI(
    model="gpt-4o-mini",
    temperature=0,
    timeout=15,
    max_retries=0,
).with_retry(
    retry_if_exception_type=(Exception,),
    wait_exponential_jitter=True,
    stop_after_attempt=3,
)

# Backup model: different provider, isolates you from a single vendor's outage
backup = ChatAnthropic(
    model="claude-haiku-4-5",
    temperature=0,
    timeout=15,
    max_retries=0,
).with_retry(
    retry_if_exception_type=(Exception,),
    wait_exponential_jitter=True,
    stop_after_attempt=3,
)

def fallback_notice(inputs: dict):
    # Runs only if primary and its retries were exhausted
    print("Falling back to backup model")
    return inputs["input"]

backup_chain = RunnableLambda(fallback_notice) | backup

resilient_model = primary.with_fallbacks(
    [backup_chain],
    exception_key="exception",
)

prompt = ChatPromptTemplate.from_template(
    "You are a support assistant. Answer concisely: {question}"
)

support_chain = prompt | resilient_model | StrOutputParser()

def safe_invoke(question: str) -> str:
    try:
        return support_chain.invoke({"question": question})
    except Exception as exc:
        # Both primary and every fallback failed — degrade gracefully
        return "We're experiencing high demand right now. Please try again shortly."

if __name__ == "__main__":
    print(safe_invoke("Why is my subscription showing as past due?"))

Notice the final try/except around the whole chain. Even with retries and fallbacks, there is always a scenario where everything fails — both providers are down, or your network egress is broken. The application still needs to respond to the user with *something* other than a stack trace. This last-resort catch is not optional; it is the difference between "brief degraded experience" and "the page crashed."

Handling Malformed Output Separately from Transport Errors

A subtlety worth calling out explicitly: retries and fallbacks as described above react to *exceptions* — network errors, timeouts, HTTP error codes. They do not automatically fix a model that returns syntactically invalid JSON to a PydanticOutputParser or a JsonOutputParser. That failure is also an exception (a parsing error), so .with_retry() will technically retry it — but retrying the exact same prompt against the exact same model often reproduces the exact same malformed output, especially at temperature=0.

The better fix for this specific failure mode is to retry with the parser feedback included, which LangChain supports via OutputFixingParser or by re-invoking with the error appended to the prompt:

from langchain_core.output_parsers import PydanticOutputParser
from langchain.output_parsers import OutputFixingParser
from pydantic import BaseModel

class Ticket(BaseModel):
    priority: str
    summary: str

base_parser = PydanticOutputParser(pydantic_object=Ticket)
fixing_parser = OutputFixingParser.from_llm(parser=base_parser, llm=primary)

chain = prompt | primary | fixing_parser

OutputFixingParser catches the parsing failure and sends the malformed output back to the model along with the error message, asking it to correct the format — a fundamentally different (and much more effective) repair strategy than blindly retrying the original request.

Observability: You Can't Fix What You Can't See

None of this matters if you don't know it's happening. Every retry and every fallback should be logged with enough context to tell you, after the fact, whether your resilience layer is a safety net or a load-bearing wall holding up a chronically flaky dependency. At minimum, log: which model handled the request, how many retry attempts it took, whether a fallback fired, and the total latency. If your fallback model is firing on more than a tiny fraction of requests, that's a signal your primary provider or model choice needs attention — not just a case closed by the fallback working as designed.

LangSmith (LangChain's tracing platform) captures retry and fallback behavior automatically if you have tracing enabled, and it's worth turning on for any chain running in production. Watching a spike in fallback invocations on a trace dashboard is a much better way to learn about a provider incident than waiting for a customer complaint.

Common Mistakes to Avoid

Retrying non-idempotent side effects. If your chain calls a tool that charges a credit card or sends an email, wrapping the whole chain in .with_retry() can trigger that side effect multiple times. Scope retries to the LLM call, not the whole pipeline, when side effects are involved.
No jitter on concurrent workloads. If you're running requests in parallel (batch processing, multiple users hitting your API at once), skipping wait_exponential_jitter means failures synchronize into retry storms.
Unbounded retries. Always set stop_after_attempt. An infinite retry loop against a permanently broken endpoint (bad API key, deprecated model) will hang requests indefinitely.
Retrying everything, including auth and validation errors. Scope retry_if_exception_type to genuinely transient failures. Retrying a 401 five times just delays the inevitable and clutters your logs.
Stacking SDK-level and LangChain-level retries. Set max_retries=0 on the underlying client so LangChain's retry policy is the only one running.
Treating fallback models as drop-in equivalents. A cheaper or faster fallback model may produce noticeably different output quality. Test your fallback path deliberately, not just your happy path — users will hit it eventually, and it needs to be good enough to stand on its own.

Wrapping Up

Error handling in LangChain isn't an afterthought bolted on with a generic try/except — it's a first-class part of the framework, expressed through composable primitives that mirror how these failures actually occur in production. .with_retry() handles the transient blips that resolve themselves in a second or two. .with_fallbacks() handles the failures that don't — provider outages, deprecated models, content policy mismatches — by routing to an alternative that can still get the job done. Timeouts make sure neither of those mechanisms gets stuck waiting on a request that will never return. Layered together, with sensible exception scoping and a final catch-all at the edge of your application, they turn "the API call failed" from an incident into a non-event.

If you want to go deeper — building multi-provider routing strategies, designing chains that degrade gracefully under partial outages, and instrumenting production LangChain apps with proper tracing and alerting — that's exactly what we cover hands-on in the LangChain Tutorial 2026 course here at teachyou.ai. We build these resilience patterns from scratch, break them on purpose, and fix them together, so you walk away with error handling instincts you can apply to any chain you build next.