Configuring Timeouts for LLM Calls

An LLM timeout is the maximum time your code waits for a model provider to respond before giving up and raising an error. Getting this wrong is one of the most common causes of hung requests, wasted compute, and angry users in production AI apps. This guide covers how to set connection timeouts, read timeouts, and total timeouts correctly for chat completions, tool calls, and streaming responses, with runnable code for the OpenAI SDK, the Anthropic SDK, and raw HTTP clients.

Why LLM Timeout Handling Is Different From Regular API Timeouts

A typical REST API call to a database or a payments service completes in tens or hundreds of milliseconds. If it takes longer than a second, something is wrong and you should fail fast. LLM calls do not behave that way.

A single chat completion can legitimately take anywhere from under a second (a short answer from a small model) to well over a minute (a long reasoning trace, a large output with tool calls, or a request queued behind provider-side rate limiting). If you copy the timeout defaults you use for your payments API into your LLM client, you will kill requests that were about to succeed, and your users will see errors instead of answers.

The fix is not "use a longer timeout everywhere." It's to separate the different phases of a request, set a timeout per phase, and make the timeout depend on what the request is actually doing (a one-line classification call is not the same shape as a 4,000-token report generation).

Connection Timeouts vs Read Timeouts vs Total Timeouts

Every HTTP client that talks to an LLM provider exposes at least three different timeout knobs, and conflating them is the single biggest source of confusion.

• Connection timeout: how long to wait for the TCP/TLS handshake to complete. This should be short, typically a few seconds. If you can't even open a connection, retrying immediately is almost always the right move, not waiting longer.
• Read timeout (sometimes called "socket timeout"): how long to wait between bytes once the connection is open. This is the one that matters most for LLM calls, because a slow model or a busy provider can go quiet for a while before sending the next chunk.
• Total (or "deadline") timeout: a hard ceiling on the entire request, from the moment you call the client to the moment you get a full response or give up. This is what protects you from a connection that stays technically alive but never finishes.

For LLM calls, treat these as independent settings:

# Conceptual shape, not tied to one SDK
connect_timeout = 5      # seconds to establish the connection
read_timeout = 60        # seconds allowed between chunks of data
total_timeout = 120       # hard ceiling on the whole call

A short connect timeout with a longer read timeout is the combination that avoids both failure modes: dead connections get retried fast, but a model that's genuinely still generating tokens doesn't get killed mid-thought.

Setting Timeouts in the OpenAI SDK

The OpenAI Python and JavaScript SDKs both accept a timeout parameter at the client level and can override it per request. Set a client-level default that matches your typical workload, then override for specific calls that you know will run long.

from openai import OpenAI
import httpx

client = OpenAI(
    api_key="your-api-key",
    timeout=httpx.Timeout(
        connect=5.0,
        read=60.0,
        write=10.0,
        pool=5.0,
    ),
    max_retries=2,
)

# Per-call override for a request you expect to be long
response = client.chat.completions.create(
    model="gpt-4.1",
    messages=[{"role": "user", "content": "Summarize this 40-page report."}],
    timeout=180.0,
)

In JavaScript, the shape is nearly identical:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  timeout: 60_000, // milliseconds
  maxRetries: 2,
});

const response = await client.chat.completions.create(
  {
    model: "gpt-4.1",
    messages: [{ role: "user", content: "Summarize this report." }],
  },
  { timeout: 180_000 }
);

Notice max_retries sits next to the timeout. That matters because the SDK's built-in retry logic runs on top of whatever timeout you set, so a single "logical" call can take up to timeout * (max_retries + 1) in the worst case. Set retries and timeout together, not independently.

Setting Timeouts in the Anthropic SDK

The Anthropic SDK follows the same pattern: a client-level default, an optional per-request override, and built-in retries that compound with whatever timeout you configure.

import anthropic

client = anthropic.Anthropic(
    api_key="your-api-key",
    timeout=60.0,
    max_retries=2,
)

message = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Draft a release note for this feature."}],
    timeout=90.0,  # override for this specific call
)

For requests that generate long output or invoke multiple tool calls in sequence, don't try to guess a single "safe" number. Instead, size the timeout to the request: a short classification prompt with max_tokens=20 should have a tight timeout, while a request with max_tokens=8000 and tool use enabled needs real headroom.

def timeout_for_request(max_tokens: int, uses_tools: bool) -> float:
    base = 15.0
    per_token = 0.02  # tune this against your own p95 latency, don't hardcode a guess
    tool_overhead = 30.0 if uses_tools else 0.0
    return base + (max_tokens * per_token) + tool_overhead

This turns timeout configuration into a function of the request shape instead of a magic constant sprinkled across the codebase.

Setting Timeouts With Raw HTTP Clients

Not every integration goes through an official SDK. If you're calling a provider's REST endpoint directly, or building an internal proxy in front of multiple providers, you need to set timeouts on the underlying HTTP client yourself.

With Python's httpx:

import httpx

timeout = httpx.Timeout(connect=5.0, read=60.0, write=10.0, pool=5.0)

with httpx.Client(timeout=timeout) as http_client:
    response = http_client.post(
        "https://api.example-llm-provider.com/v1/chat/completions",
        json={"model": "your-model", "messages": [{"role": "user", "content": "Hi"}]},
        headers={"Authorization": "Bearer your-api-key"},
    )
    response.raise_for_status()

With Node's built-in fetch and AbortController, since fetch has no native timeout parameter:

async function callLLM(payload, timeoutMs = 60000) {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), timeoutMs);

  try {
    const res = await fetch("https://api.example-llm-provider.com/v1/chat/completions", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: `Bearer ${process.env.LLM_API_KEY}`,
      },
      body: JSON.stringify(payload),
      signal: controller.signal,
    });
    if (!res.ok) throw new Error(`LLM request failed: ${res.status}`);
    return await res.json();
  } finally {
    clearTimeout(timer);
  }
}

AbortController gives you a total timeout, not a separate connect/read split. If you need that finer granularity in Node, reach for undici's Agent with connectTimeout and bodyTimeout set independently, which mirrors the httpx.Timeout behavior above.

Streaming Changes the Timeout Math

Streaming responses (stream=True in most SDKs) invert the usual timeout problem. Instead of one long wait for a single response, you get a sequence of small chunks over an open connection. A total timeout that's sized for a non-streaming call will often be wrong here, because what actually matters is the gap between chunks, not the total wall-clock time of the whole stream.

import time

def stream_with_chunk_timeout(stream, max_gap_seconds=15):
    last_chunk_time = time.monotonic()
    for chunk in stream:
        now = time.monotonic()
        if now - last_chunk_time > max_gap_seconds:
            raise TimeoutError("No data received within the chunk timeout window")
        last_chunk_time = now
        yield chunk

This pattern, sometimes called an "idle timeout," is the right tool for streaming: it lets a long-running generation continue indefinitely as long as tokens keep arriving, but kills the connection the moment the stream goes silent. Pair it with a separate hard ceiling (say, five minutes) as a backstop against a stream that trickles one token every ten seconds forever without technically going idle.

Retry Logic and Backoff Around Timeouts

A timeout by itself is not a resilience strategy, it's just a failure signal. What you do after the timeout fires is what actually determines whether your users see an error or a slightly slower successful response.

import time
import random

def call_with_retries(fn, max_attempts=3, base_delay=1.0):
    last_error = None
    for attempt in range(max_attempts):
        try:
            return fn()
        except TimeoutError as e:
            last_error = e
            if attempt == max_attempts - 1:
                break
            delay = base_delay * (2 ** attempt) + random.uniform(0, 0.5)
            time.sleep(delay)
    raise last_error

A few rules that matter more than the exact backoff formula:

• Only retry idempotent calls automatically. If the LLM call triggers a side effect (sending an email, charging a card via a tool call), don't blindly retry on timeout, because the first attempt may have actually succeeded server-side even though you never got the response.
• Cap total retry time, not just retry count. Three retries at exponential backoff can still add up to minutes. Track elapsed time against a wall-clock budget for the whole operation, not just per-attempt.
• Distinguish a timeout from a rate limit. A 429 with a Retry-After header should honor that header instead of your generic backoff, a plain timeout should not.

Per-Route Timeout Budgets in a Multi-Agent System

If you're building anything with multiple LLM calls chained together (an agent loop, a retrieval-then-generate pipeline, a multi-step tool-calling workflow), a single global timeout per call is not enough. You need a budget that's allocated across the whole chain, or one slow step silently eats the time the rest of the pipeline needed.

class TimeoutBudget:
    def __init__(self, total_seconds: float):
        self.deadline = time.monotonic() + total_seconds

    def remaining(self) -> float:
        return max(0.0, self.deadline - time.monotonic())

    def check(self):
        if self.remaining() <= 0:
            raise TimeoutError("Timeout budget exhausted before pipeline completed")


def run_pipeline(user_query: str):
    budget = TimeoutBudget(total_seconds=45)

    budget.check()
    retrieved_docs = retrieve(user_query, timeout=min(10, budget.remaining()))

    budget.check()
    plan = call_llm_for_plan(user_query, retrieved_docs, timeout=min(15, budget.remaining()))

    budget.check()
    final_answer = call_llm_for_answer(plan, timeout=budget.remaining())

    return final_answer

Each step draws down from the same shared clock instead of getting its own independent allowance. This is what actually keeps a five-step agent loop bounded, rather than five independently-reasonable 30-second timeouts adding up to two and a half minutes of worst-case latency.

Circuit Breakers and Fallback Models

Timeouts tell you a single request failed. A circuit breaker tells you the provider itself is degraded, and stops sending it new requests for a cooldown window instead of timing out on every single call while things are unhealthy.

class CircuitBreaker:
    def __init__(self, failure_threshold=5, cooldown_seconds=30):
        self.failure_threshold = failure_threshold
        self.cooldown_seconds = cooldown_seconds
        self.failures = 0
        self.opened_at = None

    def is_open(self) -> bool:
        if self.opened_at is None:
            return False
        if time.monotonic() - self.opened_at > self.cooldown_seconds:
            self.opened_at = None
            self.failures = 0
            return False
        return True

    def record_failure(self):
        self.failures += 1
        if self.failures >= self.failure_threshold:
            self.opened_at = time.monotonic()

    def record_success(self):
        self.failures = 0
        self.opened_at = None

Wire this in front of your LLM calls and pair it with a fallback: when the primary model's circuit is open, route to a secondary model or provider instead of piling up timeouts against a service that's already struggling. This is also where per-route timeout budgets pay off, because a fallback call needs its own remaining time from the budget, not a fresh full allowance.

Monitoring Timeout Rates in Production

A timeout value that looked correct in testing can silently go stale as your prompts, models, or traffic patterns change. Track these as first-class metrics, not just log lines:

• Timeout rate as a percentage of total LLM calls, broken out by route and by model.
• p50 / p95 / p99 latency per route, so you can see the gap between "typical" and "your timeout" shrinking over time.
• Retry exhaustion rate: how often all retries fail, which tells you whether your timeout is genuinely too short versus the provider being down.
• Chunk-gap distribution for streaming calls, so an idle timeout can be tuned against real data instead of a guess.

import logging

logger = logging.getLogger("llm_client")

def log_timeout_event(route: str, model: str, elapsed: float, configured_timeout: float):
    logger.warning(
        "llm_timeout",
        extra={
            "route": route,
            "model": model,
            "elapsed_seconds": elapsed,
            "configured_timeout_seconds": configured_timeout,
        },
    )

Feed this into whatever dashboarding you already use. The specific tool matters less than the discipline of reviewing timeout rate weekly, the same way you'd review error rate or p95 latency for any other backend dependency.

Common Mistakes When Configuring LLM Timeouts

• Using one timeout value for every route. A one-line intent classifier and a long-form document generator have completely different latency profiles. Size the timeout to the request.
• Setting the timeout so high it masks real failures. If your timeout is ten minutes, users will sit on a hung request for ten minutes before your code even attempts a retry. A tighter timeout with a good retry strategy usually beats a single huge one.
• Forgetting that retries multiply the effective timeout. Three retries at a 60-second timeout is a 180-second worst case, not a 60-second one. Communicate the effective ceiling to whatever is calling your service, including any upstream load balancer or gateway timeout.
• Not separating connect timeout from read timeout. A short connect timeout with a long read timeout fails fast on dead connections while still giving a live one room to finish.
• Ignoring streaming's different failure mode. A total-timeout approach on a stream either kills legitimate long generations or lets a stalled stream hang far too long. Use an idle/chunk timeout instead.
• No shared budget across multi-step pipelines. Independent per-call timeouts in an agent loop can add up to a worst case far worse than what any single step's timeout suggests.
• No visibility into timeout rate. If you can't see how often calls are timing out per route and per model, you're tuning by anecdote, not data.

FAQ

What's a reasonable default read timeout for a single-turn LLM chat call? There's no universal number, since it depends on model size, expected output length, and provider load. Start by measuring your own p95 latency for that specific route under real traffic, then set the read timeout at roughly two to three times that figure, and revisit it once you have a few weeks of production data.

Should I set the timeout on the client or per request? Do both. Set a sane client-level default that covers the majority of your calls, and override it per request for routes you know will run long, such as document summarization or multi-step tool calls. This avoids scattering one-off magic numbers across the codebase while still handling outlier routes correctly.

Does a timeout error mean the LLM provider actually failed? Not necessarily. The request may still be processing on the provider's side even after your client gives up waiting. This is exactly why blind retries are risky for calls with side effects, and why idempotency matters when you design retry logic around timeouts.

How do I choose between a total timeout and an idle timeout for streaming? Use an idle timeout as the primary control, since it correctly allows long but actively-progressing generations to finish. Add a total timeout as a backstop so a stream that never fully stalls, but also never finishes, still gets cut off eventually.

Do timeouts differ between chat completions and tool-calling or agentic workflows? Yes, and this is where most production issues start. A single tool-calling turn can involve the model reasoning, calling a tool, waiting on that tool's result, then reasoning again, all before you see a final response. Size timeouts for these routes around the full round trip, or better, use a shared timeout budget across the whole chain instead of one flat number per call.

Is it safe to just set a very long timeout and rely on retries for everything else? No. A very long timeout means your users, or any upstream service calling yours, wait the full duration before anything happens, including your retry logic kicking in. A tighter, well-measured timeout combined with fast retries and a circuit breaker gives you both quick failure detection and resilience, which a single long timeout cannot provide on its own.