Rate Limiting Strategies for LLM APIs

LLM rate limiting is the set of controls that keep your app inside a provider's request and token budgets while staying fast under load. In practice it means two jobs: reacting correctly when a provider returns a 429, and proactively pacing your own traffic so you rarely hit one. This guide covers both, with runnable code you can drop into a Python or Node service today.

If you ship anything on top of an LLM API, rate limits are not an edge case, they are the normal operating condition at scale. A single burst of users, a retry storm, or one badly written loop can exhaust your quota and take the whole feature down. The good news is that the failure modes are well understood, and a few small patterns handle almost all of them.

Why LLM rate limiting is different

Most API rate limiting counts one thing: requests per second. LLM APIs count at least two, and often more:

• Requests per minute (RPM): how many calls you can make.
• Tokens per minute (TPM): how many tokens, input plus output, you can process.
• Sometimes concurrent requests, tokens per day, or input-token and output-token budgets tracked separately.

TPM is the one that surprises people. You can be nowhere near your request limit and still get throttled because a handful of long-context calls burned your token budget. A single request stuffing 100k tokens of context can cost more of your minute than a hundred short calls. So any real strategy has to account for tokens, not just call count.

The second difference is variable cost per call. A classic web endpoint is roughly constant cost. An LLM call's cost depends on prompt length and how much the model generates, and you do not know the output length until the response finishes. That uncertainty is why good LLM rate limiting estimates cost up front and reconciles after.

Third, latency is high and spiky. A call can take anywhere from 300 milliseconds to 60 seconds for a long generation. That means retries are expensive and queues back up fast. You cannot treat an LLM call like a cheap idempotent GET.

Read the response headers first

Before you build anything, use what the provider already tells you. Most major LLM APIs, including Anthropic's and OpenAI's, return rate-limit headers on every response. The exact names vary, but you will typically see remaining requests, remaining tokens, and a reset time. On a 429 you usually also get a retry-after header telling you how many seconds to wait.

The single highest-value change most teams can make is: when you get a 429, honor retry-after instead of guessing. Here is the minimal version in Python using httpx.

import time
import httpx

def call_with_retry_after(client, url, headers, payload, max_retries=5):
    for attempt in range(max_retries):
        resp = client.post(url, headers=headers, json=payload)
        if resp.status_code != 429:
            resp.raise_for_status()
            return resp.json()

        # Provider told us exactly how long to wait. Trust it.
        retry_after = resp.headers.get("retry-after")
        wait = float(retry_after) if retry_after else 2 ** attempt
        time.sleep(wait)

    raise RuntimeError("Exhausted retries after repeated 429s")

Reading the proactive headers lets you slow down before you get blocked. If anthropic-ratelimit-tokens-remaining (or the equivalent for your provider) is close to zero, pause new work for a moment rather than firing off calls that will bounce.

Exponential backoff with jitter

retry-after is not always present, and some transient errors (500, 503, connection resets) are not 429s at all. For those you need backoff. The rule that matters: always add jitter. Without jitter, every client that failed at the same instant retries at the same instant, and you get a synchronized thundering herd that re-triggers the limit.

Full jitter is the version to use. Instead of waiting exactly base * 2 ** attempt, you wait a random amount between zero and that ceiling.

import random
import time

def backoff_with_jitter(attempt, base=1.0, cap=60.0):
    ceiling = min(cap, base * (2 ** attempt))
    return random.uniform(0, ceiling)

def call_resilient(do_call, is_retryable, max_retries=6):
    for attempt in range(max_retries):
        try:
            return do_call()
        except Exception as err:
            if not is_retryable(err) or attempt == max_retries - 1:
                raise
            time.sleep(backoff_with_jitter(attempt))

Cap the maximum wait so a single request does not hang for ten minutes. Cap the retry count so you fail fast enough to return an error to the user instead of holding a connection open forever. Six retries with a 60-second cap is a reasonable default for background work; for interactive requests, two or three retries with a lower cap keeps latency sane.

Only retry idempotent or safe-to-repeat work. An LLM completion is usually safe to retry because you discard the failed attempt. But if your call has side effects (writes to a database, sends an email, charges a card), guard those separately so a retry does not double-execute them.

Token bucket: pace your own traffic

Reacting to 429s is table stakes. The next level is not hitting them in the first place. The token bucket algorithm is the standard tool. You have a bucket that refills at a steady rate up to some capacity. Each request takes tokens from the bucket. If the bucket is empty, the request waits. This smooths bursts while allowing short spikes up to the bucket capacity.

For LLM APIs you run two buckets: one for requests per minute, one for tokens per minute. Here is a small thread-safe token bucket in Python.

import threading
import time

class TokenBucket:
    def __init__(self, rate_per_sec, capacity):
        self.rate = rate_per_sec
        self.capacity = capacity
        self.tokens = capacity
        self.updated = time.monotonic()
        self.lock = threading.Lock()

    def _refill(self):
        now = time.monotonic()
        elapsed = now - self.updated
        self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
        self.updated = now

    def acquire(self, amount=1, timeout=30.0):
        deadline = time.monotonic() + timeout
        while True:
            with self.lock:
                self._refill()
                if self.tokens >= amount:
                    self.tokens -= amount
                    return True
            if time.monotonic() >= deadline:
                return False
            time.sleep(0.02)

Wire it up with a request bucket and a token bucket. If your limit is 500 RPM and 100,000 TPM, set the request bucket to a rate of about 8.3 per second, and the token bucket to about 1,666 per second, with capacities matching your allowed burst.

# 500 requests/min, 100k tokens/min
req_bucket = TokenBucket(rate_per_sec=500 / 60, capacity=500)
tok_bucket = TokenBucket(rate_per_sec=100_000 / 60, capacity=100_000)

def guarded_call(do_call, estimated_tokens):
    if not req_bucket.acquire(1):
        raise RuntimeError("Local request budget exhausted, shed load")
    if not tok_bucket.acquire(estimated_tokens):
        raise RuntimeError("Local token budget exhausted, shed load")
    return do_call()

The key detail: you must estimate token cost before the call. Count the input tokens with the provider's tokenizer, then add a conservative estimate for the output based on your max_tokens setting. After the response comes back with real usage numbers, reconcile the difference by returning or removing tokens from the bucket. Over-estimating slightly is safer than under-estimating, because under-estimating pushes you into provider 429s.

Set your local buckets slightly below the provider's actual limits, say 85 to 90 percent. That headroom absorbs estimation error and clock skew, and keeps you off the provider's hard wall where behavior gets unpredictable.

Distributed rate limiting across replicas

A single-process token bucket breaks the moment you run more than one instance. Ten replicas each thinking they own the full 500 RPM will collectively fire 5,000 RPM and get throttled instantly. You need a shared counter, and Redis is the usual answer.

The clean approach is a Redis-backed sliding window or token bucket, executed as a Lua script so the check-and-decrement is atomic. Here is a token-bucket Lua script you can run with EVAL.

-- KEYS[1] = bucket key
-- ARGV[1] = rate per second, ARGV[2] = capacity
-- ARGV[3] = now (seconds), ARGV[4] = requested amount
local data = redis.call("HMGET", KEYS[1], "tokens", "ts")
local tokens = tonumber(data[1])
local ts = tonumber(data[2])
local rate = tonumber(ARGV[1])
local capacity = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local amount = tonumber(ARGV[4])

if tokens == nil then
  tokens = capacity
  ts = now
end

local elapsed = math.max(0, now - ts)
tokens = math.min(capacity, tokens + elapsed * rate)

local allowed = 0
if tokens >= amount then
  tokens = tokens - amount
  allowed = 1
end

redis.call("HMSET", KEYS[1], "tokens", tokens, "ts", now)
redis.call("EXPIRE", KEYS[1], 120)
return allowed

Calling it from Python:

import time
import redis

r = redis.Redis()
with open("token_bucket.lua") as f:
    bucket_script = r.register_script(f.read())

def try_acquire(key, rate, capacity, amount):
    return bucket_script(
        keys=[key],
        args=[rate, capacity, time.time(), amount],
    ) == 1

# usage
if not try_acquire("llm:tokens", 100_000 / 60, 100_000, estimated_tokens):
    raise RuntimeError("Cluster token budget exhausted")

Because the whole read-modify-write runs inside one Lua script, Redis executes it atomically and no two replicas can double-spend the same tokens. Keep a short EXPIRE so idle keys clean themselves up. If Redis itself becomes a bottleneck or a single point of failure, fall back to a local bucket sized at provider_limit / replica_count so you degrade instead of crashing.

Queue, prioritize, and shed load

When demand exceeds your provider budget, you have three options: queue the work, drop it, or push back on the caller. Pick per use case.

• Background and batch jobs: queue them. A worker pool that pulls from a queue and respects the token bucket naturally paces itself. Add a max queue depth so a runaway producer cannot exhaust memory.
• Interactive user requests: do not queue for 45 seconds, that feels broken. Shed load fast with a clear 429 or a "we are busy, try again" message, and surface a retry-after to the client.
• Mixed traffic: run priority queues. Paid users or latency-sensitive paths jump the line; free-tier or bulk work waits.

A simple asyncio worker pool in Python that respects concurrency and a token bucket:

import asyncio

async def worker(name, queue, semaphore, call_llm):
    while True:
        job = await queue.get()
        try:
            async with semaphore:
                await call_llm(job)
        except Exception as err:
            print(f"{name} failed job {job['id']}: {err}")
        finally:
            queue.task_done()

async def run_pool(jobs, call_llm, concurrency=8):
    queue = asyncio.Queue(maxsize=1000)
    semaphore = asyncio.Semaphore(concurrency)
    workers = [
        asyncio.create_task(worker(f"w{i}", queue, semaphore, call_llm))
        for i in range(concurrency)
    ]
    for job in jobs:
        await queue.put(job)
    await queue.join()
    for w in workers:
        w.cancel()

The semaphore caps concurrent in-flight calls, which matters because many providers also limit concurrency, and because unbounded concurrency will blow your token budget in one burst. Bounded concurrency plus a token bucket is a strong combination.

Circuit breakers for provider outages

Rate limits are transient. Outages are not. If a provider is returning 529s or timing out on every call, retrying just wastes time and money and makes your latency worse. A circuit breaker stops the bleeding: after a threshold of consecutive failures, it "opens" and fails fast for a cooldown period, then lets a trickle of test calls through to check if the provider recovered.

import time

class CircuitBreaker:
    def __init__(self, threshold=5, cooldown=30.0):
        self.threshold = threshold
        self.cooldown = cooldown
        self.failures = 0
        self.opened_at = None

    def allow(self):
        if self.opened_at is None:
            return True
        if time.monotonic() - self.opened_at >= self.cooldown:
            return True  # half-open: allow one trial call
        return False

    def record_success(self):
        self.failures = 0
        self.opened_at = None

    def record_failure(self):
        self.failures += 1
        if self.failures >= self.threshold:
            self.opened_at = time.monotonic()

When the breaker is open, serve a fallback: a cached answer, a smaller or different model, a degraded non-LLM response, or an honest error. Combining a circuit breaker with a secondary model or provider is how mature systems ride out an outage without going fully dark.

A Node example, end to end

Not everyone runs Python. Here is the same core idea in Node, combining retry-after handling with full jitter, using the built-in fetch.

function jitteredDelay(attempt, base = 1000, cap = 60000) {
  const ceiling = Math.min(cap, base * 2 ** attempt);
  return Math.random() * ceiling;
}

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function callLLM(url, headers, body, maxRetries = 5) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const resp = await fetch(url, {
      method: "POST",
      headers,
      body: JSON.stringify(body),
    });

    if (resp.status !== 429 && resp.status < 500) {
      if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
      return resp.json();
    }

    const retryAfter = resp.headers.get("retry-after");
    const wait = retryAfter
      ? Number(retryAfter) * 1000
      : jitteredDelay(attempt);
    await sleep(wait);
  }
  throw new Error("Exhausted retries");
}

Same three ideas: prefer retry-after, fall back to jittered backoff, cap the retries. For pacing across a Node cluster, use the same Redis Lua script; the language on the client does not matter.

Putting it together

A production LLM client layers these controls in order:

1. Local token bucket and request bucket, set to about 85 percent of the provider limit, to pace outbound traffic.
2. A shared Redis bucket when you run more than one replica, so the cluster respects one global budget.
3. A bounded worker pool or semaphore capping concurrent calls.
4. retry-after handling first, then exponential backoff with full jitter for other transient errors.
5. A circuit breaker with a fallback for real outages.
6. Load shedding with a clear signal to interactive callers when the queue is full.

You do not need all six on day one. Start with retry-after plus jittered backoff, because that alone prevents the worst retry storms. Add the token bucket when you start seeing 429s under normal load. Add Redis when you scale past one instance. Add the circuit breaker the first time a provider outage teaches you why you need it.

Instrument everything. Log every 429, every retry, every circuit-breaker trip, and your observed TPM and RPM. You cannot tune limits you cannot see, and the numbers you assume on day one will not match what you observe in production.

FAQ

What is the difference between RPM and TPM limits?

RPM is requests per minute, a count of how many calls you make. TPM is tokens per minute, a count of input plus output tokens processed. You can hit either one independently. A few large-context calls can blow your TPM while your RPM stays low, so any serious LLM rate limiting strategy tracks both budgets separately.

Should I always retry on a 429?

Retry, but with limits. Honor the retry-after header when present, cap the number of retries (two or three for interactive, more for background), and cap the maximum wait. Always add jitter so retrying clients do not synchronize into a thundering herd. And never blindly retry calls with side effects.

How do I estimate tokens before a call?

Use the provider's tokenizer to count the input tokens exactly, then add an estimate of output tokens based on your max_tokens cap. Over-estimate slightly to stay conservative. After the response returns real usage numbers, reconcile the difference back into your token bucket so your accounting stays accurate.

Do I need Redis for rate limiting?

Only if you run more than one instance of your service. A single process can use an in-memory token bucket. Once you scale horizontally, each replica needs to share one global budget, and a Redis token bucket executed as an atomic Lua script is the standard, reliable way to do it.

What is the difference between a rate limiter and a circuit breaker?

A rate limiter paces traffic to stay within a budget, it slows you down. A circuit breaker detects that a dependency is failing and stops calling it entirely for a cooldown, it fails fast. Rate limiters handle normal throttling; circuit breakers handle outages. Production systems use both.

How much headroom should I leave below the provider limit?

Roughly 10 to 15 percent. Set local buckets to 85 to 90 percent of the provider's stated limit. That margin absorbs token estimation error, clock skew across replicas, and the small burst that happens when several workers wake at once. Running right at the wall makes provider behavior unpredictable.

Can I just increase my rate limits instead?

Sometimes, and you should ask your provider for higher tiers as you grow. But higher limits do not remove the need for these patterns. Retry storms, token spikes, and outages happen at every scale, and a client that respects backpressure is cheaper and more reliable regardless of how high the ceiling is.