Rate Limiting and Backpressure for AI Agent Systems

Your agent worked perfectly in the demo. Then you shipped it, ten users showed up at once, and within an hour you were staring at a wall of 429s, a provider bill that jumped overnight, and a support channel full of "the assistant just hangs." Nothing about your prompts changed. What changed is that you went from one caller making one request at a time to a system where agents call tools, tools call APIs, and APIs call other APIs — and every one of those hops can be a bottleneck.

Rate limiting and backpressure are the least glamorous parts of agent engineering and the ones that decide whether your system survives contact with real traffic. This article is about the mechanics: where limits actually bite in an agent pipeline, how to implement token bucket and sliding window limiters, how to propagate backpressure instead of just eating errors, and how to design retry and queueing behavior that keeps a multi-agent system stable when everything upstream is straining.

Why agent systems break rate limits differently than normal apps

A traditional web app has a fairly predictable request shape: a user clicks something, one request goes out, one response comes back. You can rate limit at the edge and mostly be done with it.

Agent systems don't behave like that. A single user turn can fan out into:

• One or more LLM calls, sometimes chained (planner call, then executor call, then a critique call)
• Multiple tool calls per LLM turn, often in parallel
• Retries on any of the above when a call times out or errors
• Sub-agents that themselves make LLM and tool calls

That means the "1 user action" you're rate limiting for might actually be 15-40 downstream calls, and the multiplier is not fixed — it depends on how many reasoning steps the model decides to take. A user with a hard question can single-handedly generate the request volume of fifty simple ones. This is why agent systems hit provider rate limits at what looks like low user counts: you're not rate limited on users, you're rate limited on the fan-out those users trigger.

The other difference is that agent calls are usually not idempotent-safe to blindly retry. If a tool call already charged a customer, wrote a database row, or sent an email, retrying it after a timeout can double the side effect. Backpressure design in agent systems has to account for this — it's not enough to slow down, you also need to know what's safe to redo.

Where limits actually live in an agent pipeline

Before writing any limiter code, map out every place a limit can be hit. In a typical agent stack there are at least four distinct layers, and conflating them is the most common design mistake:

1. Provider-side rate limits — requests per minute (RPM) and tokens per minute (TPM) enforced by your LLM provider. These are usually tiered by account/spend level and are the hardest limit because you can't negotiate them in real time.
2. Your own ingress limits — how many requests per user, per API key, or per IP your service will accept, independent of what the LLM provider allows.
3. Tool and downstream API limits — the search API, the vector database, the payment gateway, the internal microservice your agent calls as a tool. Each has its own ceiling, often lower than you'd expect.
4. Concurrency limits inside your own process — how many agent loops, browser instances, or subprocess workers you can run at once before you exhaust memory, file descriptors, or CPU.

A system that only rate limits at layer 2 (ingress) will still fall over at layer 1 or 3 the moment traffic clusters. You need a limiter, or at least an awareness of headroom, at every layer that can say no.

# A minimal mental model: each layer needs its own budget tracker
LIMITS = {
    "provider_rpm": 500,       # what the LLM API allows per minute
    "provider_tpm": 150_000,   # tokens per minute, often the tighter constraint
    "ingress_per_user_rpm": 20,
    "tool_search_api_rps": 5,
    "max_concurrent_agent_loops": 25,
}

Tokens per minute is worth calling out specifically, because teams often rate limit only on request count and get blindsided by TPM. A single agent call with a long context window and a large tool schema can consume more of your token budget than ten short calls combined. If you're not tracking estimated tokens per request before you send it, you're flying blind on the limit most likely to bite first.

Token bucket: the workhorse algorithm

The token bucket algorithm is the standard choice for rate limiting because it naturally allows bursts up to a cap while enforcing a steady average rate — which matches how agent traffic actually behaves (bursty tool calls, then quiet reasoning time).

The idea: a bucket holds tokens, refills at a fixed rate, and every request consumes one or more tokens. If the bucket is empty, the request waits or is rejected.

import time
import threading

class TokenBucket:
    def __init__(self, capacity: float, refill_rate: float):
        """
        capacity: max tokens the bucket can hold (burst size)
        refill_rate: tokens added per second (sustained rate)
        """
        self.capacity = capacity
        self.refill_rate = refill_rate
        self.tokens = capacity
        self.last_refill = time.monotonic()
        self.lock = threading.Lock()

    def _refill(self):
        now = time.monotonic()
        elapsed = now - self.last_refill
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
        self.last_refill = now

    def try_consume(self, amount: float = 1) -> bool:
        with self.lock:
            self._refill()
            if self.tokens >= amount:
                self.tokens -= amount
                return True
            return False

    def wait_time(self, amount: float = 1) -> float:
        with self.lock:
            self._refill()
            if self.tokens >= amount:
                return 0.0
            deficit = amount - self.tokens
            return deficit / self.refill_rate


# Usage: limiting calls to an LLM provider at 8 requests/sec, burst of 20
llm_limiter = TokenBucket(capacity=20, refill_rate=8)

def call_llm_with_limit(prompt: str):
    if not llm_limiter.try_consume(1):
        wait = llm_limiter.wait_time(1)
        time.sleep(wait)
        llm_limiter.try_consume(1)
    return "response from provider"  # actual call goes here

Notice the bucket is keyed on a single dimension here (requests). In practice you want two buckets running side by side for LLM calls — one for request count, one for estimated tokens — and you gate on whichever one is tighter for that specific call.

def estimate_tokens(prompt: str, tool_schema_size: int = 0) -> int:
    # Rough heuristic: ~4 characters per token for English text
    return len(prompt) // 4 + tool_schema_size

request_limiter = TokenBucket(capacity=20, refill_rate=8)
token_limiter = TokenBucket(capacity=25_000, refill_rate=2_500)  # per second

def call_llm_gated(prompt: str, tool_schema_size: int = 0):
    est_tokens = estimate_tokens(prompt, tool_schema_size)

    req_wait = 0.0 if request_limiter.try_consume(1) else request_limiter.wait_time(1)
    tok_wait = 0.0 if token_limiter.try_consume(est_tokens) else token_limiter.wait_time(est_tokens)

    delay = max(req_wait, tok_wait)
    if delay > 0:
        time.sleep(delay)
        request_limiter.try_consume(1)
        token_limiter.try_consume(est_tokens)

    return "response from provider"

This two-dimensional gating is the single highest-leverage change most teams can make to their rate limiting code, because TPM exhaustion is what actually produces the mysterious 429s that request-count limiters miss entirely.

Sliding window limiting for per-user fairness

Token buckets are great for protecting a shared resource like your LLM provider quota. But when you need to enforce per-user fairness — so one power user's agent session doesn't starve everyone else — a sliding window counter is often a cleaner fit, especially if you're backing it with Redis in a distributed deployment.

import time

class SlidingWindowLimiter:
    """
    Uses a sorted set per key to track request timestamps.
    Works against Redis (ZADD/ZREMRANGEBYSCORE/ZCARD) or in-memory for a single process.
    """
    def __init__(self, redis_client, max_requests: int, window_seconds: int):
        self.redis = redis_client
        self.max_requests = max_requests
        self.window_seconds = window_seconds

    def allow(self, key: str) -> bool:
        now = time.time()
        window_start = now - self.window_seconds

        pipe = self.redis.pipeline()
        pipe.zremrangebyscore(key, 0, window_start)
        pipe.zcard(key)
        pipe.zadd(key, {str(now): now})
        pipe.expire(key, self.window_seconds)
        results = pipe.execute()

        current_count = results[1]
        return current_count < self.max_requests


# Usage: cap each user at 20 agent turns per 60 seconds
limiter = SlidingWindowLimiter(redis_client=redis_conn, max_requests=20, window_seconds=60)

def handle_agent_turn(user_id: str, message: str):
    if not limiter.allow(f"agent_turn:{user_id}"):
        raise RateLimitExceeded(
            "You're sending requests too quickly. Please wait a moment."
        )
    # proceed with agent invocation

The reason sliding window beats a naive fixed window (e.g., "reset counter every minute on the clock") is that fixed windows let a user send double their quota right at the boundary — 20 requests at 0:59 and another 20 at 1:00 is 40 requests in two seconds. Sliding window smooths that out by always looking at the trailing N seconds, not calendar-aligned buckets.

For multi-instance deployments, doing this in Redis rather than in-process memory matters — otherwise each instance enforces its own separate limit and a user routed across instances can multiply their effective quota by your instance count.

Backpressure: what to do when the limiter says no

Rate limiting tells you when you're over budget. Backpressure is what you do about it — and this is where most agent systems get sloppy. The lazy answer is "retry with exponential backoff and hope." That works for transient network blips. It does not work as a strategy for sustained overload, because if every caller backs off and retries, you just get a synchronized retry storm a few seconds later, which is often worse than the original spike.

A backpressure strategy for agent systems needs at least these three components:

• Admission control — decide whether to accept new work at all, before you start burning resources on it. If your queue depth or concurrent-agent-loop count is already past a threshold, reject or queue new requests immediately rather than starting them and failing partway through.
• Priority-aware shedding — not all agent work is equal. A background summarization job can wait; a user actively watching a chat stream cannot. When you must shed load, shed the lowest-priority work first.
• Explicit signaling to the caller — tell the upstream caller (a user, another agent, a scheduler) that you're under pressure, with a concrete retry-after hint, rather than letting them guess from a generic error.

import queue
import time
from dataclasses import dataclass, field
from enum import IntEnum

class Priority(IntEnum):
    LOW = 0
    NORMAL = 1
    INTERACTIVE = 2

@dataclass(order=True)
class AgentJob:
    priority: int
    submitted_at: float = field(compare=False)
    payload: dict = field(compare=False)

class BackpressureQueue:
    def __init__(self, max_queue_size: int, max_concurrent: int):
        self.q = queue.PriorityQueue(maxsize=max_queue_size)
        self.max_concurrent = max_concurrent
        self.in_flight = 0
        self.lock = threading.Lock()

    def submit(self, payload: dict, priority: Priority = Priority.NORMAL):
        job = AgentJob(priority=-priority, submitted_at=time.time(), payload=payload)
        try:
            self.q.put_nowait(job)
        except queue.Full:
            # Admission control: refuse instead of accepting work we can't do
            raise SystemOverloaded(
                retry_after_seconds=2,
                reason="Agent queue is full, shedding new low-priority work first"
            )

    def can_dispatch(self) -> bool:
        with self.lock:
            return self.in_flight < self.max_concurrent

    def dispatch_next(self):
        if not self.can_dispatch():
            return None
        job = self.q.get()
        with self.lock:
            self.in_flight += 1
        return job

    def mark_done(self):
        with self.lock:
            self.in_flight -= 1

The SystemOverloaded exception here isn't just a 500 — it carries a retry_after_seconds value that gets surfaced to the caller, whether that's a UI showing "please wait a moment" or another agent that reschedules the sub-task instead of hammering the same endpoint again immediately.

If you're fronting this with an HTTP API, the equivalent is returning a proper 429 Too Many Requests with a Retry-After header rather than a generic 500. Callers, including other agents in a multi-agent system, should treat 429 and 503 as fundamentally different signals: 429 means "you specifically are over budget, back off," while 503 means "the whole system is struggling, back off harder and maybe try a different path."

Retry strategy: exponential backoff with jitter, and knowing what not to retry

Exponential backoff is well understood, but two details are where implementations usually go wrong: missing jitter, and retrying things that aren't safe to retry.

import random
import time

def call_with_backoff(fn, max_retries: int = 5, base_delay: float = 0.5, max_delay: float = 30.0):
    """
    Exponential backoff with full jitter (per AWS's well-tested formula).
    Only retries errors explicitly marked as retryable.
    """
    for attempt in range(max_retries):
        try:
            return fn()
        except RetryableError as e:
            if attempt == max_retries - 1:
                raise
            capped = min(max_delay, base_delay * (2 ** attempt))
            sleep_time = random.uniform(0, capped)
            time.sleep(sleep_time)
        except NonRetryableError:
            raise  # e.g., a tool call that already had a side effect
    raise RuntimeError("unreachable")


class RetryableError(Exception):
    pass

class NonRetryableError(Exception):
    pass

Full jitter — picking a random delay between zero and the capped exponential value, rather than sleeping the exact computed value — matters more than people expect. Without it, every client that got rate-limited at the same moment backs off on the same schedule and slams the server again in lockstep on the next attempt. Jitter spreads those retries out so they don't reconverge into another spike.

The retryable-vs-not distinction is where agent-specific judgment comes in. A timeout on a read-only tool call (a search, a lookup) is almost always safe to retry. A timeout on a tool call that writes data, sends a message, or moves money is not safe to blindly retry unless that tool call is idempotent — meaning it takes an idempotency key and the downstream system will deduplicate. If you're building tools for an agent to call, giving every state-changing tool an idempotency key parameter is one of the cheapest reliability wins available, because it turns "unsafe to retry" into "safe to retry" with almost no design cost.

Circuit breakers: stop calling things that are already down

Retrying with backoff handles transient failures. It does nothing for sustained outages — if a downstream tool API is fully down, backoff just means you fail slowly instead of quickly, while still consuming threads, connections, and time budget on every attempt.

A circuit breaker sits in front of a dependency and stops sending traffic to it once failures cross a threshold, giving it time to recover instead of pummeling it with retries from every agent instance.

import time
from enum import Enum

class CircuitState(Enum):
    CLOSED = "closed"       # normal operation
    OPEN = "open"           # failing, reject calls immediately
    HALF_OPEN = "half_open" # testing if recovered

class CircuitBreaker:
    def __init__(self, failure_threshold: int = 5, recovery_timeout: float = 30.0):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.failure_count = 0
        self.state = CircuitState.CLOSED
        self.opened_at = None

    def call(self, fn):
        if self.state == CircuitState.OPEN:
            if time.monotonic() - self.opened_at >= self.recovery_timeout:
                self.state = CircuitState.HALF_OPEN
            else:
                raise CircuitOpenError("Downstream tool is unavailable, failing fast")

        try:
            result = fn()
        except Exception:
            self.failure_count += 1
            if self.failure_count >= self.failure_threshold:
                self.state = CircuitState.OPEN
                self.opened_at = time.monotonic()
            raise
        else:
            if self.state == CircuitState.HALF_OPEN:
                self.state = CircuitState.CLOSED
            self.failure_count = 0
            return result

class CircuitOpenError(Exception):
    pass

In an agent system, wrap this around every external tool call, not just the LLM provider call. Agents often have five or ten tools registered, and if one of them (say, a flaky internal API) starts failing, you want the agent to fail that single tool call fast and let the model's own reasoning route around it — try a different tool, tell the user it's unavailable — rather than have every agent turn stall for the full request timeout on a tool that's already known to be down.

Designing the user-facing degradation path

All of the above is invisible plumbing unless it changes what the user experiences when the system is under load. The systems that feel reliable under pressure are the ones that degrade in visible, honest steps rather than silently hanging or throwing a raw stack trace.

A reasonable degradation ladder for an agent product looks like:

1. Normal operation — full tool access, full context, streaming responses.
2. Soft degradation — reduce parallel tool calls, trim context window, disable the most expensive optional tools (e.g., a heavy research sub-agent), but keep the core conversation working.
3. Queued — accept the request but tell the user honestly it's queued, with an estimate, rather than a spinner that could mean anything.
4. Hard reject with guidance — when the queue itself is full, reject with a clear message and a concrete retry time, not a generic error.

def handle_request(user_id: str, message: str, system_load: float):
    if system_load < 0.7:
        return run_full_agent(user_id, message)
    elif system_load < 0.9:
        return run_degraded_agent(user_id, message, disable_tools=["deep_research"])
    elif system_load < 1.0:
        position = enqueue(user_id, message)
        return {"status": "queued", "position": position, "eta_seconds": position * 4}
    else:
        raise SystemOverloaded(retry_after_seconds=15, reason="At capacity")

The key idea is that "system_load" here should be a real composite signal — queue depth, provider rate limit headroom, circuit breaker states, concurrent agent loop count — not a guess. Most of the work in this article (buckets, sliding windows, circuit breakers) exists to produce that signal cheaply and accurately, so the degradation logic has something honest to check against.

Observability: you can't backpressure what you can't see

None of this holds up in production without visibility into where the pressure actually is. At minimum, track and alert on:

• Token bucket fill level over time, per provider and per limiter dimension (requests vs. tokens)
• 429 rate from your LLM provider, broken out by model — a rising 429 rate is your earliest warning, well before users complain
• Queue depth and time-in-queue for the backpressure queue
• Circuit breaker state transitions, logged with the tool name so you know exactly which dependency is unstable
• Retry counts per request — a request that succeeded after 4 retries "worked" but cost 5x the resources of one that succeeded on the first try, and that's a cost signal worth tracking separately from a correctness signal

A dashboard that shows these five things will tell you, before your users do, whether you're about to have a bad night.

Bringing it together

Rate limiting and backpressure aren't a single library you drop in — they're a set of decisions repeated at every layer your agent touches: the LLM provider, your own ingress, every tool you've wired up, and your own process's concurrency ceiling. Token buckets handle bursty traffic against a shared quota. Sliding windows enforce fairness per user. Backpressure queues with priority and admission control stop you from accepting work you can't finish. Backoff with jitter and circuit breakers stop failures from cascading into full outages. And none of it means anything to a user unless it surfaces as an honest, graduated degradation experience instead of a hang or a stack trace.

The teams that get bitten hardest by this are the ones who only discover their limits exist when a real usage spike finds them for the first time — usually right after a launch or a viral moment, which is the worst possible time to be debugging 429s for the first time. Building this in from the start is cheap. Retrofitting it under an incident is not.

If you want to go deeper on the architecture side of this — how agent orchestration, tool design, and production reliability patterns like these fit together in a real multi-agent system — our 30 Days of Hermes Agent course walks through building a production-grade agent system from scratch, including the exact queueing, retry, and circuit breaker patterns covered here, applied to a real working codebase instead of isolated snippets.