Multi-Region LLM Deployment

An LLM multi-region deployment runs your inference traffic across two or more geographic regions so users hit a nearby endpoint and a regional outage does not take the whole product down. The hard parts are not the model, they are latency-based routing, health checks that understand token streaming, and keeping prompt caches warm where the traffic actually lands. This guide walks through the architecture, the routing config, the failover logic, and the data-residency rules that decide where a request is even allowed to run.

If you run a single-region setup today and users on the other side of the world complain that the assistant "feels slow," most of that pain is round-trip network time to your inference region plus time-to-first-token. A well-built llm multi-region layout attacks both.

Why llm multi-region is different from a normal web tier

You have probably deployed a stateless web app to three regions behind a global load balancer. LLM traffic breaks several assumptions that setup relies on.

Requests are long-lived. A single chat completion can stream for 20 to 90 seconds. Your load balancer idle timeouts, your proxy buffering, and your health-check intervals were all tuned for sub-second requests. Streaming responses need the whole path to flush chunks immediately, so any buffering proxy in the middle destroys the feel of the product even when total latency is fine.

Responses are expensive and non-idempotent-ish. Retrying a failed GET is free. Retrying a half-streamed completion means you either pay for tokens twice or show the user a truncated answer followed by a fresh one. Failover has to happen before the first token, not mid-stream, or you need client logic to discard partial output.

State lives in caches, not in the request. Modern inference gets most of its speedup from prompt caching: the provider or your own server keeps the KV cache for a shared prefix (system prompt, tool definitions, long document) so repeat calls skip recomputation. That cache is regional. Route the same conversation to a different region and you lose the cache hit, so latency and cost both jump. This is the single most overlooked fact in llm multi-region design.

Capacity is rate-limited per region and per account. You cannot assume a region can absorb a full failover of another region's traffic. You need to know your per-region throughput ceiling before you draw the failover arrows.

Two deployment models: hosted API vs self-hosted weights

Before any routing config, decide which of these you are running, because the multi-region story is completely different.

Model A, hosted provider API. You call a managed endpoint (Claude via the Anthropic API or Amazon Bedrock, OpenAI, Google Vertex AI, and so on). The provider already runs multiple regions. Your "multi-region" job is mostly about which regional endpoint you send each request to, plus data-residency rules. You do not manage GPUs. Bedrock and Vertex expose regional endpoints like us-east-1 or europe-west4, and Anthropic's own API and Google Vertex both offer region-scoped or global routing options. Your control surface is the endpoint hostname and the region field.

Model B, self-hosted open-weight models. You run something like Llama, Mistral, or Qwen on your own GPUs with a serving engine such as vLLM, TGI, or SGLang. Now you own everything: GPU capacity per region, model weight distribution, autoscaling, KV-cache management, and the load balancer. This is far more work and only pays off at high, steady volume or under strict data-control requirements.

Most teams should start with Model A and only move to Model B when cost at scale, latency floors, or data control force the issue. The rest of this guide covers both, and calls out which model each section applies to.

Reference architecture

A practical llm multi-region topology, provider-agnostic:

• Global anycast DNS or a global load balancer (Cloudflare, AWS Global Accelerator, GCP global external LB) as the single entry hostname.
• A thin gateway or router service deployed in each region. This is your code: it does auth, rate limiting, request shaping, provider selection, retries, and failover. Keep it stateless.
• The inference backend per region. For Model A that is the provider's regional endpoint. For Model B that is your vLLM or TGI fleet behind a regional internal LB.
• A regional cache and session store (Redis or similar) for conversation metadata, idempotency keys, and rate-limit counters. Keep session-to-region affinity here.
• Centralized logging and metrics shipped out of every region to one observability backend so you can see cross-region behavior in one place.

The gateway is the piece you actually build. Everything else is configuration. Keep model calls out of your monolith and behind this gateway so routing, failover, and provider swaps are one service's job.

Routing: get the user to the right region

Three routing strategies, usually combined.

Latency or geo routing sends the user to the nearest healthy region. A global load balancer does this automatically by measuring RTT or by geo-mapping the client IP. This is your default and it handles the common case: EU users to an EU region, US users to a US region.

Session affinity pins an ongoing conversation to the region that holds its prompt cache. This matters because of KV-cache locality. Once a conversation starts in eu-west, keep sending its turns there while it is warm. Implement affinity with a signed hint the client returns on each turn, or a lookup in your session store keyed by conversation ID. Do not rely on sticky cookies alone across a global LB.

Residency routing overrides both of the above when law or contract requires it. An EU customer's data may be legally required to stay in EU regions regardless of latency. Residency is a hard constraint and must be evaluated first, before latency or affinity. Encode it as an attribute on the tenant, not a guess from IP.

Order of evaluation, first match wins:

1. Residency constraint for this tenant. If set, the candidate region list is restricted to allowed regions only.
2. Session affinity. If the conversation is warm in an allowed region, prefer it.
3. Latency. Among remaining allowed regions, pick the nearest healthy one.

Here is the core of a gateway router in Python. It picks a target region from tenant residency, session affinity, and health, and it is provider-agnostic.

import time
from dataclasses import dataclass, field

@dataclass
class Region:
    name: str
    endpoint: str
    healthy: bool = True
    latency_ms: float = 9999.0
    inflight: int = 0
    max_inflight: int = 200  # per-region concurrency ceiling

@dataclass
class RouteContext:
    tenant_allowed_regions: set  # empty set means "no residency limit"
    session_region: str | None   # warm cache location, if any
    session_ttl_epoch: float     # when the cache affinity expires

def choose_region(regions: dict[str, Region], ctx: RouteContext) -> Region:
    # 1. Residency: hard filter first.
    candidates = [
        r for r in regions.values()
        if not ctx.tenant_allowed_regions
        or r.name in ctx.tenant_allowed_regions
    ]
    candidates = [r for r in candidates if r.healthy and r.inflight < r.max_inflight]
    if not candidates:
        raise NoRegionAvailable("no healthy region satisfies residency + capacity")

    # 2. Affinity: reuse the warm region if still allowed, healthy, and unexpired.
    if ctx.session_region and time.time() < ctx.session_ttl_epoch:
        for r in candidates:
            if r.name == ctx.session_region:
                return r

    # 3. Latency: nearest healthy allowed region wins.
    return min(candidates, key=lambda r: r.latency_ms)

class NoRegionAvailable(Exception):
    pass

Note that residency is applied as a filter that can legally return zero candidates. When it does, you fail the request rather than silently routing to a forbidden region. That is the correct behavior; a compliance breach is worse than an error.

Failover that respects streaming

The rule: fail over before the first token, never mid-stream.

Your gateway opens the upstream connection, and if the connection or the first chunk fails within a short deadline, it retries against the next region. Once the first token has streamed to the client, you do not silently retry, because the user has already seen partial output. At that point you either surface the error or, if you buffered, discard and restart with an explicit client signal.

import httpx

async def stream_with_failover(payload, regions, ctx, first_token_deadline=4.0):
    tried = []
    while True:
        region = choose_region(regions, ctx, exclude=tried)
        region.inflight += 1
        try:
            async with httpx.AsyncClient(timeout=None) as client:
                async with client.stream(
                    "POST", region.endpoint, json=payload,
                    timeout=httpx.Timeout(connect=2.0, read=None, write=5.0, pool=2.0),
                ) as resp:
                    resp.raise_for_status()
                    first = True
                    async for chunk in resp.aiter_bytes():
                        if first:
                            first = False
                            # Past this point we are committed to this region.
                        yield chunk
                    return
        except (httpx.ConnectError, httpx.ReadTimeout, httpx.HTTPStatusError) as e:
            tried.append(region.name)
            if len(tried) >= len(regions) or _already_streamed(e):
                raise
            # else loop and pick the next region
        finally:
            region.inflight -= 1

Two deadlines matter here. The connect timeout (2 seconds above) catches a dead region fast. A separate first-token deadline catches a region that accepts the connection but is overloaded and slow to start generating. Do not set a read timeout on the stream itself, because a long generation is normal, not a failure.

Retries also need budget control. Cap total attempts, add jitter, and respect the provider's Retry-After header on 429s. A retry storm across regions during a partial outage will convert a small incident into a full one. When a region returns 429 or 529 (overloaded), back off from that region specifically rather than hammering it.

Health checks that understand inference

A TCP ping or an HTTP 200 on /health tells you the box is up. It does not tell you the model server can generate tokens. Build a synthetic probe that runs a tiny real completion (a few tokens, fixed prompt) on an interval and measures time-to-first-token. Mark a region unhealthy when TTFT crosses a threshold or the probe errors, and require a couple of consecutive good probes before marking it healthy again to avoid flapping.

For Model B self-hosted, also scrape your serving engine's own metrics. vLLM exposes a Prometheus endpoint with queue depth, running vs waiting requests, and KV-cache utilization. When the waiting queue grows and cache utilization is near full, that region is saturated: shed or reroute new traffic before latency spikes, do not wait for the probe to trip.

# vLLM exposes Prometheus metrics; watch queue and cache pressure per region.
curl -s http://inference-eu-west.internal:8000/metrics \
  | grep -E 'vllm:num_requests_(waiting|running)|vllm:gpu_cache_usage_perc'

Data residency and compliance, the part that fails audits

This is where naive multi-region deployments get teams in trouble. Latency routing alone will happily send an EU user's data to a US region because it was 20 milliseconds closer. If that tenant is under a data-residency commitment, you just breached it.

Concrete rules to bake in:

• Residency is a tenant attribute, set explicitly, not inferred from client IP. A German company's employee traveling in the US is still a German tenant.
• Evaluate residency first in routing, as a hard filter that can return zero regions.
• Check your provider's regional guarantees. Hosted APIs differ: some offer region-pinned endpoints that guarantee processing and any zero-retention data handling within that geography, others route globally by default. On Bedrock and Vertex you pick the region explicitly; confirm whether "cross-region inference" features can spill to other geographies and disable that for residency-bound tenants.
• Prompt caches are data too. A warm KV cache in a region means that tenant's prefix content lived there. Residency applies to the cache, not just the request log.
• Log where each request ran. Store the resolved region on every request record so you can prove compliance in an audit.

If you serve a mix of residency-bound and unconstrained tenants, the unconstrained ones get full latency routing and the bound ones get a restricted region set. Same gateway, different candidate lists.

Cost and capacity planning

A few numbers to work out before going live, without inventing any.

Per-region throughput ceiling. Know how many concurrent requests or tokens per second each region can serve, whether that is your provider rate limit (Model A) or your GPU count times per-GPU throughput (Model B). Your failover plan is only real if the surviving region can absorb the dead one's load. If two regions each run at 70 percent, neither can take the other's traffic. Run each below 50 percent if you want true active-active failover, or accept graceful degradation (queueing, shedding low-priority traffic) during an outage.

Cross-region data transfer. Shipping large document context or embeddings between regions costs money and adds latency. Keep the retrieval store and the inference in the same region. Do not put your vector database in one region and your inference in another.

Cache hit economics. Prompt caching can sharply cut the cost and latency of the repeated prefix on cache hits. Fragmenting traffic across too many regions shrinks each region's cache hit rate. There is a real tension between latency (more regions, closer to users) and cache efficiency (fewer regions, warmer caches). Pick the smallest region set that meets your latency target, not the largest you can afford.

Testing and rollout

Do not trust a multi-region setup you have not broken on purpose.

• Kill a region in staging and confirm traffic drains to the survivors within your target window and that no request streams from two regions.
• Load-test a single region at your planned failover volume to confirm the ceiling you assumed is real.
• Test the residency filter: send a bound tenant's request and assert it never resolves to a forbidden region, including when the allowed region is down (it should error, not spill).
• Verify streaming end to end through every proxy hop. A buffering proxy will pass functional tests and still ruin time-to-first-token in production. Watch for it with the synthetic TTFT probe.
• Roll out region by region behind a flag. Add the second region as failover-only first, then promote it to active once the probes and dashboards look clean.

FAQ

Do I need multi-region for an LLM app at all? Not always. If your users are concentrated in one geography and you have no residency requirement, a single well-chosen region plus good caching may be enough, and it is far simpler. Reach for llm multi-region when you have geographically spread users hurting on latency, a hard uptime target that a single region cannot meet, or data-residency obligations. Adopt it for a concrete reason, not by default.

Should I use a hosted provider API or self-host open weights across regions? Start with a hosted API (Model A). The provider already runs the regions and the GPUs; your job shrinks to endpoint selection and residency. Self-hosting open weights (Model B) only pays off at high steady volume, when you need latency floors a shared endpoint cannot promise, or when data control requires the weights on your own hardware. The operational cost of running GPU fleets in several regions is large; do not take it on speculatively.

How do I keep prompt caching working across regions? Pin each conversation to the region where its cache is warm using session affinity, and expire that affinity when the cache would have aged out. Route by conversation ID, not just by nearest region. Accept that the first turn of a conversation in a new region pays the full uncached cost. Fewer regions means warmer caches, so do not over-shard.

What happens to a request that is streaming when its region dies? If it has not sent a first token yet, your gateway retries against another region transparently. If tokens have already streamed, you cannot silently retry without showing the user duplicate or contradictory output. Surface an error or restart with an explicit client signal. The design rule is to make the failover decision before the first token whenever possible, using a short first-token deadline.

How is residency routing different from latency routing? Latency routing optimizes for speed and will cross geographic boundaries to save milliseconds. Residency routing is a legal or contractual hard constraint that forbids certain boundaries entirely. Evaluate residency first as a filter that restricts which regions are even candidates, then apply latency among what remains. Never let a latency win override a residency rule.

Can I run active-active, or should one region be a standby? Either works, but active-active only survives a failover if each active region runs with enough headroom to absorb another's traffic. If your regions are already near capacity, an active-passive setup with a clearly provisioned standby, or active-active with load shedding during incidents, is more honest. Size the ceiling first, then choose the topology.

Where should my vector database and retrieval store live? In the same region as the inference that uses them. Cross-region retrieval adds both latency and data-transfer cost to every request, and it can create a residency problem if the store sits in a different geography than the tenant allows. Co-locate retrieval, inference, and cache per region.