LLM Gateways and Model Routing: A Practical Guide for 2026

An LLM gateway is a proxy layer that sits between your application code and one or more model providers, handling authentication, retries, rate limiting, routing, and cost tracking so your app code does not have to. If you are calling more than one model, more than one provider, or you are tired of hardcoding API keys and retry logic into every service that talks to an LLM, you need one. This guide covers what a gateway actually does, how model routing decisions get made in practice, and the failure modes that catch teams off guard once the gateway becomes load-bearing infrastructure.

What an LLM gateway actually does

Strip away the marketing and an LLM gateway is doing five jobs, none of which are glamorous but all of which matter once you have real traffic.

Unified request interface. Your application code sends one request shape (usually OpenAI-compatible or a gateway-specific schema) and the gateway translates it into whatever the target provider expects. Anthropic's Messages API, OpenAI's Chat Completions or Responses API, and Google's Gemini API all have different field names, different streaming formats, and different tool-calling conventions. The gateway absorbs that difference so your business logic does not have a provider-specific branch in it.

Authentication and key management. Instead of every service holding its own ANTHROPIC_API_KEY or OPENAI_API_KEY, the gateway holds the provider credentials and issues its own internal tokens (often per team, per project, or per environment) to callers. This is the difference between rotating a key in one place versus grepping your entire codebase for who has a copy.

Routing and fallback. This is the part people mean when they say "model routing" and it is covered in its own section below.

Rate limiting and retries. Providers enforce limits by requests per minute (RPM), tokens per minute (TPM), and sometimes tokens per day (TPD), which vary by usage tier. When a provider returns a 429 (rate limit) or a 529 (overloaded, in Anthropic's case), the response includes a retry-after header telling you how long to back off. A gateway centralizes that backoff logic instead of every microservice implementing its own half-correct retry loop, and it can queue or shed load before you ever hit the provider's limit.

Observability and cost tracking. Every provider returns token usage in its response (usage.input_tokens, usage.output_tokens, and for providers with prompt caching, cache_read_input_tokens and cache_creation_input_tokens). A gateway aggregates this across every call, tags it by team or feature, and turns "we spent money on LLMs this month" into "this feature spent this much, and here is the per-request breakdown."

None of these five jobs require model routing across providers. You could build a gateway that only ever talks to one provider and it would still be worth the code, because key management, retries, and cost tracking are useful in isolation. Routing is where things get interesting, and where they get risky.

Why teams reach for a gateway

The trigger is almost always one of three things.

You are calling more than one model for cost or capability reasons. A common pattern: a fast, cheap model handles classification or extraction, a mid-tier model handles the bulk of user-facing generation, and a top-tier model is reserved for the hardest requests or as a fallback when the mid-tier model's output fails a quality check. As of 2026, Anthropic's lineup illustrates the shape of this tradeoff: Claude Haiku 4.5 is the fastest and cheapest tier, Claude Sonnet 5 is the balanced workhorse most production traffic runs on, and Claude Opus 4.8 is the most capable Opus-tier model for the hardest agentic and long-horizon work. Pricing differs meaningfully across the tiers, and a gateway is what lets you route by task difficulty instead of hardcoding one model everywhere.

You need provider redundancy. Every provider has outages. If your product depends on a single provider with no fallback, a provider-side incident becomes your incident. A gateway with a configured fallback chain can catch a 5xx or 529 from your primary provider and retry against a secondary provider or a secondary region without the end user ever seeing an error.

You need centralized governance. Once more than one team is calling LLMs, someone needs to answer "how much are we spending, on what, and is anyone using a model we didn't approve." A gateway is the natural enforcement point for allowed models, spending caps per team, and audit logging, because it is the one place every request already passes through.

If none of these apply, you probably do not need a gateway yet. A single service calling a single model directly with the official SDK is simpler and has fewer moving parts to debug. Add the gateway when the second provider, the second team, or the first real outage shows up.

Model routing strategies

"Model routing" covers several distinct decisions that get conflated under one term. Separating them helps you reason about what you are actually optimizing for.

Static routing by task type

The simplest strategy: you know in advance that summarization goes to model A, code generation goes to model B, and classification goes to model C, because you benchmarked them and picked winners. This is configuration, not intelligence. It is also the strategy most production systems should start with, because it is debuggable. When something goes wrong you know exactly which model handled the request and you can reproduce it.

Cost-based routing

Route by estimated cost of the request. A short classification prompt with a small, bounded output goes to the cheapest capable model. A long-context research task with open-ended output goes to a model priced for that context window. The complication is that you often do not know the true cost until after the call completes, since output length is the hard-to-predict half of the bill. Teams handle this by capping max_tokens aggressively for cheap-tier requests and monitoring actual spend per route rather than trying to predict it perfectly upfront.

Quality-based routing (cascade)

Try the cheap model first. If its output fails a validation check (a schema mismatch, a low-confidence score, an empty tool call when one was expected), escalate to a more capable model and retry. This is sometimes called a cascade. It works well for tasks with a cheap, automatable pass/fail signal, like structured extraction where you can validate the output against a JSON schema. It works poorly for open-ended generation where "quality" cannot be checked without another LLM call, at which point you are paying for the judge as well as both attempts.

Latency-based routing

Some requests are latency-sensitive (a chat UI waiting on a token stream) and some are not (an overnight batch job). Route the former to the fastest available model and provider region; route the latter to whatever is cheapest, potentially using an async batch API that trades turnaround time for a substantial price cut. Anthropic's Message Batches API, for example, processes asynchronously at a meaningful discount off standard per-token pricing, with most batches completing within an hour and a firm 24-hour ceiling. That is a real lever for anything that does not need a synchronous response: nightly summarization runs, backfilling embeddings, bulk classification.

Failover routing

Distinct from cost or quality routing: this triggers only on error, not on every request. Primary provider returns a 5xx, a 529, or times out; the gateway retries against a configured fallback (a different provider, a different region, or a different model from the same provider) without changing behavior on the happy path. Anthropic's own API supports a related idea at the model layer for its newest model, Claude Fable 5: a fallbacks parameter that lets a single request specify a substitute model to run automatically if safety classifiers decline the primary request, with the retry billed as a fallback attempt rather than a full duplicate charge. That is a narrower, provider-native version of the same failover instinct a gateway applies more broadly across providers and error types.

A practical routing layer combines two or three of these, not all five. The most common combination in production: static routing by task type as the default, cost-based tiering within a task type (cheap model first, escalate on cache miss or validation failure), and failover routing wrapping the whole thing so a provider outage does not become a user-facing outage.

Building versus buying

You have three real options, in increasing order of control and decreasing order of speed to ship.

A hosted gateway service. Several vendors run multi-provider gateways as a managed service: you point your SDK's base URL at them, they handle key management, routing, and observability, and you pay a markup or a flat fee on top of provider costs. Fastest to integrate, least control over exact routing logic, and you are trusting a third party with your provider keys and your traffic.

An open-source gateway you self-host. Projects like LiteLLM and others provide a proxy server you run yourself, configured via YAML or code, that normalizes multiple providers behind one interface. You get more control and keep keys in your own infrastructure, at the cost of running and patching another service.

A thin routing layer you write yourself. For teams calling two or three providers with straightforward rules, a few hundred lines of code wrapping each provider's official SDK, with a shared retry and fallback function, is often less operational overhead than adopting a framework. This is a defensible choice when your routing logic is genuinely simple and you want to avoid a dependency whose abstractions do not map cleanly onto provider-specific features you actually need, like extended thinking parameters, prompt caching controls, or structured output schemas that vary by provider.

The wrong choice, seen often, is adopting a heavy gateway framework and then fighting it every time a provider ships a feature the abstraction was not designed for. Provider-specific features move fast right now: adaptive thinking effort levels, server-side tool use, prompt caching TTLs, structured output enforcement. A gateway abstraction that flattens every provider into a lowest-common-denominator interface will lag behind what each provider's native SDK exposes. If your product depends on a provider-specific feature, make sure your gateway choice has an escape hatch to call that provider natively when needed, rather than being stuck with only what the abstraction supports.

Cost tracking and observability

The reason cost tracking belongs in the gateway layer rather than scattered across services is that token usage is the only reliable unit of LLM cost, and it is only visible in the response, not the request. You cannot know in advance how many output tokens a generation will produce. A gateway that logs usage from every response, tagged with which team or feature triggered the call, turns a vague sense of "AI costs are going up" into an actual breakdown you can act on.

A few things worth getting right here:

Log the model actually used, not the model requested. If your routing logic escalated a request from a cheap model to an expensive one, or a fallback kicked in and served a different provider, your logs need to reflect what happened, not what was originally asked for. Otherwise your cost dashboard silently drifts from reality.

Separate cache-affected tokens from full-price tokens. Providers with prompt caching report cache reads and cache writes as distinct fields in the usage object, priced differently from standard input tokens (cache reads are typically a fraction of the full input price, cache writes carry a premium). If your dashboard sums everything into one "input tokens" number, you will misread your actual spend, and you will miss the signal that tells you caching is or is not working. A cache hit rate that silently drops to zero because something upstream started varying a timestamp or a UUID in a shared prompt prefix is a common, expensive, and completely invisible failure unless you are watching the cache fields specifically.

Track retries and fallbacks as separate line items. A request that failed once and succeeded on retry cost you two calls' worth of tokens (or more, if the failure happened mid-stream after partial output was already billed). If your cost attribution only counts the final successful response, you will underestimate spend and miss a signal that a particular route is unreliable.

Alert on rate-limit pressure, not just errors. The x-ratelimit-remaining-* headers most providers return tell you how close you are to a limit before you actually hit it. A gateway that only reacts after a 429 is reacting too late; the useful gateway watches the remaining-quota headers and starts shedding or queuing load before the hard limit hits.

Common mistakes

Treating the gateway as infallible. The gateway is new infrastructure with its own failure modes: it can be slow, it can have a bug in its routing config, it can hold a stale credential after a key rotation. A gateway with no health checks and no fallback for its own downtime has just moved your single point of failure one layer over, not removed it.

Routing purely on price without a quality floor. Chasing the cheapest available model per request without any check on output quality degrades the product in ways that are hard to see in aggregate metrics and easy to see in individual user complaints. Pair any cost-based routing with a lightweight quality signal, even something as blunt as a schema validation check or a length sanity check, before shipping the response.

Ignoring provider-specific breaking changes. Providers deprecate parameters and change defaults. A gateway abstraction can mask this until it suddenly does not; a parameter your gateway was silently dropping or translating stops being accepted, and every request through that route starts failing at once. Keep a changelog-watching habit for every provider your gateway talks to, not just the one you use most.

Not testing the fallback path. Fallback logic that has never actually been exercised in anger is a coin flip whether it works when you finally need it. Simulate a primary-provider failure in a staging environment periodically (a config flag that forces the gateway to treat every primary call as failed) so you know the fallback path actually serves traffic correctly, not just that the code compiles.

Retrying non-retryable errors. A 400 (bad request) or 401 (authentication error) will not succeed on retry; retrying it just burns time and, on some providers, still counts against rate limits. Only 429, 5xx, and 529-class errors are worth automatic retry with backoff. Baking that distinction into the gateway once means every caller downstream gets it for free instead of every service reimplementing its own (often wrong) retry logic.

FAQ

Do I need an LLM gateway if I only use one provider? Not for routing, but the other four jobs (key management, retries, rate limiting, cost tracking) are still useful with a single provider. A thin internal wrapper around the provider's official SDK that centralizes retry and logging logic is a lightweight version of a gateway and is worth building even at one-provider scale, because it is the foundation you extend when a second provider or a second team shows up.

What is the difference between a gateway and an agent framework? A gateway operates at the level of individual API calls: routing, retries, auth, cost tracking. An agent framework operates at the level of a multi-step task: planning, tool use, looping until a goal is met. They are complementary, not competing. An agent framework's individual model calls can and often should go through a gateway underneath.

Should routing decisions happen client-side or server-side? Server-side, almost always. Client-side routing (deciding which model to call from a browser or mobile app) exposes your routing logic and, in most naive implementations, your provider credentials to anyone who inspects network traffic. Keep the routing decision, the provider keys, and the retry logic behind your own API boundary.

How do I handle rate limits across multiple provider accounts? This is one of the stronger arguments for a gateway: it can hold multiple credentials for the same provider (different accounts, different organizations) and round-robin or load-balance across them to multiply your effective rate limit headroom, as long as your provider's terms of service permit that pattern. Check the provider's usage policies before doing this, since some explicitly restrict using multiple accounts to bypass rate limits.

Does prompt caching work across a gateway? It can, but only if the gateway preserves the exact byte-for-byte prefix of the prompt across requests, since caching is a strict prefix match on the rendered request. If your gateway reorders fields, reformats JSON without deterministic key ordering, or injects a timestamp anywhere before the cache breakpoint, you will silently lose cache hits even though the cache_control markers are present. Verify actual cache hit rates via the usage fields in the response, not by assuming caching is working because you configured it.

What is the right way to test a routing configuration before it hits production traffic? Replay a sample of real historical requests (with sensitive data scrubbed) through the new routing config in a shadow mode where the gateway logs what it would have done without actually serving that response to a user. Compare the shadow routing decisions and estimated costs against your current production routing before flipping traffic over. This catches obviously wrong routing rules (everything falling through to the most expensive model, a fallback chain that never terminates) before they show up as a cost spike or an availability incident.