Model Routing Strategies for Cost and Quality

Model routing is the practice of deciding, per request, which language model should handle it instead of hardcoding one model for your whole application. Done well, it cuts your inference bill by half or more while keeping output quality where it needs to be, because most requests in a real product are simple and only a minority actually need your most capable (and most expensive) model. This guide walks through the signals a router should use, the three common routing architectures, a working code example, and the failure modes that turn a router into a liability instead of a savings mechanism.

Why model routing matters once you're in production

When you're prototyping, it's normal to point every call at the strongest model you have access to. You don't know yet which requests are hard, so you don't want to under-serve any of them. That's a fine default for a demo. It's a bad default for a product with real traffic.

Once you have volume, the cost curve stops being abstract. A classification endpoint, a support-ticket triage step, a "summarize this paragraph" feature: these are typically simple, high-volume, and latency-sensitive. Routing them to your top-tier model means paying premium per-token rates for work a smaller model handles just as well, and usually faster. Meanwhile, a smaller subset of your traffic (complex multi-step reasoning, long-context synthesis, anything customer-facing where a wrong answer is expensive) genuinely benefits from your strongest model. Model routing is how you stop treating those two categories the same.

The other reason routing matters: model providers now ship multiple tiers on purpose. Anthropic's current lineup, for example, spans Claude Haiku 4.5 at the low end, Claude Sonnet 5 in the middle, and Claude Opus 4.8 at the top, with Claude Fable 5 available above Opus for the hardest long-horizon agentic work. Pricing scales with capability: Haiku 4.5 lists at $1 per million input tokens and $5 per million output tokens, Sonnet 5 at $3/$15 (with an introductory $2/$10 rate through the end of August 2026), Opus 4.8 at $5/$25, and Fable 5 at $10/$50. That's roughly a 10x spread between the cheapest and most expensive tier on the same provider. If your router can correctly send 70% of traffic to the cheap tier, your blended cost drops dramatically without anyone noticing a quality dip. Other providers publish similarly tiered lineups (small/medium/large or "mini"/flagship splits): the router pattern in this article applies regardless of which vendor's models you're routing between.

The core routing signals

Every router, no matter how it's implemented, is making a decision based on some combination of these signals:

• Task type. Classification, extraction, and formatting are usually easy. Multi-step reasoning, code generation across files, and open-ended synthesis are usually hard.
• Input length and structure. A three-sentence support message and a 40-page contract are not the same job even if the instruction ("summarize this") is identical.
• Confidence or ambiguity in the request. Vague, underspecified requests often need a stronger model that can infer intent correctly on the first pass, because a retry loop is more expensive than routing well the first time.
• Downstream cost of a wrong answer. A miscategorized support ticket costs a human a few seconds of review. A wrong number in a financial summary costs trust. Route by consequence, not just by apparent difficulty.
• Latency budget. Smaller models are usually faster per token and have shorter queues under load. If a feature is on the critical path of a user-facing interaction, that's a pull toward the cheaper, faster tier even when quality margins are close.
• User or plan tier. Some products deliberately route free-tier users to a cheaper model and paid-tier users to a stronger one. This is a product decision as much as a technical one, and it should be made explicitly, not as an accidental side effect of a routing bug.

A good router usually combines two or three of these rather than relying on one. Input length alone is a weak proxy for difficulty: a one-line prompt can still require deep reasoning ("what's the tax implication of this transaction structure"), and a long input can still be a trivial task ("does this document contain the word 'terminated'").

Static rules: the first router you should build

Before reaching for a classifier or a fancy scoring model, build the dumbest router that could possibly work: a set of deterministic rules based on request metadata you already have.

def choose_model(task: dict) -> str:
    task_type = task["type"]
    input_tokens = task["estimated_input_tokens"]

    if task_type in ("classify", "extract_field", "format_check"):
        return "claude-haiku-4-5"

    if task_type == "summarize" and input_tokens < 4000:
        return "claude-haiku-4-5"

    if task_type in ("code_review", "multi_step_agent", "long_context_synthesis"):
        return "claude-opus-4-8"

    # default: balanced tier for everything else
    return "claude-sonnet-5"

This is not sophisticated, and that's the point. Static rules are auditable, they're free to run (no extra model call to decide), and they fail predictably: if you misclassify a task type, you know exactly why, because you wrote the rule. Most teams should run a static router in production for months before they have enough labeled failure data to justify anything more complex.

The failure mode of static routing is that it doesn't adapt to individual request difficulty within a task type. Not every "summarize" request is equally easy. That's what the next tier of routing solves.

Classifier-based routing

A classifier router adds one more step: before the real request goes to a model, a fast, cheap call (or a lightweight non-LLM classifier) estimates task difficulty and picks a tier accordingly.

The classifier itself should almost always run on your cheapest model, or better yet, not be an LLM call at all if you can get away with a small trained classifier or even a few heuristics (input length, presence of code blocks, number of distinct sub-questions, whether the request references external documents). If you do use an LLM as the classifier, keep it to one word of output and a tight prompt:

import anthropic

client = anthropic.Anthropic()

CLASSIFIER_PROMPT = """Classify the difficulty of the following request as
exactly one word: simple, moderate, or complex.

simple: single-step lookup, classification, short extraction, formatting
moderate: multi-part questions, medium-length summarization, straightforward code edits
complex: multi-step reasoning, long-context synthesis, ambiguous requests, agentic workflows

Request: {request}

Answer with exactly one word."""

def classify_difficulty(request: str) -> str:
    response = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=8,
        messages=[{"role": "user", "content": CLASSIFIER_PROMPT.format(request=request)}],
    )
    return response.content[0].text.strip().lower()

TIER_MAP = {
    "simple": "claude-haiku-4-5",
    "moderate": "claude-sonnet-5",
    "complex": "claude-opus-4-8",
}

def route(request: str) -> str:
    difficulty = classify_difficulty(request)
    return TIER_MAP.get(difficulty, "claude-sonnet-5")  # default to mid-tier on unknown output

The obvious cost tradeoff: you're now making two model calls instead of one for every request. That only pays off if the classifier call is cheap enough (it should be, on your smallest model, with an 8-token response cap) and if it correctly routes enough traffic away from your most expensive tier to offset the extra call. Measure this. If 90% of your traffic would have gone to the mid-tier model anyway, a classifier step is pure overhead; static rules were already good enough.

One adjustment worth knowing about if you're using Claude models specifically: several current Claude models support an effort parameter that controls how much the model reasons internally, independent of which model you picked. This gives you a second routing dimension within a single model tier, not just across tiers. A Sonnet 5 call at effort: "low" is faster and cheaper than the same call at effort: "high", without changing which model handles it:

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=2048,
    thinking={"type": "adaptive"},
    output_config={"effort": "low"},  # low | medium | high | xhigh | max
    messages=[{"role": "user", "content": user_request}],
)

For a lot of routing problems, tuning effort per request is a cheaper lever to pull than switching models entirely, since it doesn't require maintaining a separate classifier or worrying about cross-model prompt differences. A practical router often combines both: pick the model tier by task type, then pick the effort level by estimated complexity within that tier.

Cascade routing: try cheap first, escalate on failure

The third common pattern is a cascade: always attempt the cheapest model first, and only escalate to a stronger model if the cheap model's output fails some quality check.

def cascade_request(prompt: str, verify_fn) -> tuple[str, str]:
    """Try the cheapest model first, escalate on failure. Returns (output, model_used)."""
    tiers = ["claude-haiku-4-5", "claude-sonnet-5", "claude-opus-4-8"]

    for model in tiers:
        response = client.messages.create(
            model=model,
            max_tokens=2048,
            messages=[{"role": "user", "content": prompt}],
        )
        output = response.content[0].text

        if verify_fn(output):
            return output, model

    # every tier failed verification; return the last (strongest) attempt anyway
    return output, tiers[-1]

Cascades work well when you have a cheap, reliable way to check the output without another full model call: a schema validator on structured output, a regex match on an expected format, a confidence score the model itself returns, or a rule-based sanity check ("does this SQL query reference a table that exists"). Cascades work poorly when verification itself requires judgment, because now you've either built another routing problem (who verifies the verifier) or you're paying for a second LLM call to grade the first one, which erodes the savings.

The other cost to watch with cascades: latency. A request that fails verification twice before succeeding on the third tier has paid the latency of three sequential calls, not one. If your product has a hard response-time budget, cascades need a timeout or a maximum-tiers-to-try limit, and you should decide upfront whether a slow-but-cheap path or a fast-but-expensive path wins when they conflict.

Putting it together: a router with fallback

A production router usually needs to handle one more thing beyond picking a model: what happens when the chosen model is rate-limited, overloaded, or errors out. Retrying against the same model with backoff is standard, but a router can also fall back to an adjacent tier so the user doesn't just see a spinner.

import anthropic

client = anthropic.Anthropic()

FALLBACK_CHAIN = {
    "claude-haiku-4-5": "claude-sonnet-5",
    "claude-sonnet-5": "claude-opus-4-8",
    "claude-opus-4-8": None,  # already at the top of the normal chain
}

def call_with_fallback(model: str, messages: list, max_tokens: int = 2048):
    current = model
    while current is not None:
        try:
            return client.messages.create(
                model=current,
                max_tokens=max_tokens,
                messages=messages,
            )
        except anthropic.RateLimitError:
            current = FALLBACK_CHAIN.get(current)
        except anthropic.APIStatusError as e:
            if e.status_code >= 500:
                current = FALLBACK_CHAIN.get(current)
            else:
                raise
    raise RuntimeError("All models in the fallback chain failed")

Two things worth flagging in code like this. First, catch specific exception types (RateLimitError, then a broader APIStatusError checked for a 5xx status) rather than a single bare except Exception, so a genuine 400 from a malformed request doesn't silently get treated as "try a bigger model," which would just repeat the same bad request at higher cost. Second, decide upfront whether escalating on failure should also happen for capacity-related errors (rate limits, overload) or only for hard errors. Escalating on rate limits is often the right call, since it turns a user-facing failure into a slightly more expensive success instead.

Monitoring: the part teams skip

A router you can't measure is a router you're guessing about. At minimum, log three things per request: which model handled it, what the routing signal was (task type, classifier output, cascade tier reached), and the actual token usage and latency. You don't need a dashboard on day one, but you need the raw log lines, because the questions you'll eventually ask are:

• What fraction of traffic lands on each tier, and does that match what you expected when you designed the rules
• Are requests escalating (in a cascade) more often than your cost model assumed
• Is the cheap tier producing outputs that get corrected, edited, or regenerated by users at a meaningfully higher rate than the expensive tier, which would mean your routing threshold is set wrong
• What's your blended cost per request now versus what it would have been on a single-model baseline

That last number is the one that justifies the whole exercise to whoever approves your infrastructure budget. Compute it monthly, not once at launch, because traffic composition shifts as your product changes and a router tuned for last quarter's traffic mix can quietly drift into either overspending or under-serving.

Common mistakes in model routing

• Routing by input length alone. Length is a weak proxy for difficulty. A router that only checks token count will misroute plenty of short-but-hard and long-but-easy requests.
• No escape hatch for misrouted requests. If a user or an internal monitor flags that a response was wrong, there should be a manual override to force the next similar request through the strongest tier, not just a hope that the router self-corrects.
• Treating the classifier call as free. It isn't. Measure whether the classification step earns back its own cost in savings elsewhere.
• Ignoring effort or reasoning-depth settings. On models that expose a tunable reasoning parameter, switching models is a bigger and blunter lever than it needs to be. Tune the cheaper lever first.
• Static thresholds that never get revisited. Model capability and pricing both change over time. A routing rule written against last year's model lineup can be actively wrong six months later; revisit thresholds on a schedule, not just when something breaks.
• No fallback path. A router with no plan for rate limits or provider outages turns a capacity blip into a full outage for your feature.
• Optimizing for cost with no quality floor. The point of routing is to spend less where quality doesn't suffer, not to spend the least possible amount everywhere. Set a minimum acceptable quality bar per task type before you start tuning for cost, or the router will quietly degrade the product to save money nobody asked it to save.

FAQ

What's the difference between model routing and prompt caching? They solve different problems and stack well together. Prompt caching reduces the cost of repeated context within calls to the same model. Model routing decides which model handles the call in the first place. A well-built system does both: route to the cheapest model that can do the job, and cache the shared prefix (system prompt, tool definitions, few-shot examples) on whichever model gets picked.

Should I build my own router or use a routing service? Third-party routing services exist and can be useful if you want to route across multiple providers without maintaining that logic yourself. For most teams, a router that only needs to pick between tiers of a single provider is simple enough to own directly: it's a function that inspects a request and returns a model string, plus logging. Start there before adopting an external dependency for something this small.

How do I decide my initial routing thresholds without production data? Start conservative: route only the task types you're confident are simple (classification, short extraction, formatting checks) to the cheapest tier, and default everything else to your mid-tier model. Collect a few weeks of logs, look at where users correct or regenerate outputs, and use that signal to widen the cheap-tier bucket gradually. Don't guess your way to an aggressive threshold on day one.

Does routing hurt consistency, since different requests might get different models? It can, if you're not careful about which parts of your system are routing-sensitive. Anything where users compare outputs side by side (two summaries of similar documents that came from different tiers) can surface visible quality differences. For those surfaces, either pin a single model or make sure your quality floor is tight enough that tier differences aren't noticeable. For independent, one-off requests (support ticket triage, background classification), inconsistency across tiers is invisible to any single user and not worth worrying about.

Is it worth routing between providers, not just between one provider's tiers? Sometimes, but it adds real complexity: different providers have different prompt formats, different tool-calling conventions, and different failure modes, so a cross-provider router needs an abstraction layer over all of that. Get single-provider tier routing working and measured first. Cross-provider routing is worth it mainly when you have a specific reason (redundancy against a single provider's outages, or a task type where a competing model is measurably better), not as a default architecture choice.

How often should I re-evaluate which tasks go to which tier? Whenever a provider ships a new model generation, and at minimum quarterly even without a new release, since your own traffic mix changes. A routing rule that made sense for your product six months ago can be stale simply because your feature set grew, not because anything about the models changed.