RAG Query Routing: Sending Questions to the Right Index

AUTHOR: Pramod Dutta

RAG query routing is the step in a retrieval pipeline that decides which index, retriever, or data source should handle a given question before any embedding search happens. Most tutorials show RAG with a single vector store, but real systems accumulate multiple indexes fast: a product docs index, a support ticket index, a SQL database of order history, maybe a web search tool for anything current. Query routing is the dispatcher that looks at an incoming question and picks the right destination, or destinations, instead of blasting every query at every source.

If you've only ever built single-index RAG, routing feels like an unnecessary layer. It stops feeling optional the moment you add a second data source. Search a support-ticket index with a question about pricing tiers and you'll get confident, well-ranked, completely wrong results, because the retriever will happily return the closest match even when the closest match isn't useful. Routing is what prevents that.

Why RAG query routing matters once you have more than one index

A single vector index works fine when all your content lives in one place and covers one topic. The moment you split content by type, freshness, or access level, you get index sprawl:

• A knowledge base index (product docs, FAQs) that changes rarely.
• A support-ticket index (past conversations) that's noisy but has real troubleshooting detail.
• A structured database (orders, invoices, user accounts) better served by SQL than by embedding search.
• A code index (your own repo, or a customer's) with a completely different chunking strategy.
• A live web search or news API for anything time-sensitive.

Without routing, you have two bad options. Option one: query every index for every question and merge results. This multiplies retrieval latency and cost by the number of indexes, and it dilutes your context window with irrelevant chunks pulled from indexes that had nothing useful to offer but returned their closest match anyway. Option two: pick one index and always search it, which means questions outside that index's scope get answered with hallucinated or irrelevant context.

Query routing solves this by adding a lightweight decision step before retrieval: given this question, which index (or indexes) should we actually search? Done well, it cuts retrieval cost, reduces irrelevant context, and lets you scope permissions and freshness per source.

The three routing strategies

There are three broad ways to implement a router, and most production systems end up combining at least two of them.

1. Rule-based routing

The router is a set of explicit conditions: regex matches, keyword lists, metadata filters. If the query contains "refund" or "order status," send it to the orders database. If it mentions a product name from a fixed list, send it to that product's doc index.

Rule-based routing is fast, free (no LLM call), and fully deterministic, which matters for debugging and for compliance-sensitive routing (you can prove exactly why a query went where it went). The tradeoff is brittleness: users don't phrase things the way your keyword list expects, and every new phrasing requires a rule update.

def rule_based_route(query: str) -> str:
    q = query.lower()
    if any(word in q for word in ["refund", "order status", "invoice", "charge"]):
        return "orders_db"
    if any(word in q for word in ["error", "not working", "broken", "crash"]):
        return "support_tickets"
    if any(word in q for word in ["latest", "today", "this week", "news"]):
        return "web_search"
    return "docs_index"

Use rule-based routing for the small number of high-confidence, high-volume patterns in your traffic. It's rarely the whole solution, but it's a cheap first filter that catches the obvious cases before anything more expensive runs.

2. Semantic (embedding-based) routing

Instead of matching keywords, you embed the incoming query and compare it against a small set of reference embeddings, one per route, usually built from a handful of example questions per index. The route whose reference embeddings are closest to the query wins.

This is the same nearest-neighbor idea as retrieval itself, just applied to route selection instead of document selection. It's more robust to phrasing variation than rules, and it doesn't require an LLM call, so it's cheap and low-latency.

import numpy as np
from openai import OpenAI

client = OpenAI()

route_examples = {
    "orders_db": [
        "where is my order",
        "I want a refund",
        "how much was I charged last month",
    ],
    "support_tickets": [
        "the app crashes when I upload a file",
        "I'm getting a 500 error on login",
        "the sync feature stopped working",
    ],
    "docs_index": [
        "what plans do you offer",
        "how do I set up SSO",
        "what's the API rate limit",
    ],
}

def embed(texts):
    resp = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return np.array([d.embedding for d in resp.data])

def build_route_centroids(route_examples):
    centroids = {}
    for route, examples in route_examples.items():
        vecs = embed(examples)
        centroids[route] = vecs.mean(axis=0)
    return centroids

def cosine_sim(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def semantic_route(query: str, centroids: dict) -> str:
    query_vec = embed([query])[0]
    scores = {route: cosine_sim(query_vec, c) for route, c in centroids.items()}
    return max(scores, key=scores.get)

Precompute the centroids once and cache them; only the query embedding happens per request. This approach scales well to a handful of routes (roughly 3 to 15) but degrades as routes multiply and their reference examples start overlapping semantically. If two indexes cover adjacent topics, embedding-based routing will confuse them more often than a human would.

Add a confidence threshold. If the top score and the second-best score are close, don't commit to a single route, fall back to a broader search or ask a clarifying question:

def semantic_route_with_confidence(query: str, centroids: dict, margin: float = 0.05):
    query_vec = embed([query])[0]
    scores = {route: cosine_sim(query_vec, c) for route, c in centroids.items()}
    ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)
    top_route, top_score = ranked[0]
    second_score = ranked[1][1] if len(ranked) > 1 else 0
    if top_score - second_score < margin:
        return None  # ambiguous, handle separately
    return top_route

3. LLM-based routing

The most flexible and most expensive option: ask an LLM to classify the query, typically via structured output or tool/function calling, so the model returns a route name (or a list of routes) instead of free text.

import json
from anthropic import Anthropic

client = Anthropic()

ROUTES = {
    "orders_db": "Order status, billing, refunds, invoices, account charges.",
    "support_tickets": "Bug reports, error messages, feature not working.",
    "docs_index": "Product documentation, setup guides, pricing, feature explanations.",
    "web_search": "Anything requiring current events, news, or information published after the knowledge cutoff.",
}

router_tool = {
    "name": "route_query",
    "description": "Select which index or indexes should be searched to answer the user's question.",
    "input_schema": {
        "type": "object",
        "properties": {
            "routes": {
                "type": "array",
                "items": {"type": "string", "enum": list(ROUTES.keys())},
                "description": "One or more routes to search, ordered by relevance.",
            },
            "reasoning": {"type": "string"},
        },
        "required": ["routes"],
    },
}

def llm_route(query: str) -> list[str]:
    route_descriptions = "\n".join(f"- {name}: {desc}" for name, desc in ROUTES.items())
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=300,
        tools=[router_tool],
        tool_choice={"type": "tool", "name": "route_query"},
        messages=[{
            "role": "user",
            "content": f"Available routes:\n{route_descriptions}\n\nQuery: {query}",
        }],
    )
    for block in response.content:
        if block.type == "tool_use":
            return block.input["routes"]
    return ["docs_index"]  # safe default

LLM-based routing handles ambiguity and multi-part questions better than the other two, since the model can reason about intent and return multiple routes when a question genuinely spans sources ("why was I charged twice and is this a known bug" needs both orders_db and support_tickets). The cost is an extra model call on every query, adding latency (typically 200ms to 1s depending on the model) and a per-query API cost. Use a small, fast model for routing specifically. You don't need your most capable model to classify intent; save that budget for the actual answer generation.

A layered routing architecture

In practice, the strongest setups layer these three strategies instead of picking one:

1. Rules first. Catch the high-confidence, high-volume cases for free. If a query obviously matches a rule, skip the rest of the pipeline.
2. Semantic routing second. For everything the rules don't catch, embed the query and compare against route centroids. Cheap, fast, no LLM call.
3. LLM routing as the fallback. Only invoke it when semantic routing's confidence margin is too thin to commit, or when you explicitly want multi-route reasoning for complex questions.

def route(query: str, centroids: dict) -> list[str]:
    rule_hit = rule_based_route(query)
    if rule_hit:
        return [rule_hit]

    semantic_hit = semantic_route_with_confidence(query, centroids)
    if semantic_hit:
        return [semantic_hit]

    return llm_route(query)

This keeps average latency and cost low, since most traffic resolves at step 1 or 2, while step 3 handles the genuinely ambiguous tail without needing to be fast.

Routing to retrieval strategies, not just indexes

Query routing isn't only about picking which index to search, it also decides how to search. Some questions need dense vector retrieval, some need keyword (BM25) search, some need a direct SQL query, and some need a live tool call. Fold that decision into the router too:

ROUTE_CONFIG = {
    "orders_db": {"type": "sql", "handler": query_orders_sql},
    "support_tickets": {"type": "hybrid", "handler": hybrid_search_tickets},
    "docs_index": {"type": "vector", "handler": vector_search_docs},
    "web_search": {"type": "tool", "handler": call_web_search_api},
}

def handle_query(query: str, centroids: dict):
    routes = route(query, centroids)
    all_context = []
    for r in routes:
        config = ROUTE_CONFIG[r]
        result = config["handler"](query)
        all_context.append(result)
    return all_context

A question like "what's the status of order 8842" should never touch a vector index at all. It's a structured lookup, and routing it straight to a SQL query is both faster and more accurate than trying to embed-search your way to an order ID.

Handling multi-index questions

Some queries legitimately need more than one source. "Why does the sync feature keep failing and what's your refund policy if I cancel" spans support tickets and docs. Two approaches work here:

Parallel fan-out with a route list. Let the router return multiple routes (as the LLM router above does), query them concurrently, then merge and re-rank the combined results before passing them to the generation step. Concurrency matters here: querying three indexes sequentially triples your latency, so fire the requests together with asyncio.gather or your framework's equivalent.

Query decomposition. For genuinely compound questions, split the query into sub-questions first, route each sub-question independently, then synthesize. This is more expensive (another LLM call for decomposition) but produces cleaner retrieval than trying to force one embedding to represent two different intents.

import asyncio

async def fan_out_route(query: str, routes: list[str]):
    tasks = [ROUTE_CONFIG[r]["handler"](query) for r in routes]
    results = await asyncio.gather(*tasks)
    return dict(zip(routes, results))

Don't reach for decomposition by default, it adds a full extra LLM round trip. Reserve it for routes flagged as multi-intent by your router, or for query patterns you've observed actually need it.

Evaluating your router

A router is a classifier, and you should evaluate it like one. Build a labeled test set of at least 100 to 200 representative queries with their correct route(s), pulled from real query logs if you have them, and track:

• Routing accuracy: percentage of queries sent to the correct index (or, for multi-route cases, whether the correct set was returned).
• Precision per route: of the queries sent to orders_db, how many actually belonged there? Low precision means you're polluting that index's context with irrelevant retrieval.
• Recall per route: of the queries that should have gone to docs_index, how many actually did? Low recall means real questions are getting misrouted elsewhere and coming back with bad answers.
• Latency added by the router itself, measured separately from retrieval and generation, so you can see whether the router is becoming the bottleneck.

Re-run this eval whenever you add a new index or route. New routes shift the decision boundary for existing ones, an embedding-based router that worked cleanly with three routes might start confusing two of them once a fourth, semantically adjacent route is added.

Common mistakes

Too many fine-grained routes. Splitting a single logical index into ten narrow ones because they were built at different times, rather than because they need different retrieval logic, mostly just gives your router more ways to be wrong. Merge indexes that share a retrieval strategy and audience; route by metadata filter within the index instead of by separate index.

No fallback route. Every router needs a default for the "none of the above" case. Without one, ambiguous queries either fail silently or get force-fit into whatever route scored highest, even if that score was low-confidence.

Routing decisions made invisible. Log the route chosen and the reasoning (if using an LLM router) for every query in production. When users report bad answers, the first thing to check is whether they got routed to the right place at all, before you start debugging the retrieval or generation steps.

Ignoring cost asymmetry between routes. A SQL lookup costs a database round trip. A web search tool call might cost real money per request. Weighting all routes as equally "free" in your router's decision logic can quietly blow up your API bill if the LLM router over-selects an expensive tool.

Treating routing as a one-time build. Query patterns shift as your product changes and users find new ways to ask things. Revisit your rule list and reference examples on a regular cadence, not just when something visibly breaks.

FAQ

Does every RAG system need query routing? No. If you have a single index covering a single domain, routing adds complexity with no benefit. Add it when you introduce a second data source with different content, format, or access requirements, that's the point where naive single-index search starts returning irrelevant results for out-of-scope questions.

Is semantic routing the same thing as retrieval? Conceptually similar (both do nearest-neighbor comparison over embeddings) but operationally different. Retrieval compares a query against thousands or millions of document chunks. Routing compares a query against a handful of route centroids, usually fewer than twenty. The scale difference is why routing can run cheaply on every request while full retrieval happens only after the route is chosen.

Should I use an LLM for routing or stick to embeddings? Start with rules and embeddings; they're fast and cheap, and they cover the majority of well-defined route boundaries. Add an LLM router as a fallback for the ambiguous tail, or when you need multi-route reasoning for compound questions. Running an LLM call on every single query, including the obvious ones, is usually wasted latency and cost.

How many routes is too many for embedding-based routing? There's no hard cutoff, but accuracy tends to degrade past 15 to 20 routes, especially if some of them cover semantically adjacent topics. Beyond that point, consider a hierarchical router: route to a broad category first with embeddings, then route within that category with a second, more specific step.

What happens if the router picks the wrong index? Whatever the retriever finds in the wrong index becomes the LLM's context, and the model will generate an answer grounded in irrelevant material, often confidently. This is why routing accuracy directly caps your system's overall accuracy: no amount of good generation prompting fixes a query that never reached the index with the actual answer.

Can I route to a "no retrieval needed" path? Yes, and you should. Not every user message needs RAG. Greetings, clarifying questions, or general knowledge the model already has don't benefit from a retrieval step, and skipping it saves latency and avoids injecting irrelevant context. Add a no_retrieval route to your router and let the model answer directly when it's selected.

Does query routing replace re-ranking? No, they solve different problems. Routing decides which index to search before retrieval happens. Re-ranking reorders the results retrieval already returned, within one index or across the merged results of several. Use both: route to narrow the search space, then re-rank to surface the best chunks from whatever the router selected.