RAG Caching Strategies: Cutting Latency and Cost

A production RAG pipeline pays for the same work over and over: the same embedding gets recomputed for near-duplicate queries, the same document chunks get re-retrieved for the same topic, and the same system prompt plus retrieved context gets re-sent to the model on every turn. RAG caching attacks each of those redundant steps independently, and done right it can cut both latency and per-query LLM spend without touching answer quality. Done wrong, it serves stale or wrong context and nobody notices until a user complains. This article walks through the four places caching pays off in a retrieval pipeline, the tradeoffs at each layer, and the invalidation problems that make RAG caching harder than caching a typical API response.

The reason RAG caching deserves its own treatment, separate from generic LLM response caching, is that a RAG pipeline has multiple stages that each produce cacheable intermediate results: embedding generation, vector search, reranking, and generation. Caching only the final answer (query -> response) gives you the smallest hit rate because it only helps on exact-duplicate questions. Caching at each stage compounds the savings because near-duplicate and partially-overlapping queries can reuse partial work even when the final answer differs.

Why RAG caching is different from normal caching

A typical web cache keys on a request and serves an identical response. RAG caching has to deal with three complications that normal HTTP caching does not:

• Fuzzy keys. Two users asking "how do I reset my password" and "password reset steps" should hit the same cache entry, but their strings are different. Exact-match caching on the raw query string misses almost all of this traffic.
• Multi-stage pipelines. A single RAG answer depends on an embedding call, a vector search, sometimes a rerank pass, and a generation call. Any of these can be cached independently, and a change in one (say, a document update) should only invalidate the stages downstream of it.
• Freshness requirements. If your knowledge base updates hourly, a stale cached answer is a correctness bug, not just a UX inconvenience. Caching aggressively without an invalidation story is how teams ship RAG systems that confidently tell users about a pricing plan that was retired last week.

Keep those three in mind. Every caching decision below is a tradeoff between hit rate, staleness risk, and implementation complexity.

Layer 1: caching embeddings

Embedding calls are cheap per-call but add up fast when you are embedding every incoming query and every document chunk during ingestion. Two different caches apply here.

Document embedding cache. During ingestion, you should never re-embed a chunk that has not changed. Hash the chunk text (plus the embedding model name and version, since a model upgrade invalidates everything) and store the embedding keyed on that hash. On re-ingestion of a source document, diff the new chunks against the hash cache and only call the embedding API for chunks that changed.

import hashlib

def chunk_cache_key(chunk_text: str, model_name: str) -> str:
    payload = f"{model_name}:{chunk_text}".encode("utf-8")
    return hashlib.sha256(payload).hexdigest()

def get_or_embed(chunk_text, model_name, embed_fn, cache):
    key = chunk_cache_key(chunk_text, model_name)
    cached = cache.get(key)
    if cached is not None:
        return cached
    vector = embed_fn(chunk_text)
    cache.set(key, vector)
    return vector

This alone often eliminates the majority of embedding calls on a re-ingestion run, since most documents in a corpus change incrementally, not wholesale.

Query embedding cache. Incoming user queries repeat more than people expect, especially in support and internal-tooling contexts where a handful of questions ("how do I request PTO", "what is the refund policy") make up a disproportionate share of traffic. An exact-match cache on the normalized query string (lowercased, whitespace-collapsed, punctuation-stripped) catches literal repeats cheaply.

def normalize_query(q: str) -> str:
    return " ".join(q.lower().strip().split())

def get_query_embedding(query, embed_fn, cache, ttl_seconds=86400):
    key = f"qemb:{normalize_query(query)}"
    cached = cache.get(key)
    if cached is not None:
        return cached
    vector = embed_fn(query)
    cache.set(key, vector, ttl=ttl_seconds)
    return vector

Exact-match query caching has a low ceiling on hit rate because real users phrase things differently. That is where semantic caching comes in.

Layer 2: semantic caching for near-duplicate queries

Semantic caching stores past query embeddings alongside their final answers, and on a new query it checks cosine similarity against recent cache entries before doing a full retrieval-plus-generation pass. If similarity clears a threshold, you serve the cached answer directly, skipping vector search and the LLM call entirely.

def semantic_cache_lookup(query_vector, cache_index, threshold=0.95):
    # cache_index is a small in-memory or Redis-backed vector index
    # of {query_vector, answer, created_at} entries
    match, score = cache_index.nearest(query_vector)
    if match and score >= threshold:
        return match["answer"]
    return None

def answer_query(query, embed_fn, cache_index, rag_pipeline_fn, store_fn):
    qvec = embed_fn(query)
    cached_answer = semantic_cache_lookup(qvec, cache_index)
    if cached_answer is not None:
        return cached_answer
    answer = rag_pipeline_fn(query, qvec)
    store_fn(cache_index, qvec, answer)
    return answer

The threshold is the whole game here. Set it too low (say, 0.85 cosine similarity) and you will serve wrong answers to questions that are topically related but factually different, like confusing "what is the refund window for annual plans" with "what is the refund window for monthly plans." Set it too high (0.98+) and the cache barely fires. In practice, tune the threshold against a labeled set of near-duplicate and non-duplicate query pairs from your own traffic rather than picking a number from a blog post, because the right threshold depends heavily on how semantically dense your query distribution is.

Two safety rules for semantic caching that are easy to skip and expensive to skip:

• Never semantic-cache queries with a time component ("what changed this week") or a user-specific component ("what is my order status"). Route those around the cache entirely, ideally by detecting them with a cheap classifier or keyword check before the similarity lookup.
• Cap the cache entry TTL well below your knowledge base's update cadence. If documents update daily, a semantic cache entry living for a week will serve confidently wrong answers for six of those days.

Layer 3: caching retrieval results

Vector search itself is not free, especially with reranking on top. Two retrieval-layer caches are worth separating.

Retrieval result cache. Cache the top-k chunk IDs (not the full text, just IDs plus scores) returned for a given query embedding, keyed the same way as the query embedding cache. On a cache hit you skip the vector search and reranking pass but still run generation fresh, which is the right tradeoff when your document set is fairly stable but you still want every answer to be freshly generated (useful when you are iterating on prompts or want per-user personalization in the final generation step).

Popular-query prefetching. For known high-traffic queries (FAQ-style questions, common support topics), precompute and cache retrieval results on a schedule rather than waiting for user traffic to populate the cache. This avoids the "first user after a cache eviction pays the full latency" problem for your highest-volume queries, and it lets you warm the cache immediately after a document update instead of waiting for organic traffic to refill it.

def prefetch_popular_queries(popular_queries, embed_fn, retrieve_fn, cache):
    for query in popular_queries:
        qvec = embed_fn(query)
        results = retrieve_fn(qvec, top_k=8)
        cache.set(f"retrieval:{normalize_query(query)}", results, ttl=3600)

Run this on a cron job right after ingestion completes, not on a fixed schedule disconnected from when your data actually changes.

Layer 4: prompt and context caching at the LLM layer

This is the layer most teams reach for first because most model providers now support it natively, and it requires the least pipeline surgery. The idea: your RAG system prompt (instructions, few-shot examples, tool definitions) and often a large chunk of retrieved context are identical or near-identical across many requests. Prompt caching lets the provider cache the KV-cache state for a shared prefix so repeated requests skip reprocessing that prefix, cutting both latency and the cost of the cached tokens.

To get value from this you need to structure your prompt so the stable parts come first and the variable parts (the specific user query, and ideally document ordering) come last:

def build_prompt(system_instructions, retrieved_chunks, user_query):
    # Stable prefix: system instructions + a fixed ordering of
    # frequently-retrieved chunks. Keep this identical across calls
    # whenever possible so the cache prefix matches.
    stable_prefix = system_instructions + "\n\n" + "\n\n".join(retrieved_chunks)
    # Variable suffix: changes every call, not cached.
    return stable_prefix + f"\n\nUser question: {user_query}"

The practical gotcha: if your retrieval step returns chunks in a different order every call, or interleaves per-user context before the retrieved documents, you break the shared prefix and the cache never hits. Put anything user-specific (name, account details, conversation history) after the retrieved context, not before it, and keep the ordering of frequently-retrieved chunks deterministic (sort by chunk ID or document ID rather than by raw similarity score, which jitters slightly between calls).

Prompt caching typically has a short TTL, often a few minutes, refreshed on each use. That is fine for a system prompt that gets hit constantly, but it means prompt caching alone will not save you on a corpus of long-tail queries that only get one request. It stacks with the retrieval and semantic caches above rather than replacing them: retrieval caching cuts vector search cost, prompt caching cuts the cost of reprocessing the resulting context tokens on the generation call.

Cache invalidation: the part that actually breaks in production

Every layer above is straightforward until a source document changes. Get invalidation wrong and RAG caching silently degrades your system's correctness while everything looks fast and healthy in your latency dashboards.

The core rule: invalidation cascades downstream, never upstream. When a document is updated:

1. Re-embed the changed chunks (document embedding cache entries for those chunks are invalidated by their new hash automatically, since the hash key changes with the content).
2. Invalidate any retrieval cache entries whose top-k results included the changed document. This requires tracking a reverse index: document ID -> which cached query keys returned it.
3. Invalidate semantic cache entries whose answer was derived from the changed document. Same reverse-index requirement.
4. Leave the query embedding cache alone. A query's embedding does not change because a document changed.

def invalidate_on_document_update(doc_id, reverse_index, retrieval_cache, semantic_cache):
    affected_query_keys = reverse_index.get(doc_id, [])
    for key in affected_query_keys:
        retrieval_cache.delete(key)
        semantic_cache.delete(key)
    reverse_index.clear(doc_id)

Building and maintaining this reverse index is the least glamorous part of RAG caching and the part teams skip first under deadline pressure. The shortcut that works reasonably well without a full reverse index: set conservative TTLs per layer, shortest on the semantic and retrieval caches (minutes to low hours), longer on the document embedding cache (which is content-addressed and self-invalidates on change anyway). TTL-based expiry is not as precise as event-driven invalidation, but it bounds your worst-case staleness to a known window, which is often good enough for internal tools and acceptable for customer-facing systems if your documents do not change faster than the TTL.

A layered caching architecture, put together

Here is how the four layers compose in a single request path:

1. Normalize and embed the query, checking the query embedding cache first.
2. Check the semantic cache against the query embedding. On a hit above threshold, and if the query has no time- or user-specific markers, return the cached answer and stop.
3. On a miss, check the retrieval cache for this query embedding. On a hit, skip vector search and reranking.
4. On a retrieval miss, run vector search and reranking, then populate the retrieval cache.
5. Build the generation prompt with a stable prefix (system instructions plus deterministically-ordered retrieved chunks) so the LLM provider's prompt cache can hit on the prefix.
6. Generate the answer, then write it into the semantic cache keyed on the query embedding.

Each layer has an independent hit rate and an independent cost-to-maintain. Semantic caching gives the biggest latency win (it skips retrieval and generation entirely) but carries the highest correctness risk and needs the most tuning. Prompt caching is the lowest-effort win if your provider supports it and your prompt structure is disciplined, but it only cuts generation-side cost, not retrieval latency. Retrieval caching sits in the middle: moderate effort, moderate savings, lower risk than semantic caching since generation still runs fresh.

Measuring whether caching is actually paying off

Before optimizing, instrument each layer separately rather than looking at end-to-end latency alone. Track, per layer: hit rate, and for the semantic cache specifically, a sampled accuracy check where you periodically re-run cached queries through the full pipeline and compare answers. A semantic cache with a 40% hit rate and a 2% mismatch rate against fresh answers is a different system than one with a 40% hit rate and a 15% mismatch rate, and end-to-end latency numbers alone will not tell you which one you have.

A reasonable rollout order for a team adding RAG caching to an existing system: start with the document embedding cache (zero correctness risk, pure ingestion cost savings), add prompt caching at the LLM layer (low risk, provider-managed), then retrieval caching (moderate effort, build the reverse index from day one rather than retrofitting it), and treat semantic caching as the last, highest-payoff, highest-risk layer to add once you have traffic data to tune the similarity threshold against.

FAQ

Does RAG caching reduce answer quality? Not if scoped correctly. Document and query embedding caches never change what gets retrieved or generated, they only avoid redundant computation of identical inputs. Retrieval and semantic caching can affect quality if invalidation is sloppy or the similarity threshold is too loose, which is why those two layers need the TTL and reverse-index discipline described above.

What is a good similarity threshold for semantic caching? There is no universal number. Cosine similarity thresholds in the 0.93 to 0.97 range are common starting points, but the right value depends on your embedding model and how semantically dense your query traffic is. Build a labeled set of duplicate and near-miss query pairs from real traffic and tune against that rather than trusting a fixed default.

Should I cache at the vector database level or in application code? Most vector databases do not offer semantic result caching out of the box; that layer usually needs to live in application code or a dedicated caching service in front of the retrieval call. Document and query embedding caches are simplest as a key-value store (Redis, or even a local disk cache for smaller corpora) sitting alongside your embedding pipeline, independent of the vector database itself.

How does RAG caching interact with per-user personalization? Keep personalization out of the cached prefix. If retrieval or generation depends on user-specific context (role, permissions, account state), either exclude those queries from the semantic and retrieval caches entirely, or include the relevant personalization dimension in the cache key so different user segments do not collide on the same cache entry.

Is prompt caching enough on its own, without the other layers? For low-traffic or highly diverse query sets, prompt caching alone (on a stable system prompt) is often the right stopping point since the retrieval and semantic layers need decent traffic volume to build up meaningful hit rates. For high-traffic, FAQ-heavy workloads like support bots, skipping retrieval and semantic caching leaves significant latency and cost on the table.

How do I handle caching for streaming responses? Cache the complete generated answer after the stream finishes, not the stream itself. On a cache hit, you can replay the cached text as a simulated stream (chunk it and send with small delays) if your frontend expects streaming behavior, or return it as a single response if the client handles both cases.