Semantic Caching for RAG: Reducing Redundant LLM Calls

Why your RAG app is paying for the same answer twice

Picture a support chatbot built on a RAG pipeline. One user asks "How do I reset my password?" Another asks "What's the process to change my password?" A third types "password reset steps." Semantically, these are the same question. But if your caching strategy relies on exact string matching, your system treats all three as brand-new requests. It re-embeds the query, re-runs vector search, re-assembles context, and re-calls the LLM — three times over for what is functionally one answer.

This is the hidden tax most RAG systems pay without realizing it. Traditional caches key off exact strings or hashes, so they only catch requests that are character-for-character identical. In practice, real users rarely phrase things identically. They paraphrase, they add filler words, they make typos, they ask in different languages. Exact-match caching catches almost none of that variance, which means your most expensive resource — the LLM call — gets invoked for queries you have effectively already answered.

Semantic caching fixes this by matching on meaning instead of text. Instead of asking "have I seen this exact string before?" it asks "have I seen a query this *close in meaning* before?" When the answer is yes, you skip the LLM call (and often the retrieval step too) and return a cached response in milliseconds instead of seconds. For any RAG application running at real user volume, this is one of the highest-leverage optimizations you can ship — and it's the kind of production concern we dig into in depth in our Introduction to RAG course.

What semantic caching actually is

At its core, semantic caching stores three things per cache entry: the original query, an embedding vector representing that query's meaning, and the response that was generated for it (sometimes along with the retrieved context, for auditability). When a new query comes in, instead of hashing the raw text and doing a dictionary lookup, you:

1. Embed the incoming query using the same embedding model used for your vector store.
2. Search the cache's vector index for the nearest neighbor(s) to that embedding.
3. Check whether the similarity score crosses a threshold you've defined (say, cosine similarity above 0.92).
4. If it crosses the threshold, treat it as a cache hit and return the stored response.
5. If not, treat it as a miss — run the full RAG pipeline, then write the new query/embedding/response triple back into the cache.

This is conceptually a second, smaller vector database sitting in front of your main retrieval pipeline. The main vector store holds your knowledge base chunks; the semantic cache holds a history of question-answer pairs your system has already handled. It's a cache in the traditional sense — it exists purely to avoid redoing expensive work — but the lookup key is fuzzy rather than exact.

The distinction matters because it changes what "correctness" means for the cache. A traditional cache is either right or wrong — the key either matches or it doesn't. A semantic cache has a tunable notion of "close enough," and getting that threshold wrong in either direction creates real problems, which we'll get into shortly.

Why exact-match caching fails for LLM workloads

Standard caching techniques — Redis with a hash of the request body, an LRU cache keyed by the literal prompt string — work great for deterministic, structured requests. An API call for GET /users/42 will always look identical to GET /users/42. But natural language queries don't behave like API paths.

Consider the surface-level variation in how people ask the same question:

• "What's your refund policy?"
• "Can I get my money back?"
• "how do refunds work"
• "Refund policy??"
• "I want a refund, what are the rules"

An exact-match or even a normalized-string-match cache (lowercasing, stripping punctuation) will treat every one of these as a cache miss relative to the others. The semantic content is nearly identical, but the surface form diverges enough that string-based techniques can't reconcile them. You'd need the user to type verbatim what a previous user typed, which essentially never happens outside of scripted, structured inputs.

This is precisely the gap embeddings are built to close. Embedding models are trained so that semantically similar sentences land close together in vector space, regardless of exact phrasing. That's the same property that makes vector search useful for document retrieval in RAG — and it's directly reusable for caching queries, since a cache lookup is just retrieval against a much smaller, self-generated corpus of past questions.

The core architecture: embedding, index, and threshold

A minimal semantic cache needs three components, and you likely already have two of them if you've built a RAG pipeline.

An embedding model. Reuse the same model you use to embed your document chunks (e.g., an OpenAI embedding model, a Sentence-Transformers model, or a Cohere embedding endpoint). Using the same model for both the knowledge base and the cache keeps your architecture simple, though it isn't strictly required — the cache's embedding space just needs to be internally consistent.

A vector index for cache entries. This can be a lightweight in-memory index (FAISS, Annoy) for smaller deployments, or a dedicated vector database (Qdrant, Weaviate, Milvus, pgvector) if you want persistence, TTL support, and horizontal scale. Many teams use the same vector database they already run for retrieval, just in a separate collection/namespace so cache entries never mix with document chunks.

A similarity threshold and eviction policy. This is the part that requires actual engineering judgment rather than just wiring components together. The threshold decides how aggressively you collapse "different" queries into the same cached answer. The eviction policy decides how stale entries get pruned so the cache doesn't grow unbounded or serve outdated information forever.

Here's a simplified but functional implementation using a Python-style pseudo-interface that mirrors what libraries like GPTCache expose:

import numpy as np
from dataclasses import dataclass
from typing import Optional

@dataclass
class CacheEntry:
    query: str
    embedding: np.ndarray
    response: str
    hits: int = 0

class SemanticCache:
    def __init__(self, embed_fn, similarity_threshold=0.92, max_entries=5000):
        self.embed_fn = embed_fn
        self.threshold = similarity_threshold
        self.max_entries = max_entries
        self.entries: list[CacheEntry] = []

    def _cosine_sim(self, a: np.ndarray, b: np.ndarray) -> float:
        return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

    def lookup(self, query: str) -> Optional[str]:
        query_vec = self.embed_fn(query)
        best_score, best_entry = -1.0, None

        for entry in self.entries:
            score = self._cosine_sim(query_vec, entry.embedding)
            if score > best_score:
                best_score, best_entry = score, entry

        if best_entry is not None and best_score >= self.threshold:
            best_entry.hits += 1
            return best_entry.response

        return None

    def store(self, query: str, response: str):
        if len(self.entries) >= self.max_entries:
            # evict least-used entry (simplest LFU-style policy)
            self.entries.sort(key=lambda e: e.hits)
            self.entries.pop(0)

        embedding = self.embed_fn(query)
        self.entries.append(CacheEntry(query, embedding, response))


def answer_query(query: str, cache: SemanticCache, rag_pipeline_fn):
    cached = cache.lookup(query)
    if cached is not None:
        return cached, True  # cache hit, no LLM call

    response = rag_pipeline_fn(query)  # full retrieve + generate
    cache.store(query, response)
    return response, False

This linear-scan version is fine for a few thousand entries and useful for understanding the mechanics, but production systems swap the entries list for an actual approximate-nearest-neighbor index once the cache grows past tens of thousands of entries, since a linear scan becomes the new bottleneck.

Picking a similarity threshold without guessing

The threshold is the single most consequential number in this whole system, and there's no universal correct value — it depends on your domain, your tolerance for wrong answers, and how varied your users' phrasing is.

Set the threshold too low (accepting weaker matches as hits) and you get false cache hits: a user asks about canceling a subscription, and the cache serves them the cached answer for a different-but-nearby question about downgrading a subscription. In a support or medical or legal context, this isn't a minor UX papercut — it's actively wrong information delivered confidently.

Set the threshold too high (only accepting near-identical matches) and your cache hit rate collapses toward zero, defeating the purpose. You'll pay for the vector search step on every query while capturing almost none of the savings.

A few practical approaches to tuning this in production:

• Start conservative and loosen gradually. Begin around 0.95 cosine similarity, monitor hit rate and manually audit a sample of hits for correctness, then lower the threshold in small increments while re-auditing.
• Segment by query type. FAQ-style factual queries (fixed answers, low ambiguity) can tolerate a lower threshold than open-ended or multi-part queries where small wording changes can flip intent.
• Use a two-stage check. Retrieve the top candidate by embedding similarity, then run a fast, cheap secondary check — a smaller model or lightweight heuristic — that verifies the cached answer still applies before returning it. This adds latency but reduces the blast radius of a bad threshold.
• Log every hit with its similarity score. When something goes wrong downstream, you want to be able to look back and see exactly which score threshold let a bad match through.

Cache invalidation: the problem semantic caching makes harder

Classic cache invalidation is famously one of the two hard problems in computer science, and semantic caching inherits that problem with an added twist: it's not just about *when* to invalidate, but *how much of the cache* a single underlying data change should invalidate.

In a normal key-value cache, if the "refund policy" document changes, you invalidate the one cache entry keyed to "refund policy." In a semantic cache, dozens of differently-phrased queries may all map near that same region of embedding space and all be serving the now-stale answer. You need a strategy to invalidate the whole cluster, not just one entry.

Practical strategies teams use:

• TTL on every entry. Simplest approach — every cached response expires after a fixed window (minutes for fast-changing data, days for stable reference material). This bounds staleness without requiring you to track document-to-cache-entry relationships.
• Tag cache entries with source document IDs. When you generate a cached response, record which retrieved chunks contributed to it. When those source documents get updated in your knowledge base, invalidate every cache entry tagged with that document ID, even if the query text looked completely different.
• Version-stamp the whole cache on ingestion runs. If your RAG knowledge base is rebuilt or re-indexed on a schedule, bump a global cache version and treat all older entries as stale. This is blunt but simple, and often good enough for internal tools or smaller knowledge bases.
• Active monitoring over passive trust. Don't assume the cache is correct forever just because nothing has thrown an error — periodically re-run a sample of cached queries through the live pipeline and diff the answers.

The right choice depends on how often your underlying data changes. A support knowledge base that updates weekly can get away with generous TTLs. A RAG system over live pricing or inventory data needs tight invalidation tied directly to the data source, or semantic caching will actively serve wrong answers faster than a non-cached system would.

Where semantic caching fits in the RAG pipeline

It's worth being precise about *where* the cache check happens, because there are actually two reasonable insertion points, and they save different things.

Full-pipeline caching intercepts the query before retrieval even happens. A cache hit skips embedding-for-search, vector retrieval, re-ranking, context assembly, and the generation call entirely. This is the biggest win and what most people mean by "semantic caching for RAG," since it collapses the entire request-response cycle to a single cache lookup.

Retrieval-only caching caches just the retrieved chunks for a given query, but still calls the LLM fresh every time. This is useful when your generation step needs to incorporate something that changes per-request — conversation history, user-specific personalization, a system prompt that varies by tenant — where reusing a full generated response would be wrong, but the expensive retrieval step (especially if it involves re-ranking or multiple retrieval passes) is still safe to reuse.

Most production systems benefit from layering both: a full-response cache for genuinely repeatable queries (FAQs, common lookups), and a retrieval-level cache underneath for queries where full-response caching is too risky due to per-user context, but expensive vector search can still be shared across similar queries.

Measuring whether it's actually working

Shipping a semantic cache without instrumentation is shipping a black box. At minimum, track:

• Hit rate — the percentage of incoming queries served from cache. This is your headline savings metric.
• Latency delta — median and p95 response time for cache hits versus cache misses. This quantifies the user-facing speed win.
• Cost delta — LLM tokens (and therefore spend) avoided per cache hit, aggregated over a billing period.
• False-hit rate — the percentage of "hits" that, on manual or automated review, actually returned an inappropriate answer for the query asked. This is the metric most teams skip and the one most likely to bite you.
• Threshold drift over time — as your user base or query patterns shift (new features, new terminology), a threshold tuned for last quarter's traffic may not hold up. Revisit it periodically rather than setting it once.

A useful mental model: hit rate tells you how much money you're saving, false-hit rate tells you how much trust you're spending to save it. Optimize for the ratio between the two, not for hit rate alone.

# lightweight instrumentation wrapper around the cache from earlier
class InstrumentedCache(SemanticCache):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.hits = 0
        self.misses = 0

    def lookup(self, query: str):
        result = super().lookup(query)
        if result is not None:
            self.hits += 1
        else:
            self.misses += 1
        return result

    @property
    def hit_rate(self) -> float:
        total = self.hits + self.misses
        return self.hits / total if total else 0.0

Wire something like this into your existing observability stack (Prometheus, Datadog, or even structured logs shipped to a warehouse) so hit rate and false-hit rate show up next to your other latency and cost dashboards — not as a separate, easily-ignored side metric.

Common pitfalls worth naming directly

Treating the cache as infinitely trustworthy. Once a wrong answer gets cached, it will confidently serve that wrong answer to every subsequent similar query until it's evicted or invalidated — arguably worse than a single bad LLM response, because it's now systematically wrong rather than randomly wrong.

Ignoring domain-specific negation and specificity. Embedding models can place "how do I enable notifications" and "how do I disable notifications" closer together than you'd expect, since they're topically similar even though the intent is opposite. A naive threshold can conflate these. Domain-specific evaluation of your threshold, not just generic benchmarks, is essential.

Forgetting multi-turn context. A query like "what about for enterprise plans" only makes sense with the preceding conversation turn attached. Caching that fragment on its own, divorced from context, produces nonsense hits later. Either include relevant context in what you embed for cache lookups, or scope semantic caching to single-turn, self-contained queries only.

Skipping the cold-start period. A brand-new semantic cache has a 0% hit rate by definition. Don't judge the technique a failure in week one — track the hit rate curve over the first few weeks as the cache populates with real traffic patterns.

Bringing it together

Semantic caching is one of those optimizations that sounds like a minor infrastructure detail but compounds into a serious cost and latency advantage once your RAG system has real traffic. The mechanics are approachable — embed the query, search a small vector index of past queries, return the cached response above a similarity threshold — but the engineering judgment lives in the details: picking a threshold that matches your domain's risk tolerance, designing invalidation that tracks source-document changes rather than just time, and instrumenting false-hit rate so you catch problems before users do.

None of this replaces a solid retrieval and generation pipeline — it sits in front of one. If you're still working through chunking strategy, retrieval quality, or re-ranking before you get to this stage, that foundational material is exactly what we cover in Introduction to RAG, and it's the natural starting point before layering caching, evaluation, and other production concerns on top.