Semantic Caching for LLM Applications

Semantic caching llm systems solve a problem that plain string-match caching cannot: two users asking "how do I reset my password" and "what's the process for changing my password" should hit the same cached answer, even though the text is different. A traditional cache keyed on an exact prompt hash misses this every time, so you keep paying for and waiting on inference for questions you've already answered. Semantic caching stores past query-response pairs as embeddings and serves a cached response whenever a new query is close enough in vector space to a previous one, cutting both cost and latency for repetitive traffic.

This matters most in production systems with high query overlap: customer support bots, internal documentation assistants, RAG pipelines over a fixed knowledge base, and any app where the same handful of intents get phrased a hundred different ways. If your traffic is mostly unique, one-off prompts, semantic caching won't help much. If it's repetitive with paraphrasing, it can meaningfully cut your API spend and shave hundreds of milliseconds off response time.

What semantic caching actually is

A normal cache (Redis, Memcached, an LRU dict) works on exact keys. You hash the input, look it up, and get a hit only on an identical string. That's fine for deterministic function calls but useless for natural language, where the same intent has near-infinite phrasings.

Semantic caching replaces the exact-match key with a similarity search:

1. Embed the incoming query with a sentence embedding model.
2. Search a vector store for previously cached queries whose embeddings are within some similarity threshold.
3. If a close-enough match exists, return its stored response (a cache hit), optionally skipping the LLM call entirely.
4. If no match is close enough, call the LLM, store the new query embedding and response, and return the fresh answer (a cache miss).

The core tradeoff is the similarity threshold. Set it too loose and you return wrong answers for queries that aren't actually the same intent (a false hit, the dangerous failure mode). Set it too tight and you rarely hit the cache, losing the cost benefit (a false miss, the safe but wasteful failure mode).

Minimal implementation with a vector store

Here's a working semantic cache using a local vector index. This example uses a simple in-memory approach with numpy for clarity; swap in Redis, Qdrant, Pinecone, or pgvector for production scale.

import numpy as np
from openai import OpenAI

client = OpenAI()

class SemanticCache:
    def __init__(self, similarity_threshold=0.92):
        self.threshold = similarity_threshold
        self.queries = []       # original query text
        self.embeddings = []    # list of numpy vectors
        self.responses = []     # cached LLM responses

    def _embed(self, text):
        resp = client.embeddings.create(
            model="text-embedding-3-small",
            input=text
        )
        vec = np.array(resp.data[0].embedding)
        return vec / np.linalg.norm(vec)  # normalize for cosine sim

    def _cosine_sim(self, a, b):
        return float(np.dot(a, b))

    def lookup(self, query):
        if not self.embeddings:
            return None
        query_vec = self._embed(query)
        sims = [self._cosine_sim(query_vec, e) for e in self.embeddings]
        best_idx = int(np.argmax(sims))
        best_sim = sims[best_idx]
        if best_sim >= self.threshold:
            return {
                "response": self.responses[best_idx],
                "matched_query": self.queries[best_idx],
                "similarity": best_sim
            }
        return None

    def store(self, query, response):
        self.queries.append(query)
        self.embeddings.append(self._embed(query))
        self.responses.append(response)


def ask(cache, query, model="gpt-4o-mini"):
    hit = cache.lookup(query)
    if hit:
        print(f"cache hit ({hit['similarity']:.3f}) matched: {hit['matched_query']}")
        return hit["response"]

    completion = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": query}]
    )
    answer = completion.choices[0].message.content
    cache.store(query, answer)
    return answer

Running this against paraphrased queries shows the behavior:

cache = SemanticCache(similarity_threshold=0.90)

print(ask(cache, "How do I reset my password?"))
print(ask(cache, "What's the process for changing my password?"))  # likely a hit
print(ask(cache, "How do I delete my account?"))  # a miss, different intent

The in-memory list works for demos but doesn't scale past a few thousand entries because the similarity search is O(n). Production systems need an actual vector index.

Production setup with Redis and RedisVL

Redis is a common choice because most teams already run it for regular caching, and its vector search module handles the approximate nearest neighbor lookup efficiently.

import redis
from redisvl.index import SearchIndex
from redisvl.query import VectorQuery
from redisvl.schema import IndexSchema

schema = IndexSchema.from_dict({
    "index": {"name": "llm_semantic_cache", "prefix": "cache"},
    "fields": [
        {"name": "query_text", "type": "text"},
        {"name": "response_text", "type": "text"},
        {
            "name": "query_vector",
            "type": "vector",
            "attrs": {
                "dims": 1536,
                "distance_metric": "cosine",
                "algorithm": "hnsw",
                "datatype": "float32"
            }
        }
    ]
})

client = redis.Redis(host="localhost", port=6379)
index = SearchIndex(schema, client)
index.create(overwrite=False)

def cache_lookup(query_vec, threshold=0.08):
    # RedisVL returns distance, not similarity; smaller = closer
    q = VectorQuery(
        vector=query_vec,
        vector_field_name="query_vector",
        return_fields=["query_text", "response_text"],
        num_results=1
    )
    results = index.query(q)
    if results and float(results[0]["vector_distance"]) <= threshold:
        return results[0]["response_text"]
    return None

def cache_store(query_text, query_vec, response_text):
    index.load([{
        "query_text": query_text,
        "response_text": response_text,
        "query_vector": np.array(query_vec, dtype=np.float32).tobytes()
    }])

Note the distance-vs-similarity flip: RedisVL's HNSW index with cosine distance returns smaller numbers for closer matches, the opposite of the cosine similarity used in the numpy example. Get this backwards and your threshold logic silently inverts, which is a common source of "the cache never hits" or "the cache hits everything" bugs.

Choosing a similarity threshold

There's no universal number; it depends on your embedding model and your tolerance for wrong answers. A practical way to tune it:

1. Collect a set of real query pairs from logs: known-duplicate pairs (same intent, different wording) and known-distinct pairs (different intent, similar wording).
2. Embed both sets and compute similarity scores.
3. Plot the distributions. You want the threshold sitting in the gap between the "distinct" cluster's high end and the "duplicate" cluster's low end.
4. If the clusters overlap heavily, semantic caching probably won't be reliable for that traffic, exact-match caching or no caching may be safer.

As a starting point, cosine similarity thresholds in the 0.90-0.95 range with OpenAI's text-embedding-3-small are common, but validate against your own data before trusting it in production. Domains with high lexical overlap but different meaning (legal text, medical dosing instructions, financial figures) need tighter thresholds or should avoid semantic caching for the final answer entirely, even if they use it for retrieval.

Cache invalidation and staleness

Semantic caches go stale the same way any cache does, but the failure is quieter because a paraphrase of an outdated question can silently return an outdated answer. Handle this with:

• TTL on entries. Expire cached responses after a fixed window (minutes for fast-changing data like pricing or inventory, days for stable content like documentation).
• Versioned cache keys. Tie the cache namespace to a content version (a docs build hash, a product catalog version). Bump the version and the old cache becomes unreachable without deleting anything.
• Explicit invalidation hooks. When source content changes (a KB article is edited, a policy updates), delete or flag the affected cache entries rather than waiting for TTL.
• Confidence-aware serving. Log the similarity score with every cache hit and alert if the average similarity of hits creeps down over time, an early signal that traffic patterns have shifted, or that data has drifted.

Where semantic caching fits with RAG

In a retrieval-augmented generation pipeline, you can cache at two different layers:

• Query-to-answer caching: cache the final generated response, keyed on the user's question. This gives the biggest cost win (skips both retrieval and generation) but carries the most risk if underlying documents change.
• Query-to-retrieval caching: cache just the retrieved chunks for a given query, still calling the LLM to generate. This is safer for volatile knowledge bases because the generation step always reflects the latest system prompt and any freshly injected context, while still saving the retrieval and embedding cost.

A hybrid approach: cache retrieval results aggressively (short TTL, loose threshold) and cache final answers conservatively (only for queries confirmed stable, like FAQ-style questions).

Measuring whether it's worth it

Before rolling this into production, instrument three numbers:

• Hit rate: percentage of queries served from cache. Below 15-20% and the added complexity (vector store, threshold tuning, invalidation logic) usually isn't worth it.
• False hit rate: sampled manual review of cache hits to check whether the cached answer actually matches the new query's intent. This is the metric that catches threshold-too-loose problems before users do.
• Latency delta: p50/p95 response time for cache hits versus misses. A vector lookup plus embedding call still takes time, if it's not meaningfully faster than a small model's generation, the latency win disappears even if the cost win remains.

A cache with a 40% hit rate and a 2% false hit rate is generally a solid tradeoff for a support bot. A 40% hit rate with a 15% false hit rate is not, tighten the threshold or switch to retrieval-only caching until you fix it.

FAQ

Does semantic caching work with any embedding model? Yes, but the threshold you pick is specific to the model and even the model version. Switching embedding models means re-tuning thresholds from scratch, don't assume a 0.92 cutoff transfers between models.

Should I cache streaming responses? You can, but store the full assembled text and replay it as a simulated stream on a hit, or serve it non-streamed. Trying to cache partial stream chunks adds complexity for little benefit.

Does semantic caching reduce hallucinations? No, it has nothing to do with accuracy. If the original cached response was wrong, the cache just serves that wrong answer faster and to more people. Validate answers before caching them, especially for auto-generated content that later gets treated as ground truth.

Is semantic caching different from prompt caching offered by LLM providers? Yes. Provider-side prompt caching (like reusing a cached prefix of a long system prompt or context) speeds up processing of repeated prefixes within the model's own inference. Semantic caching sits in front of the model entirely and can skip the API call altogether. They're complementary: use provider prompt caching for long shared context, and semantic caching to avoid calling the model at all for repeat questions.

What's a reasonable place to start if I have no infrastructure yet? Start with a managed vector store you already use for RAG (pgvector, Qdrant, Redis) rather than standing up something new. Add a semantic cache as a thin layer in front of your existing LLM call, log everything, and only tune the threshold once you have a few hundred real hit/miss examples to look at.

Can semantic caching leak data between users? It can if you don't scope the cache correctly. Multi-tenant applications need cache keys or namespaces scoped per tenant (and often per user, for personalized answers), otherwise one customer's cached response can leak into another's session. Treat cache scoping with the same care as any other data isolation boundary in a multi-tenant system.