Caching LLM Calls in LangChain

LangChain caching intercepts a chat model call before it hits the provider, checks whether the exact same prompt and parameters were sent before, and if so returns the stored response instead of paying for another API round trip. For any app that repeats prompts, think chatbots answering FAQs, RAG pipelines reprocessing the same documents, or test suites that call an LLM on every run, caching turns a paid, slow network call into a free, instant lookup. This article walks through every cache backend LangChain ships with, how to scope caching per chain instead of globally, and where caching quietly breaks (streaming, non-deterministic temperature, and tool calls).

How LangChain Caching Works Under the Hood

A LangChain chat model wraps its generate call with a cache lookup. The cache key is built from the serialized prompt, the model name, and the model parameters (temperature, max tokens, and so on). If a matching key exists in the cache store, LangChain returns the cached ChatResult without touching the network. If not, it calls the provider, stores the result, and returns it.

This has three direct consequences worth internalizing before you wire anything up:

• Caching is exact-match by default. Changing one word in the prompt, or bumping temperature from 0 to 0.1, produces a cache miss.
• Caching is deterministic-friendly. If you run with temperature=0, repeated identical prompts really do produce the same output, so caching is safe. At higher temperatures, caching still works mechanically, but you're now caching one specific sample from a distribution, which can feel wrong for creative tasks.
• The cache lives outside your chain's business logic. You turn it on globally or scope it to specific model instances, and the rest of your code doesn't need to know it exists.

Setting a Global Cache

The simplest way to start is a global, process-wide cache. Every chat model instantiated after this call automatically uses it.

from langchain_core.globals import set_llm_cache
from langchain_core.caches import InMemoryCache

set_llm_cache(InMemoryCache())

InMemoryCache is a plain Python dict under the hood. It's perfect for local development and one-off scripts, but it disappears the moment your process restarts, and it doesn't share state across multiple workers. Test it with a simple loop:

import time
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

start = time.time()
response_one = llm.invoke("Explain the CAP theorem in two sentences.")
print(f"First call: {time.time() - start:.2f}s")

start = time.time()
response_two = llm.invoke("Explain the CAP theorem in two sentences.")
print(f"Second call: {time.time() - start:.2f}s")

assert response_one.content == response_two.content

Run this and the second call returns near-instantly, often under a millisecond, because it never leaves the process.

Persisting the Cache with SQLite

InMemoryCache doesn't survive a restart, which is a dealbreaker for anything beyond a quick experiment. SQLiteCache writes to a local file, so the cache persists across runs, deployments, and crashes. It's the right default for a single-server app, a batch job, or a CI pipeline that reruns the same evaluation prompts.

from langchain_core.globals import set_llm_cache
from langchain_community.cache import SQLiteCache

set_llm_cache(SQLiteCache(database_path=".langchain_cache.db"))

Because it's backed by an actual database file, you can inspect it directly:

sqlite3 .langchain_cache.db "select count(*) from full_llm_cache;"

A common pattern is to commit an empty .gitignore entry for the cache file so it doesn't bloat your repo, but keep it around locally during development so repeated test runs don't re-burn API credits:

# .gitignore
.langchain_cache.db

For a batch job that processes thousands of documents and might be rerun after a crash, SQLite caching alone can save real money. If a job dies at document 800 out of 1000, rerunning it from the top only pays for the 200 documents that weren't cached.

Scoping Cache to a Single Model Instead of Globally

set_llm_cache applies to every model in the process, which is often too blunt. If you have one chat model doing deterministic classification (safe to cache) and another doing creative generation at temperature 0.9 (where caching a single sample is misleading), you want per-instance control.

Every chat model accepts a cache parameter directly:

from langchain_openai import ChatOpenAI
from langchain_core.caches import InMemoryCache

shared_cache = InMemoryCache()

classifier_llm = ChatOpenAI(model="gpt-4o-mini", temperature=0, cache=shared_cache)
creative_llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.9, cache=False)

Setting cache=False on a specific model disables caching for that instance even if a global cache is set elsewhere in the process. This is the pattern to reach for whenever part of your pipeline needs fresh output every time (a random idea generator, a chatbot's small talk) while another part benefits from reuse (intent classification, entity extraction, structured summarization).

Redis Caching for Multi-Process and Multi-Server Apps

SQLite works for a single machine, but the moment you run multiple workers, containers, or servers behind a load balancer, you need a shared cache store they can all read and write. Redis is the standard choice here because it's fast, supports TTLs natively, and most teams already run it for sessions or rate limiting.

from langchain_core.globals import set_llm_cache
from langchain_community.cache import RedisCache
import redis

redis_client = redis.Redis(host="localhost", port=6379, db=0)
set_llm_cache(RedisCache(redis_client, ttl=3600))

The ttl parameter (in seconds) is important in production. Without it, cached entries live forever, which is fine for static reference content but dangerous if the underlying prompt logic changes and old cached answers should expire. A one-hour TTL for a customer support bot, or a 24-hour TTL for a documentation Q&A system, keeps the cache useful without serving stale answers indefinitely.

Verify the cache is actually shared across two separate processes by running the same prompt from two different Python scripts pointed at the same Redis instance; the second script's call should return instantly regardless of which process made the first call.

Semantic Caching: Matching Similar, Not Identical, Prompts

Exact-match caching misses an enormous number of real-world duplicates. "What's the refund policy?" and "How do refunds work?" are semantically the same question but produce two different cache keys under standard caching. Semantic caching solves this by embedding the incoming prompt, comparing it against embeddings of previously cached prompts, and returning the cached answer if the similarity score clears a threshold.

RedisSemanticCache from langchain_community pairs Redis's vector search with an embedding model:

from langchain_core.globals import set_llm_cache
from langchain_community.cache import RedisSemanticCache
from langchain_openai import OpenAIEmbeddings

set_llm_cache(
    RedisSemanticCache(
        redis_url="redis://localhost:6379",
        embedding=OpenAIEmbeddings(),
        score_threshold=0.2,
    )
)

The score_threshold controls how aggressive the matching is. A lower distance threshold means only near-identical prompts match, which is safer but catches fewer duplicates. A higher threshold catches more paraphrases but risks returning an answer to a question the user didn't quite ask. Start conservative (a tight threshold), run it against a sample of real user queries, and loosen it only after manually reviewing a batch of the matches it produces.

Semantic caching costs an embedding call on every request, even on a cache miss, so it's not free. It pays off specifically in high-traffic, high-repetition scenarios like customer support widgets or internal FAQ bots, where the same handful of intents get rephrased constantly. For low-traffic or highly varied prompts (code generation, long-form writing), the embedding overhead usually isn't worth it and exact-match caching with Redis or SQLite is the better fit.

Caching Inside a Chain, Not Just a Bare Model

Caching applies at the chat model level, which means it works transparently even when the model is buried inside a larger LCEL chain, an agent, or a RetrievalQA-style pipeline. You don't need to change chain code; you only need to configure the cache on the underlying model before building the chain.

from langchain_core.globals import set_llm_cache
from langchain_community.cache import SQLiteCache
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from langchain_core.output_parsers import StrOutputParser

set_llm_cache(SQLiteCache(database_path=".langchain_cache.db"))

prompt = ChatPromptTemplate.from_template("Summarize this in one sentence: {text}")
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
chain = prompt | llm | StrOutputParser()

text = "LangChain is a framework for building applications powered by language models."
result_one = chain.invoke({"text": text})
result_two = chain.invoke({"text": text})

The second chain.invoke call hits the cache at the model step, even though the cache was never mentioned in the chain definition itself. This is useful for RAG pipelines: the retrieval step still runs every time (unless you cache that separately), but the expensive generation step gets deduplicated whenever the retrieved context and question combination repeats.

Where Caching Silently Breaks

Three situations catch teams off guard after they've shipped caching to production.

Streaming responses bypass the cache differently depending on backend and version. When you call .stream() instead of .invoke(), the cache still stores the final assembled response, but on a cache hit, LangChain has to replay the cached text as a fake stream rather than truly streaming token-by-token from the provider. Functionally the output is correct, but if your UI relies on realistic token-by-token timing for a "typing" effect, a cache hit will feel instant and jarring compared to a cache miss. Test both paths explicitly rather than assuming streaming behaves identically with and without a hit.

Tool calls and function calling add non-determinism to the cache key in ways that aren't obvious. If your chat model is bound to tools with .bind_tools(), the tool schema becomes part of what's hashed into the cache key. Change a tool's description or add a new tool to the list, even one unrelated to the current prompt, and every previously cached entry for that model becomes a permanent miss. This isn't a bug, it's correct behavior, since the model's available actions did change, but it means a routine tool schema tweak silently invalidates your entire cache and momentarily spikes your API bill.

Temperature above zero makes caching feel wrong even when it's working correctly. If a user asks for "three creative taglines" and gets a cached response identical to what someone else got minutes earlier, that's a caching win from a cost perspective but a product regression from a variety perspective. Reserve caching for temperature-zero, deterministic, or FAQ-style calls, and explicitly disable it (cache=False) on any model instance meant to produce varied output.

Testing That Caching Is Actually Working

Don't just assume caching is active; verify it. Time the first and second calls to the same prompt, and inspect the cache backend directly.

import time
from langchain_core.globals import set_llm_cache, get_llm_cache
from langchain_core.caches import InMemoryCache
from langchain_openai import ChatOpenAI

set_llm_cache(InMemoryCache())
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

prompt = "List three benefits of unit testing."

t0 = time.time()
llm.invoke(prompt)
first_call_seconds = time.time() - t0

t0 = time.time()
llm.invoke(prompt)
second_call_seconds = time.time() - t0

print(f"First: {first_call_seconds:.3f}s, second: {second_call_seconds:.3f}s")
assert second_call_seconds < first_call_seconds / 5, "Cache does not appear to be hitting"

cache = get_llm_cache()
print(f"Cache backend in use: {type(cache).__name__}")

For SQLite or Redis, add a check that queries the store directly for a nonzero entry count after your test suite runs, so a regression that silently disables caching (for example, someone accidentally removing the set_llm_cache call during a refactor) gets caught in CI instead of showing up as a surprise API bill next month.

Choosing a Cache Backend

• Use InMemoryCache for local scripts, notebooks, and quick experiments where persistence doesn't matter.
• Use SQLiteCache for single-server apps, batch jobs, and CI pipelines that need the cache to survive a restart but don't need to share it across machines.
• Use RedisCache once you run more than one worker or server process and need every process reading from the same cache.
• Use RedisSemanticCache only when you have measured, high-repetition, paraphrase-heavy traffic (support bots, FAQ assistants) where the embedding overhead is clearly paid back by additional cache hits over exact matching.

Start with the simplest backend that solves the problem you actually have. Most teams over-engineer this: they reach for semantic caching before they've even turned on exact-match caching with a shared Redis instance, and end up paying for embeddings without first capturing the much larger, much simpler win of deduplicating literal repeat requests.

FAQ

Does LangChain caching work with every model provider? Yes. Caching operates at the LangChain chat model wrapper level, not inside the provider's API, so it works identically whether you're calling OpenAI, Anthropic, Google, or a local model served through Ollama. The cache key includes the model name and parameters, so switching providers or model versions naturally produces new cache entries rather than incorrectly reusing an old provider's output.

Will caching return stale answers if I update my prompt template? No, as long as the actual rendered prompt text changes. The cache key is built from the fully rendered prompt sent to the model, not the template itself, so editing ChatPromptTemplate text changes what gets hashed and produces a fresh cache miss. Stale answers only become a risk with a semantic cache, where a similarity threshold that's too loose can match a new question to an old, no-longer-accurate cached answer.

Should I cache retrieval results too, or only the LLM call? Cache them separately if your retrieval step is expensive (a slow vector database query, a rerank step, or an external API call). LangChain's LLM caching only covers the model call itself; retrieval results need their own caching layer, typically a simple key-value store keyed on the query text plus any filters, sitting in front of your retriever.

Does caching help reduce hallucinations or improve answer quality? No. Caching is purely a cost and latency optimization; it returns a previously generated answer verbatim, so it neither improves nor degrades the underlying quality of that answer. If a cached response was wrong the first time, caching will keep serving that same wrong answer on every subsequent identical request until the cache entry expires or is cleared.

How do I clear the cache during development? For InMemoryCache, just restart your process. For SQLiteCache, delete the database file or run a DELETE FROM full_llm_cache; against it directly. For RedisCache or RedisSemanticCache, flush the relevant Redis keys with redis-cli --scan --pattern "langchain*" | xargs redis-cli del, or set a short TTL during development so stale entries expire on their own without manual cleanup.

Is it safe to cache calls that include user-specific or sensitive data? Be careful here. If a prompt embeds personal information (a user's name, account details, or private documents), a shared cache means a second user who happens to send a similar or matching prompt could receive a cached response containing another user's data. Scope caching to prompts that are genuinely generic and non-personal, or namespace your cache keys by user or tenant ID so entries never cross that boundary.