Keeping RAG Fresh: Incremental Indexing and Cache Invalidation

RAG incremental indexing means updating your vector index with only the documents that actually changed, instead of re-embedding the entire corpus on a schedule. Paired with deliberate cache invalidation, it is the difference between a RAG system that answers from yesterday's data and one that reflects an edit within minutes, at a fraction of the embedding cost. This article covers the mechanics that make it work in production: stable chunk IDs, content hashing, diff-and-upsert loops, delete handling, the cache layers that quietly serve stale answers long after the index is fresh, and the metrics that prove your pipeline is keeping up.

Why RAG incremental indexing beats nightly rebuilds

The first version of every RAG pipeline is a batch job: crawl everything, chunk everything, embed everything, overwrite the index. That design is easy to reason about, and it fails in predictable ways once the corpus grows or the change rate starts to matter.

• Cost tracks corpus size, not change rate. If half a percent of your documents change per day, a nightly rebuild pays to re-embed the other 99.5 percent for nothing.
• Freshness is capped by the batch interval. An engineer fixes a wrong runbook step at 10am, and the assistant keeps recommending the broken step until tomorrow's rebuild.
• The rebuild window grows linearly with the corpus. Eventually the job takes longer than the interval it runs on, and you are permanently behind.
• A failed rebuild is an outage, not a hiccup. If the job dies halfway through an overwrite, you serve a partial index until someone notices.

Real corpora change constantly but sparsely. A fifty-thousand-page wiki might see forty edits a day. A support knowledge base turns over a few articles a week. The work you actually need to do per day is proportional to the delta, and RAG incremental indexing is the discipline of doing only that work: detect what changed, re-process exactly that, delete what disappeared, and leave everything else untouched.

The catch is that "leave everything else untouched" requires knowing, reliably, what changed. That is a bookkeeping problem, and the bookkeeping primitive is a stable ID plus a content hash.

Stable chunk IDs: the primitive everything else depends on

Incremental indexing lives or dies on identity. If you cannot say "this chunk in the index corresponds to that piece of the source document", you cannot update or delete precisely, and every sync degenerates into delete-all-and-reinsert per document. Two rules fix this.

1. Every document gets a durable doc_id derived from its source identity: the file path, the Notion page ID, the database primary key, the URL. Never derive it from content.
2. Every chunk gets an ID derived from the doc_id plus a hash of the chunk text, for example sha256(text)[:16]. This is content addressing.

Content-addressed chunk IDs turn diffing into set arithmetic. Re-chunk the new version of a document, compute the new ID set, and compare it with the old ID set from your manifest. IDs present in both sets are unchanged chunks: skip them, no embedding call, no write. IDs only in the new set are new or modified text: embed and upsert. IDs only in the old set are gone: delete. You never need a separate "did this chunk change" check, because a changed chunk simply produces a different ID.

One trap to design around: the chunk boundary cascade. With naive fixed-size chunking, inserting one sentence at the top of a document shifts every boundary after it, so every downstream chunk gets new text, a new hash, and a full re-embed. The fix is to chunk within stable structural units first: split on headings, sections, or top-level blocks, then apply size-based splitting inside each unit. An edit to section 7 then only invalidates section 7's chunks. Markdown, HTML, and most wiki formats give you this structure for free; use it.

Also separate content changes from metadata changes. If a document's permissions, labels, or owner change but the text does not, you should not re-embed anything. Vector stores support in-place metadata updates (Qdrant calls it set_payload, others have equivalents), and your diff logic should route metadata-only changes there.

A minimal RAG incremental indexing loop in Python

Here is the whole pattern in one file: a SQLite manifest as the source of truth for what is indexed, content-addressed chunk IDs, and diff-driven upserts and deletes against Qdrant. It works the same way against Pinecone, pgvector, Weaviate, or Chroma; only the client calls change.

import hashlib
import sqlite3
import uuid

from openai import OpenAI
from qdrant_client import QdrantClient, models

EMBED_MODEL = "text-embedding-3-small"

oa = OpenAI()
qd = QdrantClient(url="http://localhost:6333")

conn = sqlite3.connect("manifest.db")
conn.execute("""
    CREATE TABLE IF NOT EXISTS chunks (
        chunk_id TEXT PRIMARY KEY,
        doc_id   TEXT NOT NULL,
        text     TEXT NOT NULL
    )
""")

def make_chunk_id(doc_id: str, text: str) -> str:
    digest = hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]
    return f"{doc_id}#{digest}"

def to_point_id(chunk_id: str) -> str:
    # Qdrant point IDs must be UUIDs or integers, so derive one
    return str(uuid.uuid5(uuid.NAMESPACE_URL, chunk_id))

def sync_document(doc_id: str, new_chunks: list[str]) -> None:
    old_ids = {r[0] for r in conn.execute(
        "SELECT chunk_id FROM chunks WHERE doc_id = ?", (doc_id,))}
    new = {make_chunk_id(doc_id, c): c for c in new_chunks}

    to_add = {cid: t for cid, t in new.items() if cid not in old_ids}
    to_del = old_ids - new.keys()

    if to_add:
        texts = list(to_add.values())
        resp = oa.embeddings.create(model=EMBED_MODEL, input=texts)
        vectors = [d.embedding for d in resp.data]
        qd.upsert("docs", points=[
            models.PointStruct(
                id=to_point_id(cid),
                vector=vec,
                payload={"doc_id": doc_id, "chunk_id": cid, "text": txt},
            )
            for (cid, txt), vec in zip(to_add.items(), vectors)
        ])
        conn.executemany(
            "INSERT OR REPLACE INTO chunks VALUES (?, ?, ?)",
            [(cid, doc_id, t) for cid, t in to_add.items()])
        conn.commit()

    if to_del:
        qd.delete("docs", points_selector=models.PointIdsList(
            points=[to_point_id(cid) for cid in to_del]))
        conn.executemany("DELETE FROM chunks WHERE chunk_id = ?",
                         [(c,) for c in to_del])
        conn.commit()

Call sync_document with the freshly chunked version of any document, changed or not. If nothing changed, both to_add and to_del are empty and the function costs one SQLite query. A few details are load-bearing.

• The order of operations makes crashes safe. Vectors are upserted before the manifest records them, and deletes remove vectors before manifest rows. If the process dies in between, re-running converges: upserts are idempotent because the point ID is deterministic, and the manifest never claims a chunk that is not in the store.
• Add a document-level short-circuit for large sources. Store a whole-document hash in a docs table and return early when it matches; parsing and chunking a 200-page PDF just to discover nothing changed is wasted CPU.
• The manifest is your recovery tool. If the vector store and manifest ever disagree, you can rebuild either side from the other plus the source, without re-embedding unchanged text.

Detecting change at the source

The diff loop needs a stream of "this document may have changed" events. How you get them depends on the source, and each one has a sharp edge.

• Filesystems: use mtime plus size as a cheap prefilter, but treat the content hash as truth. Editors, sync clients, and backup restores rewrite files without changing content, and mtime lies in both directions.
• Object stores: S3 event notifications (via EventBridge or SQS) give you near-real-time create, update, and delete events. Do not treat the ETag as a content MD5: for multipart uploads it is a hash of part hashes with a part-count suffix, so the same bytes uploaded differently produce different ETags. Compute your own hash after download.
• Databases: an updated_at column plus soft-delete flags covers polling. For real-time, use change data capture: Debezium or plain Postgres logical replication emits row-level insert, update, and delete events you can map straight to sync_document calls.
• SaaS sources: Notion exposes last_edited_time for polling, Confluence and Google Drive offer webhooks or a changes feed. Webhooks get dropped, expire, and silently break on permission changes, so never rely on them alone.
• Web content: sitemap lastmod values narrow the crawl, and conditional GETs with If-Modified-Since or If-None-Match make re-fetches cheap. Treat lastmod as a hint, not a guarantee; plenty of CMSes stamp every page on every deploy.

Because every event source misses events eventually, run a reconciliation sweep: a scheduled job (nightly is typical) that lists all source documents, compares IDs and hashes against the manifest, and enqueues whatever drifted. The sweep is cheap because it compares hashes, not embeddings. Event-driven sync gives you minutes-level freshness; the sweep guarantees the index converges to correct even when the event stream lies.

RAG incremental indexing with LlamaIndex and LangChain

Both major frameworks ship this bookkeeping so you do not have to hand-roll the manifest.

LlamaIndex handles it through the ingestion pipeline's document store. Attach a docstore and a strategy, and the pipeline hashes each document, skips unchanged ones, and replaces the chunks of changed ones.

from llama_index.core import Document
from llama_index.core.ingestion import DocstoreStrategy, IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.vector_stores.qdrant import QdrantVectorStore
from qdrant_client import QdrantClient

vector_store = QdrantVectorStore(
    client=QdrantClient(url="http://localhost:6333"),
    collection_name="docs",
)

pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(chunk_size=512, chunk_overlap=64),
        OpenAIEmbedding(model="text-embedding-3-small"),
    ],
    docstore=SimpleDocumentStore(),
    docstore_strategy=DocstoreStrategy.UPSERTS,
    vector_store=vector_store,
)

docs = [Document(text=body, id_=f"wiki:{page_id}")
        for page_id, body in changed_pages()]
pipeline.run(documents=docs)
pipeline.persist("./pipeline_storage")

The two things people get wrong: you must set id_ yourself to a stable source-derived value (the default is random, which makes every run look like all-new documents), and you must persist the docstore between runs (pipeline.persist on shutdown, pipeline.load on startup), otherwise the pipeline has amnesia and re-embeds everything. UPSERTS updates changed documents; UPSERTS_AND_DELETE additionally deletes any indexed document missing from the current batch, which is only correct when you pass the full corpus.

LangChain's equivalent is the indexing API: a RecordManager tracks document hashes and timestamps, and the index function computes the same add, skip, delete decisions.

from langchain_classic.indexes import SQLRecordManager, index
from langchain_openai import OpenAIEmbeddings
from langchain_qdrant import QdrantVectorStore

vectorstore = QdrantVectorStore.from_existing_collection(
    collection_name="docs",
    embedding=OpenAIEmbeddings(model="text-embedding-3-small"),
    url="http://localhost:6333",
)

record_manager = SQLRecordManager(
    "qdrant/docs", db_url="sqlite:///record_manager.db")
record_manager.create_schema()

stats = index(
    docs,                 # iterable of Document objects
    record_manager,
    vectorstore,
    cleanup="incremental",
    source_id_key="source",
)
print(stats)  # num_added / num_updated / num_skipped / num_deleted

Since the LangChain 1.x split, these live in the langchain-classic package; on the older 0.x line the import is langchain.indexes. The cleanup mode matters: "incremental" deletes stale chunks of documents present in the batch, so you can feed it partial batches safely, while "full" also deletes anything absent from the batch and therefore requires the complete corpus every run. Either way, source_id_key must point at metadata that is stable across runs, which is the same stable-ID rule as everywhere else.

Handling deletes, renames, and moves

Deletes are where incremental pipelines rot silently. Nothing breaks when you miss one; the index just keeps answering questions from a document that no longer exists. That is embarrassing for a wiki and a compliance incident for anything under GDPR-style erasure obligations, where "we deleted the row but the embeddings still say it" does not count as deleted.

Treat deletes as first-class events. CDC streams and S3 notifications carry them natively. Polling APIs often do not, which is another job for the reconciliation sweep: any doc_id in the manifest that no longer exists at the source gets tombstoned, its chunk IDs looked up, its vectors deleted, and its manifest rows removed. Keep the tombstone record (doc_id, deleted_at) for a while; it is invaluable when someone asks why a document vanished.

Renames and moves are deletes in disguise. If your doc_id is a file path, moving a file creates a new document and orphans the old one, and you pay a full re-embed for a no-op change. Where the source system offers an identity that survives moves (a Notion page ID, a Drive file ID, a database key), prefer it over the path. Where it does not, a content-level fallback works: before treating a new doc_id as new, check whether its whole-document hash matches a recently deleted one, and if so, rewrite the metadata on the existing chunks instead of re-embedding them.

Cache invalidation: the four layers of a RAG stack

A perfectly fresh index behind a stale cache still serves stale answers, and most production RAG stacks have four distinct caches with different invalidation rules.

Layer 1: the embedding cache. Key it on the hash of the exact input text plus the model name and dimensions, mapping to the vector. Because it is content-addressed, it can never go stale: changed text is a different key. It needs no invalidation at all, only LRU eviction for space, and it makes re-runs and reconciliation sweeps nearly free. The only event that flushes it is a deliberate change to the model or to your text normalization.

Layer 2: the retrieval cache. Caching query to top-k chunk IDs (or full chunk payloads) saves vector-search latency on repeated queries, and it goes stale the moment the index changes. Per-entry invalidation is hopeless because you cannot know which cached queries would now retrieve the changed chunk. The epoch pattern solves this in O(1): put a version counter in every key and bump it when an ingest batch lands.

import hashlib
import json
import redis

r = redis.Redis()

def cached_retrieve(query: str):
    qh = hashlib.sha256(query.encode()).hexdigest()[:24]
    epoch = (r.get("index:epoch") or b"0").decode()
    key = f"retr:{epoch}:{qh}"
    hit = r.get(key)
    if hit is not None:
        return json.loads(hit)
    results = vector_search(query)
    r.setex(key, 900, json.dumps(results))  # TTL as a backstop
    return results

def on_ingest_batch_complete():
    r.incr("index:epoch")

Old-epoch keys become unreachable instantly and age out via TTL. If your change rate is high enough that the epoch bumps every few seconds, your hit rate dies; batch ingests on a short window (say, one bump per minute) to keep the cache useful.

Layer 3: the answer cache. Semantic caches that return a previously generated answer for a similar query are the biggest staleness risk, because they bypass retrieval and generation entirely. If you run one, record which chunk IDs each cached answer was generated from, and maintain a reverse index from chunk ID to answer keys. When a chunk is updated or deleted, evict exactly the answers built on it: for each changed chunk ID, look up its answer keys in the reverse index (a Redis set per chunk works), delete those answers, then delete the set. Keep a modest TTL anyway as a backstop, and skip answer caching entirely for corpora where wrong-but-confident stale answers are expensive.

Layer 4: provider prompt caching. OpenAI and Anthropic cache prompt prefixes on exact token matches. This is self-invalidating for freshness purposes: a changed retrieved chunk changes the tokens, so it simply misses the cache. The practical rule is about ordering, not invalidation: put the static system prompt and tool definitions first and the volatile retrieved context last, so the stable prefix keeps hitting while the fresh chunks vary.

The unifying principle: prefer content-addressed keys (which never need invalidation) and version-stamped keys (which invalidate in one operation) over trying to enumerate which entries a change affects. Only the answer cache justifies per-entry bookkeeping, and only because regenerating answers is expensive.

Re-embedding migrations and atomic index swaps

Some changes invalidate the entire index by definition: a new embedding model, different dimensions, a new chunking strategy, a changed normalization step. Incremental indexing does not save you here; what saves you is doing the rebuild as a blue-green deployment instead of an in-place overwrite.

1. Create a new collection (docs_v2) with the new settings.
2. Start dual-writing: every incremental update from the live pipeline goes to both collections.
3. Backfill docs_v2 from source (the embedding cache does nothing for a model change, so this is the expensive step; budget it).
4. Verify: compare document and chunk counts against the manifest, and run a fixed set of evaluation queries against both collections to confirm recall did not regress.
5. Flip the alias, then bump the cache epoch so no retrieval cache entry from the old index survives.

from qdrant_client import QdrantClient, models

qd = QdrantClient(url="http://localhost:6333")
qd.update_collection_aliases(change_aliases_operations=[
    models.DeleteAliasOperation(
        delete_alias=models.DeleteAlias(alias_name="docs_prod")),
    models.CreateAliasOperation(
        create_alias=models.CreateAlias(
            collection_name="docs_v2", alias_name="docs_prod")),
])

Your application only ever queries docs_prod. Both alias operations apply in a single request, queries never see a missing alias, and rollback is the same flip in reverse while the old collection still exists. Elasticsearch and OpenSearch have index aliases for exactly this; on stores without aliases, put the collection name in your own config service and treat that pointer as the alias.

Freshness SLOs and what to monitor

"The index is fresh" should be a measured claim, not a vibe. A small set of metrics covers it.

• Staleness lag: the time from a source update to the moment the new chunk is searchable. Stamp events with the source timestamp, record indexing completion, and track the p50 and p95. This is the metric to put an SLO on, for example "p95 under 15 minutes".
• Queue depth and oldest-message age in the ingest queue, which tell you that lag is about to blow before it does.
• Delete lag, tracked separately, because erasure obligations usually come with a deadline that is stricter than your freshness target.
• Reconciliation drift: how many documents the nightly sweep had to fix. A healthy event-driven pipeline shows near zero; a rising trend means an event source is silently broken.
• Embedding spend per day. It should track your change rate. If it suddenly tracks your corpus size, some ID or hash became unstable and you are re-embedding the world; the chunk-boundary cascade and random document IDs are the usual suspects.

Add a canary: a synthetic document your pipeline updates with a fresh timestamp every few minutes, plus a probe that queries for it and alerts when the retrieved timestamp is too old. It is end-to-end, it exercises every stage including cache invalidation, and it catches the failure modes your per-stage metrics miss.

A reference architecture that holds up

Putting the pieces together, the shape that works for most teams is small.

1. Change detection per source: CDC, webhooks, or polling, all emitting normalized events (doc_id, source, event_type, source_timestamp) onto a queue such as SQS or Redis Streams.
2. An idempotent ingest worker: fetch, parse, structural chunking, content-hash diff against the manifest, embed only new chunk IDs (through the embedding cache), upsert, delete, update manifest.
3. A batch boundary: on each completed batch, bump the index epoch and run answer-cache eviction for the changed chunk IDs.
4. A nightly reconciliation sweep that lists every source, diffs hashes against the manifest, and enqueues drift, including deletes.
5. Metrics on every stage, a freshness canary, and alias-based collections so model migrations are a flip, not a weekend.

Resist the urge to make this big. One worker process, SQLite or Postgres for the manifest, and Redis for caches handles corpora into the millions of chunks. You do not need a streaming platform on day one; you need stable IDs, honest hashing, and a sweep that catches what the events miss.

Where to start

If you have a batch-rebuild pipeline today, the migration order that pays off fastest: first make document and chunk IDs stable and content-addressed (this alone often cuts embedding spend dramatically because reconciliation stops re-embedding unchanged text), then add the manifest and diff loop, then wire real change events for your highest-churn source, and only then invest in cache layers beyond the content-addressed embedding cache. Measure staleness lag from the start so every step shows up in a number.

FAQ

How often should incremental indexing run for a RAG system?

Event-driven where the source supports it (CDC, S3 notifications, webhooks), which gets staleness lag down to seconds or minutes. For polled sources, every 5 to 15 minutes is a sensible default; tighter intervals mostly burn API quota on unchanged listings. Always pair either mode with a nightly reconciliation sweep.

Do I have to re-embed everything when I change chunk size or embedding model?

Yes. Both change the mapping from text to vectors, and mixing vectors from different models or chunkings in one collection corrupts retrieval. Do it as a blue-green rebuild with dual writes and an alias flip, not an in-place overwrite.

How do I keep the vector store and the manifest consistent without transactions?

Make every operation idempotent (deterministic point IDs, upserts) and order writes so a crash leaves the system re-runnable: upsert vectors, then record them in the manifest; delete vectors, then remove manifest rows. Re-running the same sync then converges instead of duplicating. The reconciliation sweep is the final safety net.

What about documents where only metadata changed?

Do not re-embed. Route metadata-only diffs to the store's payload update API and update the manifest row. Re-embedding on permission or label changes is one of the most common sources of surprise embedding bills.

Is a semantic answer cache worth the staleness risk?

Only with high repeat-query traffic and a real invalidation story: track source chunk IDs per cached answer, evict via a reverse index on chunk updates, and keep a TTL backstop. If you cannot build that, cache retrieval results (cheap to invalidate with an epoch key) and let generation stay live.

Does incremental indexing work with hybrid search?

Yes, and the same identity scheme carries over. Use identical chunk IDs in the keyword index (BM25 in OpenSearch, Elasticsearch, or your vector store's sparse index) and the vector index, apply the same diff-driven upserts and deletes to both, and bump one shared epoch so cached results never mix index versions.