Monitoring Embedding Drift in Production

Embedding drift is the slow, often invisible shift in what your embedding model produces for the same or similar inputs over time, and it is one of the most common reasons a RAG system that worked great in week one feels noticeably worse by week twelve. It happens for three different reasons: the underlying data distribution changes (new topics, new user language), the embedding model itself gets swapped or silently updated by a provider, or the index and the queries drift apart because one side gets re-embedded and the other does not. This article walks through how to detect embedding drift, how to measure it with real numbers, and how to wire up monitoring so you catch it before your users do.

What embedding drift actually is

An embedding model maps text (or images, or audio) into a fixed-size vector such that semantically similar inputs land close together in vector space. Drift means that relationship stops holding the way it used to. There are two flavors worth separating clearly, because they need different fixes.

Data drift: the inputs you are embedding today look statistically different from the inputs you embedded six months ago. A support bot trained on embeddings from billing questions starts fielding a wave of API integration questions. The embedding model itself hasn't changed, but the region of vector space your queries land in has shifted, and if your index was built mostly on old-topic documents, retrieval quality for the new topics quietly suffers.

Model drift: the embedding model itself changes. This is more dangerous because it is often invisible. If you call a hosted embedding API (OpenAI, Cohere, Voyage, Google) by a model name without a pinned version, the provider can update the underlying weights behind that name. Vectors generated in January and vectors generated in June may no longer be directly comparable even though the model name in your code never changed. Mixing old and new vectors in the same index is the single most common cause of a RAG system that "just started getting worse" for no obvious reason.

There is a third, adjacent problem worth naming separately: index-query skew. Your document index was embedded with model version A. At some point you upgrade to model version B for new documents, but never re-embed the old ones. Now half your index is in one geometry and half is in another, and cosine similarity between a query and an old document is meaningless because they were never in the same space to begin with. This is not really "drift" in the statistical sense, but it produces identical symptoms and the fix is the same: re-embed and re-index everything together.

Why this matters for RAG systems specifically

In a classic ML classifier, drift shows up as a dropping accuracy metric you can compute against labeled data. Embedding drift in a RAG pipeline is sneakier because there is usually no ground truth sitting around. Nobody labels "was this the correct chunk to retrieve" for every production query. So the failure mode is: retrieval quality degrades, the LLM generation step compensates by hallucinating around gaps or giving vaguer answers, and the whole thing looks like "the LLM got dumber" when the actual root cause is upstream in the embedding layer.

This is why embedding drift monitoring has to be treated as its own observability surface, separate from LLM output monitoring. If you only watch the final answer quality, you will catch the problem weeks after it started and you will misdiagnose the cause.

Detecting drift: comparing embedding distributions over time

The core technique is straightforward: take a rolling sample of production embeddings, compare their statistical distribution against a reference window (say, the first 30 days after launch, or last month), and alert when the distance between the two exceeds a threshold.

Step 1: capture and store a sample of production embeddings

You do not need to store every embedding forever, but you do need a representative sample with timestamps. If you already write vectors to a vector database, add a lightweight sidecar table or object store dump.

import json
import time
import numpy as np

def log_embedding_sample(vector: np.ndarray, metadata: dict, sample_rate: float = 0.05):
    """Randomly sample production embeddings for drift monitoring."""
    if np.random.random() > sample_rate:
        return
    record = {
        "timestamp": time.time(),
        "vector": vector.tolist(),
        "source": metadata.get("source", "unknown"),
        "model_version": metadata.get("model_version", "unpinned"),
    }
    with open("embedding_samples.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")

A 5% sample rate at reasonable production volume gives you thousands of vectors per day, which is plenty for distribution comparisons.

Step 2: compute a distribution-distance metric between windows

Two metrics do most of the work here: centroid drift (has the average vector moved) and Maximum Mean Discrepancy or a simpler proxy like average pairwise cosine distance shift. Centroid drift is cheap and a good first alarm; it will not catch every kind of drift (a distribution can rotate around the same centroid), but in practice it flags the majority of real incidents.

import numpy as np

def centroid_drift(reference_vectors: np.ndarray, current_vectors: np.ndarray) -> float:
    """Cosine distance between the centroid of two embedding batches."""
    ref_centroid = reference_vectors.mean(axis=0)
    cur_centroid = current_vectors.mean(axis=0)
    ref_norm = ref_centroid / np.linalg.norm(ref_centroid)
    cur_norm = cur_centroid / np.linalg.norm(cur_centroid)
    cosine_sim = np.dot(ref_norm, cur_norm)
    return 1 - cosine_sim

def variance_drift(reference_vectors: np.ndarray, current_vectors: np.ndarray) -> float:
    """Ratio of average vector-norm variance between two batches."""
    ref_var = np.var(np.linalg.norm(reference_vectors, axis=1))
    cur_var = np.var(np.linalg.norm(current_vectors, axis=1))
    if ref_var == 0:
        return 0.0
    return abs(cur_var - ref_var) / ref_var

Run this daily or weekly comparing "last window" against "reference window," and log the result as a time series metric you can graph. A centroid cosine distance climbing from 0.01 to 0.08 over a month is a real signal worth investigating, even before you know exactly what caused it.

For a more rigorous statistical test, use Maximum Mean Discrepancy (MMD), which compares full distributions rather than just centroids:

from sklearn.metrics.pairwise import rbf_kernel

def mmd(x: np.ndarray, y: np.ndarray, gamma: float = None) -> float:
    """Maximum Mean Discrepancy between two sets of vectors using an RBF kernel."""
    if gamma is None:
        gamma = 1.0 / x.shape[1]
    xx = rbf_kernel(x, x, gamma=gamma).mean()
    yy = rbf_kernel(y, y, gamma=gamma).mean()
    xy = rbf_kernel(x, y, gamma=gamma).mean()
    return xx + yy - 2 * xy

MMD is more sensitive but more expensive to compute at scale, so a common pattern is: run cheap centroid drift on every batch, and only run MMD when centroid drift crosses a warning threshold, as a confirmation check.

Step 3: track retrieval-level proxies, not just vector statistics

Vector distribution comparisons tell you the geometry moved, but they do not tell you retrieval quality actually got worse. Pair distribution monitoring with a small set of golden queries, a fixed list of 20-50 representative questions with known-good expected document IDs, and re-run them against your live index on a schedule.

def evaluate_golden_queries(golden_set: list[dict], retriever, k: int = 5) -> dict:
    """
    golden_set: [{"query": str, "expected_doc_ids": set[str]}, ...]
    retriever: callable(query, k) -> list[doc_id]
    """
    hits = 0
    reciprocal_ranks = []
    for item in golden_set:
        results = retriever(item["query"], k=k)
        expected = item["expected_doc_ids"]
        matched = [doc_id for doc_id in results if doc_id in expected]
        if matched:
            hits += 1
            rank = results.index(matched[0]) + 1
            reciprocal_ranks.append(1 / rank)
        else:
            reciprocal_ranks.append(0)

    return {
        "recall_at_k": hits / len(golden_set),
        "mean_reciprocal_rank": sum(reciprocal_ranks) / len(reciprocal_ranks),
    }

Run this nightly, store the result, and alert when recall_at_k drops more than, say, 10 percentage points from its 7-day rolling average. This single check catches both data drift and index-query skew, because it directly measures the thing you actually care about.

Detecting model version changes specifically

If you use a hosted embedding API, pin the model version explicitly wherever the provider supports it, rather than a bare model name. Where pinning isn't available, add a canary check: embed a small fixed set of sentences on a schedule and compare the output vectors byte-for-byte (or cosine similarity ~1.0) against a stored baseline.

import numpy as np

CANARY_SENTENCES = [
    "The quick brown fox jumps over the lazy dog.",
    "Refunds are processed within five business days.",
    "How do I reset my password?",
]

def check_model_canary(embed_fn, baseline_path: str = "canary_baseline.npy", threshold: float = 0.999):
    current = np.array([embed_fn(s) for s in CANARY_SENTENCES])
    baseline = np.load(baseline_path)
    similarities = [
        np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
        for a, b in zip(current, baseline)
    ]
    min_sim = min(similarities)
    if min_sim < threshold:
        raise RuntimeError(
            f"Embedding model canary failed: min similarity {min_sim:.4f} "
            f"below threshold {threshold}. The embedding model behind this "
            f"endpoint may have changed."
        )
    return min_sim

Run this check hourly or on every deploy. It costs a handful of API calls and will catch a silent provider-side model swap immediately, instead of you discovering it three weeks later when someone files a support ticket about bad search results.

Wiring this into a monitoring stack

Once you have the metrics, treat them like any other production signal, not a one-off script someone runs manually.

• Emit as time series metrics. Push centroid_drift, mmd_score, recall_at_k, and canary_similarity to whatever you already use (Prometheus, Datadog, CloudWatch). This lets you graph trends and set alerts using existing infrastructure instead of building a bespoke dashboard.
• Schedule the jobs. A daily cron job (or a scheduled Lambda/Cloud Function) that pulls the last 24 hours of sampled embeddings, computes drift metrics, and pushes them is enough for most teams. Golden-query recall checks are cheap enough to run more often, even hourly.
• Set alert thresholds based on your own baseline, not a generic number. Run the drift calculation for a few weeks first to see what "normal" noise looks like for your traffic, then set the alert threshold a few standard deviations above that noise floor. A fixed threshold copied from a blog post (including this one) will either be too noisy or too insensitive for your actual data.
• Alert on the trend, not just the point value. A single noisy day of centroid drift is normal. Three consecutive days trending upward, or a step change right after a deploy, is the signal worth paging someone for.
• Correlate with deploy events. Tag your drift metrics with deploy markers (model version, index rebuild timestamp, embedding library version). Most real incidents are self-inflicted: someone upgraded an embedding library, changed a preprocessing step, or re-embedded only part of the corpus. A drift spike that lines up exactly with a deploy timestamp is nearly always the actual cause.

A minimal drift-monitoring pipeline end to end

Here is how the pieces fit together as a scheduled job, roughly what you'd run as a daily cron task or a lightweight Airflow/Prefect DAG:

def daily_drift_check(reference_vectors, current_day_vectors, golden_set, retriever, embed_fn):
    results = {}

    results["centroid_drift"] = centroid_drift(reference_vectors, current_day_vectors)
    results["variance_drift"] = variance_drift(reference_vectors, current_day_vectors)

    if results["centroid_drift"] > 0.05:
        results["mmd_score"] = mmd(reference_vectors[:500], current_day_vectors[:500])

    golden_metrics = evaluate_golden_queries(golden_set, retriever)
    results.update(golden_metrics)

    try:
        results["canary_similarity"] = check_model_canary(embed_fn)
    except RuntimeError as e:
        results["canary_alert"] = str(e)

    push_to_metrics_backend(results)

    if results.get("recall_at_k", 1.0) < 0.7 or results["centroid_drift"] > 0.1:
        send_alert(f"Embedding drift detected: {results}")

    return results

Wire push_to_metrics_backend and send_alert to whatever you already use for paging (Slack webhook, PagerDuty, or even just an email). The point is that this runs unattended and someone finds out the same day drift starts, not the same week a user complains.

Fixing drift once you find it

Detection is half the job; the fix depends on which type of drift you found.

For data drift, the answer is usually to expand your document corpus and re-embed to cover the new topic distribution, and to add representative golden queries for the new topic so future drift is caught faster.

For model drift, pin the model version everywhere it is configurable, and if the provider forces an update, re-embed your entire corpus with the new model rather than mixing vector generations. Never let a query embedded with model version B get compared against a document embedded with model version A; the similarity scores are not meaningful across versions even if the dimensionality matches.

For index-query skew, run a full re-embed and re-index as an atomic operation: build the new index alongside the old one, validate golden-query recall on the new index, then cut over. Do not incrementally re-embed a live index in place, since that guarantees a period where the index is split across two incompatible geometries.

FAQ

What is embedding drift? Embedding drift is when the vectors a system produces for the same or similar inputs change meaningfully over time, either because the input data distribution shifted or because the embedding model itself changed. It causes retrieval quality in RAG and search systems to degrade without any code change looking obviously broken.

How is embedding drift different from concept drift? Concept drift is a general ML term for when the relationship between inputs and the target label changes. Embedding drift is a more specific case that applies to vector representations: the geometry of the embedding space shifts, which affects similarity search and retrieval rather than a classifier's decision boundary directly.

Can I detect embedding drift without labeled data? Yes, and this is the common case. Distribution-based checks like centroid drift and MMD need no labels at all, since they just compare statistical properties of embedding batches over time. Golden-query recall checks need a small, fixed, hand-curated set of query-to-document mappings, which is much cheaper to build than a full labeled evaluation set.

How often should I run drift checks? Distribution checks (centroid drift, MMD) are cheap enough to run daily. Golden-query recall checks are cheap enough to run hourly if your retriever is fast. Model canary checks should run on every deploy at minimum, and on a schedule (hourly or daily) in between deploys to catch silent provider-side model updates.

Does switching embedding models always require a full re-index? Yes, in almost all cases. Vectors from different models, or even different versions of the same model, are not guaranteed to live in a comparable space, so cosine similarity between them is not meaningful. Budget for a full re-embed and re-index whenever you change embedding models, and treat partial migrations as a bug waiting to happen.

What threshold should I use for centroid drift alerts? There is no universal number, and any fixed threshold you copy from an article is a starting guess, not a rule. Compute the metric against your own traffic for two to four weeks to establish a noise baseline, then set the alert a few standard deviations above that baseline. Revisit the threshold after any major product or content change, since the "normal" noise floor shifts too.

Do I need a vector database feature for this, or can I do it myself? Some vector database and observability vendors ship built-in drift dashboards, but the core math (centroid distance, MMD, golden-query recall) is simple enough to implement in a few dozen lines of Python and run as a scheduled job, as shown above. Use a vendor feature if you already have one; otherwise, rolling your own is a reasonable few hours of work and keeps you independent of any single vector database choice.