Why RAG Gets Worse Over Time

RAG quality degradation over time is retrieval that worked at ship and slowly returns worse chunks — without a deploy, without error spikes, and without a latency alarm. Three drifts cause it: the corpus grows past the geometry the index was built for, the embedding model and the stored vectors fall out of sync, or user queries shift away from the evaluation set you still trust. It is not the case where deleted or updated source documents still retrieve (that is stale index), and it is not Microsoft GraphRAG’s “DRIFT Search” feature — a different product name that collides in the SERP.

Before you rewrite prompts or swap generators, freeze a small golden query pack and compare recall@k to the baseline you recorded at ship. Decompressed (2026) reports recall falling from about 0.92 to about 0.74 in drifted systems while application error counts stay flat — standard APM cannot see this failure. Unrag’s failure taxonomy names the same symptom class: “Quality was good, now it’s worse.” Reading the retrieval trend, not the answer, is where diagnosis starts.

First, which of the three drifts is it?

Three drifts share one symptom — answers got worse over weeks — and each has a different fix. Re-embedding the whole corpus when the problem is query drift wastes days; refreshing the eval set when the problem is a model-version mismatch changes nothing. Faster than a pipeline stage-walk is differential diagnosis: one measurement that rules each cause in or out before you change code.

Decompressed (2026) treats nearest-neighbor overlap of 85–95% week-over-week as healthy and overlap below 70% as active quality loss; the same-text cosine distance between a stored vector and a fresh embed of the identical text sits near 0.0001–0.005 when stable and above 0.05 when the pipeline changed (restated by Tian Pan, 2026-04-19). Those bands are starting points — calibrate them on your corpus before you act.

Did the corpus outgrow what the index was built for?

Corpus drift is the failure where the document set changed shape and the old embeddings stayed put. Microsoft’s Azure AI Foundry note on vector drift (2026-04-04) lists this as cause two: new documents keep landing while existing vectors do not move, so recently indexed content dominates retrieval and older but still-valid content becomes harder to find — recall falls with no service outage.

Did the embedding model and the stored vectors fall out of sync?

Embedding mismatch is the failure where query vectors and document vectors no longer live in the same space. Microsoft (2026-04-04) names the discrete case: documents indexed with one embedding model, queries encoded with another. Decompressed (2026) and Codexical (2026-05-16) add the quiet cases — partial re-embeds, preprocessing or chunk-contract changes — that leave two generations of vectors in one collection with no exception thrown.

Have user queries shifted away from your eval set?

Query drift is the failure where your frozen eval still looks healthy and production does not. Unrag lists query-distribution shift among the causes of “quality was good, now it’s worse”: users ask different question types than the set you still score. The Sovereign Institute’s rename example (old docs say “Project Atlas,” users search “Horizon Platform”) is the vocabulary form of the same gap.

How do you catch RAG drift before users complain?

Three scheduled checks catch drift earlier than user tickets, and none of them is CPU or p95 latency. Pithy Cyborg (2026) states the operational minimum: a pinned embedding version with an upgrade process, a fixed retrieval test set on a schedule, and an index-freshness metric against the source corpus.

• Frozen golden pack — Codexical (2026-05-16) recommends on the order of 20–50 queries with known top results, run weekly; alert when recall@k or precision@k drops versus the ship baseline. Wire the same pack into regression testing so a bad index cannot merge blind.
• Same-text cosine sample — re-embed a rotating sample of stored documents and compare to the vectors in the index (Decompressed / Tian Pan thresholds above).
• Nearest-neighbor overlap — same queries, week apart; treat 85–95% overlap as healthy and <70% as active loss (Decompressed, 2026).

How those alerts sit next to latency, cost, and judge-score watches is monitoring RAG in production. How to score unlabeled live traffic when you have no gold answer is online evaluation. This page only names the drift-specific canaries.

How do you run all three drift checks at once?

Rather than reason about the three drifts one at a time, run them together. Given a frozen query pack with gold chunk ids, your current retriever, the embedding function the retriever uses, a sample of document texts with their stored vectors, and the model ids on both sides, this prints which check fired. It needs a labelled pack — build that under test sets before you trust the output.

import numpy as np

def diagnose_rag_drift(
    frozen_queries,
    gold_ids,
    *,
    retrieve,
    embed,
    sample_docs,
    stored_vectors,
    query_model_id,
    index_model_id,
    prior_topk=None,
    ship_recall=None,
    production_fail_rate=None,
    k=5,
    cosine_warn=0.05,
    overlap_break=0.70,
    recall_drop=0.05,
):
    """Print which of the three RAG drifts fired.

    frozen_queries, gold_ids — parallel lists for the ship-era golden pack
    retrieve(query, k)       -> list[chunk_id] ranked
    embed(text)              -> np.ndarray
    sample_docs              -> list[str] already indexed
    stored_vectors           -> list[np.ndarray] aligned with sample_docs
    prior_topk               -> optional list[list[chunk_id]] from last week's run
    ship_recall              -> float recall@k recorded at ship (or last known-good)
    production_fail_rate     -> float in [0,1] from an online/thumbs sample
    """
    def cosine(a, b):
        a, b = np.asarray(a, dtype=float), np.asarray(b, dtype=float)
        return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-12))

    fired = []

    # Cause 2 — embedding mismatch
    if query_model_id != index_model_id:
        fired.append("CAUSE 2 embedding mismatch -> model_id differs; pin + full re-embed")
    distances = [
        1.0 - cosine(stored_vectors[i], embed(sample_docs[i]))
        for i in range(len(sample_docs))
    ]
    mean_dist = float(np.mean(distances)) if distances else 0.0
    if mean_dist > cosine_warn:
        fired.append(
            f"CAUSE 2 embedding mismatch -> mean same-text distance {mean_dist:.4f} > {cosine_warn}"
        )

    # Frozen pack recall
    topk_now = [retrieve(q, k) for q in frozen_queries]
    hits = sum(1.0 if g in ids else 0.0 for g, ids in zip(gold_ids, topk_now))
    recall = hits / max(len(frozen_queries), 1)

    # Cause 1 — corpus growth (recall down + NN-overlap down, model still matched)
    model_ok = query_model_id == index_model_id and mean_dist <= cosine_warn
    overlap = None
    if prior_topk is not None:
        overlaps = [
            len(set(a) & set(b)) / float(k) for a, b in zip(prior_topk, topk_now)
        ]
        overlap = float(np.mean(overlaps)) if overlaps else 1.0
    recall_slid = ship_recall is not None and (ship_recall - recall) >= recall_drop
    if model_ok and recall_slid and (overlap is None or overlap < overlap_break):
        fired.append(
            f"CAUSE 1 corpus growth -> recall@{k}={recall:.3f} "
            f"(ship={ship_recall:.3f})"
            + (f", NN-overlap={overlap:.2f}" if overlap is not None else "")
        )

    # Cause 3 — query drift (frozen pack OK, production sample failing)
    if (
        ship_recall is not None
        and (ship_recall - recall) < recall_drop
        and production_fail_rate is not None
        and production_fail_rate >= 0.20
    ):
        fired.append(
            f"CAUSE 3 query drift -> frozen recall@{k}={recall:.3f} near ship, "
            f"production_fail_rate={production_fail_rate:.2f}"
        )

    if not fired:
        print(f"no drift check fired (frozen recall@{k}={recall:.3f}, "
              f"mean same-text distance={mean_dist:.4f})")
    for line in fired:
        print(line)
    return {"recall_at_k": recall, "mean_same_text_distance": mean_dist,
            "nn_overlap": overlap, "fired": fired}

Pass last week’s prior_topk and a production_fail_rate from online evaluation so cause 1 and cause 3 can fire. The script names the check that fired; it does not guess the fix for you.