Why RAG Gets Worse Over Time

RAG quality degradation over time is retrieval that worked at ship and slowly returns worse chunks, without a deploy, without error spikes, and without a latency alarm. Three drifts cause it: the corpus grows past the geometry the index was built for, the embedding model and the stored vectors fall out of sync, or user queries shift away from the evaluation set you still trust. It is not the case where deleted or updated source documents still retrieve (that is stale index), and it is not Microsoft GraphRAG’s product feature named “DRIFT Search.” AI Spaces (26 January 2026) describes the same quiet pattern after the first real dataset update: nothing crashes, latency looks fine, and trust erodes. Decompressed (2026) reports recall falling from about 0.92 to about 0.74 in drifted systems while application error counts stay flat. Dev.to’s production walkthrough (captured 18 September 2026) states the operational line plainly: RAG does not crash; it degrades. Before you rewrite prompts or swap generators, freeze a small golden query pack and compare recall@k to the ship baseline. The failures taxonomy places this leaf under quality loss identified by the symptom that identifies each one.

How do you tell which of the three drifts is causing RAG to get worse?

Three drifts share one symptom, answers got worse over weeks, and each has a different fix. Re-embedding the whole corpus when the problem is query drift wastes days; refreshing the eval set when the problem is a model-version mismatch changes nothing. Kapa.ai’s January 2026 mistake list names embedding rot as one silent failure among seven; AI Spaces walks a post-update degradation stack; neither publishes one table where each row is a discriminating measurement that rules a cause in or out. Faster than a pipeline stage-walk is differential diagnosis: one check that isolates corpus growth, embedding mismatch, or query drift before you change code.

Read the table as a triage card. The deltas are pass or fail against a frozen golden pack and a small production sample, not a ranking of how often each cause appears on your corpus. Decompressed (2026) treats nearest-neighbor overlap of 85-95% week-over-week as healthy and overlap below 70% as active quality loss; the same-text cosine distance between a stored vector and a fresh embed of the identical text sits near 0.0001-0.005 when stable and above 0.05 when the pipeline changed (restated by Tian Pan, 19 April 2026). Those bands are starting points. Calibrate them on your corpus before you act.

Each row below expands one measurement into symptom, cause, and fix. The first cause to check after a large ingest or knowledge-base expansion is corpus growth, because new mass changes relative neighborhoods even when the embedding model never changed.

Did the corpus outgrow what the index was built for?

Corpus drift is the failure where the document set changed shape and the old embeddings stayed put. Microsoft’s Azure AI Foundry note on vector drift (4 April 2026) lists this as cause two: new documents keep landing while existing vectors do not move, so recently indexed content dominates retrieval and older but still-valid content becomes harder to find. Recall falls with no service outage. AI Spaces (26 January 2026) calls the first dataset update the most dangerous moment in a RAG lifecycle for the same reason: vocabulary, semantic density, and document structure shift while the rest of the pipeline still assumes the old distribution.

Atlan’s accuracy survey (captured 18 September 2026) attributes a large share of production breakage to knowledge-base quality and freshness rather than to the generator alone, and cites Barnett et al. (2024, arXiv 2401.05856) for the claim that validation of a RAG system is only feasible during operation, when corpora grow and distributions move. That is the operational reason a frozen pack must be re-run after every major ingest, not only at launch. When the pack is flat and neighbors still look scrambled, the next cause is an embedding-space break.

Did the embedding model and the stored vectors fall out of sync?

Embedding mismatch is the failure where query vectors and document vectors no longer live in the same space. Microsoft (4 April 2026) names the discrete case: documents indexed with one embedding model, queries encoded with another. Decompressed (2026) and Codexical (16 May 2026) add the quiet cases: partial re-embeds, preprocessing or chunk-contract changes that leave two generations of vectors in one collection with no exception thrown. Kapa.ai (6 January 2026) calls the same pattern embedding rot: VectorDB migrations are painful, teams avoid full re-embeds, and outdated embeddings drift toward false neighbors and hallucinated outputs.

Kapa’s re-embed triggers, model improvement, roughly 10-15% corpus change, or a domain-language shift, are decision rules for when to pay for a full migration, not a substitute for the model_id and same-text checks that prove the spaces diverged. When those checks are clean and production still complains, the frozen pack may no longer represent traffic.

Have user queries shifted away from your eval set?

Query drift is the failure where your frozen eval still looks healthy and production does not. Unrag’s failure taxonomy lists query-distribution shift among the causes of “quality was good, now it’s worse”: users ask different question types than the set you still score. Atlan (captured 18 September 2026) restates the same operational risk when question distributions move from test queries to real users, and quotes Barnett et al. (2024) that pre-deployment testing cannot catch those shifts. Pryon’s accuracy guide (captured 18 September 2026) treats misalignment with user queries, failed entity recognition and misunderstood intent, as a separate accuracy pitfall from ingestion and answer matching.

Conversation-level context drift across multi-turn chat, the pattern Dev.to discusses under “context drift,” is a different failure: session memory and prompt assembly, not corpus, embedding, or eval-set geometry. That leaves the operational question of which scheduled checks catch the three drifts on this leaf before users file tickets.

How do you catch RAG drift before users complain?

Three scheduled checks catch drift earlier than user tickets, and none of them is CPU or p95 latency. Kapa.ai (6 January 2026) frames zero evals as one of the most dangerous production mistakes and lists a minimum loop: a small labeled dataset, retrieval-level evals before generation scores, weekly or biweekly automated runs in CI, and metrics tracked over time. Pithy Cyborg (2026) states the same operational minimum as a pinned embedding version with an upgrade process, a fixed retrieval test set on a schedule, and an index-freshness metric against the source corpus.

The drift-specific canaries are concrete. First, a frozen golden pack: Codexical (16 May 2026) recommends on the order of 20-50 queries with known top results, run weekly, and alert when recall@k or precision@k drops versus the ship baseline. Wire the same pack into regression testing a RAG system so a bad index cannot merge blind. Second, a same-text cosine sample: re-embed a rotating sample of stored documents and compare to the vectors in the index against Decompressed and Tian Pan bands. Third, nearest-neighbor overlap: same queries, week apart; treat 85-95% overlap as healthy and below 70% as active loss (Decompressed, 2026).

How those alerts sit next to latency, cost, and judge-score watches is monitoring RAG in production. How to score unlabeled live traffic when you have no gold answer is covered under online evaluation for unlabeled traffic. Atlan’s measurement section (captured 18 September 2026) treats continuous monitoring as required infrastructure once query distributions shift, not as an optional dashboard. The checks above name the drift canaries; they are not a tool catalog.

How do you run all three drift checks at once?

You run all three drift checks at once by calling one function that takes a frozen query pack with gold chunk ids, your current retriever, the embedding function the retriever uses, a sample of document texts with their stored vectors, and the model ids on both sides, then prints which check fired. Kapa’s eval loop and AI Spaces’ migration checks both assume you can compare retrieval before and after a change; neither ships a single three-cause namer. The script below is the differential table in code. It needs a labelled pack. Build that under how golden pairs are assembled before you trust the output.

Pass last week’s prior_topk and a production_fail_rate from an online fail-rate sample so corpus growth and query drift can fire. Thresholds below follow Decompressed (2026) published bands for same-text cosine distance and nearest-neighbor overlap. They are starting points, not universal constants. Calibrate on your corpus and embedding model before you re-embed or page anyone.

import numpy as np

def diagnose_rag_drift(
    frozen_queries,
    gold_ids,
    *,
    retrieve,
    embed,
    sample_docs,
    stored_vectors,
    query_model_id,
    index_model_id,
    prior_topk=None,
    ship_recall=None,
    production_fail_rate=None,
    k=5,
    cosine_warn=0.05,
    overlap_break=0.70,
    recall_drop=0.05,
):
    """Print which of the three RAG drifts fired.

    frozen_queries, gold_ids - parallel lists for the ship-era golden pack
    retrieve(query, k)       -> list[chunk_id] ranked
    embed(text)              -> np.ndarray
    sample_docs              -> list[str] already indexed
    stored_vectors           -> list[np.ndarray] aligned with sample_docs
    prior_topk               -> optional list[list[chunk_id]] from last week's run
    ship_recall              -> float recall@k recorded at ship (or last known-good)
    production_fail_rate     -> float in [0,1] from an online/thumbs sample
    """
    def cosine(a, b):
        a, b = np.asarray(a, dtype=float), np.asarray(b, dtype=float)
        return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-12))

    fired = []

    # Cause 2: embedding mismatch
    if query_model_id != index_model_id:
        fired.append("CAUSE 2 embedding mismatch -> model_id differs; pin + full re-embed")
    distances = [
        1.0 - cosine(stored_vectors[i], embed(sample_docs[i]))
        for i in range(len(sample_docs))
    ]
    mean_dist = float(np.mean(distances)) if distances else 0.0
    if mean_dist > cosine_warn:
        fired.append(
            f"CAUSE 2 embedding mismatch -> mean same-text distance {mean_dist:.4f} > {cosine_warn}"
        )

    # Frozen pack recall
    topk_now = [retrieve(q, k) for q in frozen_queries]
    hits = sum(1.0 if g in ids else 0.0 for g, ids in zip(gold_ids, topk_now))
    recall = hits / max(len(frozen_queries), 1)

    # Cause 1: corpus growth (recall down + NN-overlap down, model still matched)
    model_ok = query_model_id == index_model_id and mean_dist <= cosine_warn
    overlap = None
    if prior_topk is not None:
        overlaps = [
            len(set(a) & set(b)) / float(k) for a, b in zip(prior_topk, topk_now)
        ]
        overlap = float(np.mean(overlaps)) if overlaps else 1.0
    recall_slid = ship_recall is not None and (ship_recall - recall) >= recall_drop
    if model_ok and recall_slid and (overlap is None or overlap < overlap_break):
        fired.append(
            f"CAUSE 1 corpus growth -> recall@{k}={recall:.3f} "
            f"(ship={ship_recall:.3f})"
            + (f", NN-overlap={overlap:.2f}" if overlap is not None else "")
        )

    # Cause 3: query drift (frozen pack OK, production sample failing)
    if (
        ship_recall is not None
        and (ship_recall - recall) < recall_drop
        and production_fail_rate is not None
        and production_fail_rate >= 0.20
    ):
        fired.append(
            f"CAUSE 3 query drift -> frozen recall@{k}={recall:.3f} near ship, "
            f"production_fail_rate={production_fail_rate:.2f}"
        )

    if not fired:
        print(f"no drift check fired (frozen recall@{k}={recall:.3f}, "
              f"mean same-text distance={mean_dist:.4f})")
    for line in fired:
        print(line)
    return {"recall_at_k": recall, "mean_same_text_distance": mean_dist,
            "nn_overlap": overlap, "fired": fired}

The script names the check that fired; it does not guess the fix for you. When cause 2 fires, follow the migration path on the embedding-drift leaf. When cause 1 fires, prefer incremental re-embeds and hybrid lexical coverage. When cause 3 fires, refresh the gold set and the online sample before you pay for a full re-embed.

RAG gets worse over time when corpus geometry, embedding generations, or query traffic move while the monitors that would name the cause stay dark. Freeze a golden pack at ship, run the three discriminating checks on a schedule, and treat a full re-embed as the fix for mismatched vector spaces rather than as the default response to every quality ticket. The leaf that owns deletes and orphans is stale index; the leaf that owns detector depth for embedding distributions is embedding drift; this leaf owns the triage that decides which of those paths to take.