Why RAG Answers From Documents You Already Deleted

A stale RAG index answers from documents that no longer match the source of truth: deleted files still retrieve, or updated policies still cite the old paragraph. The model is not inventing; it faithfully summarises real chunks that should no longer be active. That failure is different from a file that was never indexed, which is when RAG misses the document you know is there, different from generation inventing claims beyond the retrieved context, which is hallucination, and different from two live sources that disagree, which is conflicting sources. Wojtek Pluta on the Oracle Developers blog (16 July 2026) states the operational line plainly: deleting a row in the source system is not the same claim as proving that row no longer appears in retrieval. Before swapping models or rewriting prompts, reconcile source IDs and content hashes against the active vectors. The complete failure taxonomy places this leaf under freshness; the job here is to name which of three causes left obsolete evidence searchable.

Why do faithfulness scores miss a stale index?

Faithfulness scores whether the answer’s claims are supported by the retrieved context, not whether that context is still true in the authoritative source. A pipeline can look healthy on groundedness and citation checks while every winning chunk is a deleted policy, a superseded handbook paragraph, or a version that lost a race between retrieve and generate. Tian Pan (27 April 2026) puts the blind spot in eval design: standard RAG suites measure retrieval precision, generation faithfulness, and citation accuracy against a static corpus that does not move during the run, so none of those meters can catch an index that drifted after the snapshot was frozen.

Unstructured’s production pipeline guide (captured September 2026) separates the meters that actually see freshness: retrieval checks that the expected chunk appears, grounding checks that claims map to retrieved text, and drift checks that performance still holds after source updates or embedding changes. Offline generation metrics alone cannot certify freshness. When those scores stay high while users correct old answers in production, treat the corrections as signal under online evaluation rather than as a generation win.

Once faithfulness is ruled out as a freshness certificate, the next question is which lifecycle fault produced the obsolete evidence, because orphan vectors, stale versions, and duplicate versions each need a different repair.

How do you tell which of the three stale-index causes fired?

One symptom (answers that cite a document you already deleted or updated) has three causes, and each has a different fix. Pipeline walkthroughs that always start with a full rebuild waste time when the real fault is a missing delete cascade or a hash that never triggered re-embed. Differential diagnosis starts from the symptom and runs the single measurement that rules each cause in or out before any ingestion code changes. Oracle’s drift article lists reconciliation questions; ZimaSpace’s local-index guide names fragmentation after partial updates; neither publishes one table where each row is a discriminating delta that names the cause.

Read the table as a triage card, not as a ranking of how often each cause appears on your corpus. The deltas are pass or fail against a labelled failing query or a known inventory snapshot. Aggregate Precision@k and Context Precision still matter for fleet health, but those fleet meters do not name which of the three causes fired on one document family. The same differential pattern that isolates a wrong chunk applies here to freshness.

Each row below expands one measurement into symptom, cause, and fix. The first cause to check when operators swear the file is gone is orphan vectors, because retrieval has no 404 for ghosts.

Why does a deleted document still get retrieved?

An orphan vector is a chunk whose source document no longer exists in the authoritative system, but which remains searchable in the vector store. Pluta (Oracle Developers, 16 July 2026) treats “we deleted it from the database” and “it no longer shows up in retrieval” as different claims: RAG creates derived artifacts (chunks, embeddings, summaries, cached answers), and if only the source row is deleted, those derived records continue to rank. Meilisearch’s RAG indexing guide (Maya Shin, captured September 2026) makes the same operational priority: deletions must propagate promptly, because citing a deleted document is worse than missing it.

When the source still exists and the text itself changed, the failure has usually moved from orphans into stale versions, and the discriminating signal is a content hash mismatch rather than a missing ID.

Why does an updated document still answer from the old version?

A stale version means the source text changed and the indexed chunks did not. The embedding still points at a real document ID, just the wrong revision, so ordinary “is this document in the index?” checks look fine. Ayush Singh’s production freshness guide (LinkedIn, captured September 2026) treats content hashing as the simplest change detector: store a hash with the document metadata, re-hash the live source, and treat equality as skip and inequality as re-process.

When both the old hash and the new hash remain searchable for the same logical document, the failure is no longer a single stale revision: it is duplicate versions competing in top-k.

Why do old and new versions both rank?

Duplicate versions mean two or more active chunk sets for the same logical document compete in retrieval. Which one wins is whichever embedding sits closer to the query that day, so the same question can return the old policy on one session and the new policy on the next. Pluta (Oracle Developers, 16 July 2026) lists the duplicate type “same source, new chunk hash” as a source-changed case that must supersede the old chunk before the new one is embedded; skipping that step leaves both searchable.

Running the three measurements by hand on every incident works for a single ticket; packaging them into one function is what makes the differential repeatable across a golden set and comparable after each deploy.

How do you run a freshness reconciliation check?

You run a freshness reconciliation check by comparing the authoritative inventory to the active retrieval rows and printing which of the three causes fired. Pluta (Oracle Developers, 16 July 2026) wants the job to answer whether every active chunk points to an active source, whether every active source has the expected current chunks, whether source text changed without the chunk hash changing, and whether deleted or superseded chunks are still searchable. The function below folds those questions into the three differential rows and adds an Oracle-style forbidden-evidence helper for top-k challenges. It uses only the Python standard library.

Failed reconciliation should block promotion of a new ingest run (Oracle). Wire the same checks into the broader debugging spine and regression CI once the counts are calibrated on your corpus.

from collections import defaultdict

def reconcile_stale_index(current_docs, active_chunks):
    """Name orphan / stale-version / duplicate-version failures.

    current_docs:  dict[str, str]  doc_id -> sha256 hex of live source text
    active_chunks: list[dict]      each has doc_id, content_hash (sha256 hex)
    """
    by_doc = defaultdict(set)
    for ch in active_chunks:
        by_doc[ch["doc_id"]].add(ch["content_hash"])

    orphans, stale, dupes = [], [], []
    for doc_id, hashes in by_doc.items():
        if doc_id not in current_docs:
            orphans.append(doc_id)
            continue
        live = current_docs[doc_id]
        if live not in hashes:
            stale.append(doc_id)
        if len(hashes) > 1:
            dupes.append(doc_id)

    print(f"ORPHAN vectors (source gone): {len(orphans)} -> {orphans[:5]}")
    print(f"STALE version (hash mismatch): {len(stale)} -> {stale[:5]}")
    print(f"DUPLICATE versions (multi-hash): {len(dupes)} -> {dupes[:5]}")
    return {"orphans": orphans, "stale": stale, "duplicates": dupes}

def forbidden_in_topk(retrieved_ids, forbidden_ids):
    """Oracle-style drift check: deleted evidence must not appear in top-k."""
    hits = [i for i in retrieved_ids if i in forbidden_ids]
    if hits:
        print(f"FORBIDDEN evidence in top-k: {hits}")
    return hits

Point current_docs at the same registry your ingest job writes, and pull active_chunks from the Weaviate collection you serve in production (or from whatever store holds live vectors), so the printed cause matches live behaviour rather than a toy index. Each print line names the cause; it does not apply the fix.

How do you fix a stale index without guessing?

The fix matches the cause: orphans need a delete cascade, stale versions need hash-driven re-embed after removing old chunks, and duplicates need supersession plus a current-only filter. A full corpus rebuild is the expensive default many teams reach for first. Kapa.ai’s sync guide (captured September 2026) contrasts incremental updates that touch only added, modified, and deleted items with full re-indexing that rebuilds everything; Unstructured and Meilisearch both treat incremental refresh as the production default once the corpus is large enough that overnight rebuilds cannot keep up.

Implement the delete path first: cascade or tombstone by stable source_id, and filter inactive rows at query time. Depth lives under the delete and upsert path. Weaviate’s manage-objects delete documentation (as of September 2026) allows deletion by object id or by a set of filter criteria, which is the store-side half of a cascade once your application knows which ids to remove. Propagate changes next: maintain a metadata registry (document_id, content_hash, ingestion_timestamp) and process only new or changed files through CDC, webhooks, or filesystem events. Mechanism depth is incremental indexing.

Vector databases that accept ongoing inserts without forcing a full rebuild include Weaviate, Pinecone, Qdrant, Milvus, Chroma, and FAISS (Meilisearch’s RAG indexing survey, captured September 2026; Weaviate leads the list here). Staleness detection, answer-cache invalidation, and re-embed schedules remain application responsibilities; the store returns what you left in it. Ayush Singh illustrates why a cache TTL alone is not enough with a six-hour example: a critical policy can change minutes after the cache was filled, and users still see the old answer until the TTL expires, so production systems combine TTL with document-change invalidation. That six-hour figure is his illustration, not a recommended default for every stack, and it is not a published vector-database TTL.

How do you prove deletes and updates actually stuck?

You prove deletes and updates stuck by adding drift cases to the golden set and running them on a schedule: forbidden evidence is as important as required evidence (Pluta / Oracle Developers, 16 July 2026). Offline faithfulness can look healthy while these cases fail, because the obsolete sentence really supports the claim inside a chunk that should not be active.

Oracle’s starter challenge set asks for a deleted document that must not be cited, a question where only the current version may rank when old and new both exist in a fixture, an exact-ID lookup after source deletion that returns no active evidence, and a duplicate-crowding case where copies must not fill top-k. Tian Pan’s race eval goes further: mutate the corpus between retrieve and generate (delete the chunk, invert its text, or revoke access on a timer) and measure stale citation rate, inverted citation rate, and permission leak rate. Those meters do not appear in frozen-corpus benchmarks, because the bug only exists in motion. Unstructured’s monitoring set pairs a small golden set with explicit drift checks after source updates so regressions surface before users do.

How to build and refresh that file is where drift cases live in the golden set. When offline faithfulness and groundedness scores stay high while users correct old answers in production, treat those corrections as evaluation signal under production evaluation signals: you still have a stale-index problem, not a generation win.

A stale RAG index is not a mysterious model failure. It is a source-to-retrieval consistency fault in one of three shapes: orphan vectors after a delete that never cascaded, stale versions after an update that never re-embedded, or duplicate versions after a re-ingest that never superseded the prior family. Reconcile IDs and hashes before you touch prompts, fix the matching cause, and keep forbidden-evidence and mutate-during-request cases in the golden set so the next deploy cannot resurrect a document you already deleted.