Why RAG Answers From Documents You Already Deleted

A stale RAG index answers from documents that no longer match the source of truth — deleted files still retrieve, or updated policies still cite the old paragraph. The model is not inventing: it faithfully summarises real chunks that should no longer be active. This page is about that failure. It is not a first-time miss where the file was never indexed (see missing document), and it is not generation inventing claims beyond the retrieved context (see hallucination). When two live sources disagree, that is conflicting sources.

Before you swap models or rewrite prompts, reconcile the index against the source. “We deleted it from the database” and “it no longer shows up in retrieval” are different claims (Wojtek Pluta / Oracle Developers, July 2026). The failures hub places stale index under freshness; the job here is to name which of three causes left obsolete evidence searchable.

Why do faithfulness scores miss a stale index?

Faithfulness scores whether the answer’s claims are supported by the retrieved context — not whether that context is still true. A RAG pipeline can score 0.95 on faithfulness and still return wrong business answers when the retrieved index is stale (Atlan / Emily Winks, updated 10 April 2026 — page re-fetched for this article). On that scale, 0.95 means about 95% of the claims in the answer are supported by the chunks that came back (Atlan, citing the Ragas faithfulness definition). Quote-matching and groundedness checks can pass the same way: the obsolete sentence really exists in an old chunk (DocuShell, 2026).

Stale context is not neutral. Tian Pan (7 May 2026) reports a Google Research finding that when a RAG system retrieves insufficient or outdated context, the hallucination rate jumps from 10.2% to 66.1%. Offline generation metrics alone cannot certify freshness — when they look healthy and users still see old answers, check this page and online evaluation.

First, which of the three causes is it?

One symptom — answers cite a document you already deleted or updated — has three causes, and each has a different fix. Pipeline walkthroughs rebuild the whole index and hope. Faster is differential diagnosis: run the one check that rules each cause in or out before you change ingestion code.

Competitors list taxonomies or reconciliation checklists. The table above starts from the one symptom and gives a discriminating measurement per cause — the same differential pattern as wrong chunk, applied to freshness.

Why does a deleted document still get retrieved?

An orphan vector is a chunk whose source document no longer exists in the authoritative system, but which remains searchable in the vector store. Retrieval has no 404 for ghosts — it returns a confident citation to content that operators already removed (InsiderLLM “ghost knowledge,” 2026; Tian Pan’s “zombie documents,” 15 April 2026).

Why does an updated document still answer from the old version?

A stale version means the source text changed and the indexed chunks did not. The embedding still points at a real document ID — just the wrong revision — so ordinary “is this document in the index?” checks look fine.

Why do old and new versions both rank?

Duplicate versions mean two (or more) active chunk sets for the same logical document compete in retrieval. Which one wins is whichever embedding sits closer to the query — a coin flip across sessions (InsiderLLM, 2026).

How do you run a freshness reconciliation check?

Rather than reason about the three causes one at a time, run one reconciliation that names which fired. Given a map of current document IDs to content hashes and the list of active chunks in the vector store, this prints orphan, stale-version, and duplicate-version counts. It uses only the Python standard library.

from collections import defaultdict

def reconcile_stale_index(current_docs, active_chunks):
    """Name orphan / stale-version / duplicate-version failures.

    current_docs:  dict[str, str]  doc_id -> sha256 hex of live source text
    active_chunks: list[dict]      each has doc_id, content_hash (sha256 hex)
    """
    by_doc = defaultdict(set)
    for ch in active_chunks:
        by_doc[ch["doc_id"]].add(ch["content_hash"])

    orphans, stale, dupes = [], [], []
    for doc_id, hashes in by_doc.items():
        if doc_id not in current_docs:
            orphans.append(doc_id)
            continue
        live = current_docs[doc_id]
        if live not in hashes:
            stale.append(doc_id)
        if len(hashes) > 1:
            dupes.append(doc_id)

    print(f"ORPHAN vectors (source gone): {len(orphans)} -> {orphans[:5]}")
    print(f"STALE version (hash mismatch): {len(stale)} -> {stale[:5]}")
    print(f"DUPLICATE versions (multi-hash): {len(dupes)} -> {dupes[:5]}")
    return {"orphans": orphans, "stale": stale, "duplicates": dupes}

def forbidden_in_topk(retrieved_ids, forbidden_ids):
    """Oracle-style drift check: deleted evidence must not appear in top-k."""
    hits = [i for i in retrieved_ids if i in forbidden_ids]
    if hits:
        print(f"FORBIDDEN evidence in top-k: {hits}")
    return hits

Failed reconciliation should block promotion of a new ingest run (Oracle Developers, July 2026). Wire the same checks into the broader debugging spine and regression CI once the counts are calibrated.

How do you fix a stale index without guessing?

The fix matches the cause: orphans need a delete cascade, stale versions need hash-driven re-embed after removing old chunks, and duplicates need supersession plus a current-only filter. A full corpus rebuild is the expensive default many teams reach for first — most production systems should update deltas instead (Particula, 2026).

1. Implement the delete path — cascade or tombstone by stable source_id, and filter inactive rows at query time. Depth: updating and deleting documents.
2. Propagate changes — maintain a metadata registry (document_id, content_hash, ingestion_timestamp) and process only new or changed files (Particula). Event-driven patterns (for example S3 notifications into an ingest worker) are reported in the 3–4 minute per-document range on Particula’s 2026 guide — treat that as their measured pattern, not a universal SLA. Mechanism: incremental indexing.
3. Keep inserts continuous — vector databases that accept ongoing inserts without a full rebuild include Weaviate, Pinecone and Qdrant (Particula, 2026). Staleness detection, TTLs, and re-embed schedules remain application responsibilities; the store returns what you left in it.

How do you prove deletes and updates actually stuck?

Add drift cases to your golden set and run them on a schedule: forbidden evidence is as important as required evidence (Oracle Developers, July 2026). Offline faithfulness can stay near 0.95 while these cases fail.

• Deleted-document challenge — a query that must not cite a known-deleted doc_id; forbidden_in_topk must be empty.
• Current-beats-stale challenge — where old and new versions exist in a fixture, only the current version may appear in top-k.
• Exact-ID after delete — lookup of a removed ID returns no active evidence.
• Canary set — 10–20 questions with known-correct answers run on a timer; divergence flags drift you did not anticipate (InsiderLLM, 2026).

How to build and refresh that file is building a golden test set. When offline scores stay high while users correct old answers in production, treat those corrections as evaluation signal (Atlan, 2026) under online evaluation — you still have a stale-index problem, not a generation win.