When RAG Misses the Document You Know Is There
Recall failures that survive every smoke test, and how to prove the document was never a candidate.
You know the document is there. Retrieval never returns it. A missing-document failure is a recall miss at the candidate set: the document you can open on disk never appears among the chunks passed to the model. It is not the case where a chunk came back and was the wrong one (see wrong chunk), and it is not the case where retrieval was fine but the model still answered badly — that is generation-side (lost in the middle, hallucination, conflicting sources). Barnett et al. (2024) catalogue this split as Missing Content versus Missed Top-Ranked Documents in their seven failure points (arXiv 2401.05856) — this page turns that catalogue into a proving sequence.
Before you rewrite prompts or swap models, prove the document was never a candidate. Smoke tests that only check “the bot answered” miss this failure entirely. The failures hub places it under index and retrieve; the job here is to show which gate kept the document out.
How do you prove the document was never a candidate?
A missing document is proven by ordered checks that rule causes in or out — not by changing every knob at once. Run these on one query whose answer you can point to in a known source file. Each step either finds the document or eliminates a cause; stop when the cause is named.
- Look up the document ID (or source path) directly in the index. If the ID is absent, retrieval never had a chance — fix ingest first.
- Re-run with every metadata filter removed. If the gold document appears only then, a filter was excluding it.
- Raise top-k well above production (for example to 50). If the gold appears at a rank greater than your production k, the cutoff — not absence from the corpus — was the gate.
- Compare BM25-only to dense-only on the same query. If sparse finds the gold and dense does not, vocabulary — not “the model” — is the gap.
- Grep the source for the answer string and check it against chunk boundaries. If the answer straddles a split so no single chunk matches the query, granularity kept the document out of the candidate set.
If a chunk is returned and it is simply the wrong passage, you have left this page — that is wrong chunk. If you need the full stage-by-stage procedure after recall is known to be low, use RAG debugging.
Which of the five causes kept it out?
Five causes produce the same symptom — the document never reaches the model — and each has a different fix. Pipeline walkthroughs try every stage in order. Faster is differential diagnosis: one measurement per row that rules that cause in or out before you change code.
| Cause | What you see | Detection (run this first) | Fix |
|---|---|---|---|
| 1 · Never indexed | File on disk; no row for its ID in the vector store | index.contains(doc_id) = false | Re-ingest and verify the ID is searchable |
| 2 · Filter excludes it | Indexed and embedded; invisible until filters drop | recall(no-filter) − recall(with-filter) | Metadata filtering |
| 3 · Vocabulary mismatch | Query words ≠ document words; dense returns near-misses | recall@k(sparse) − recall@k(dense) | Hybrid search |
| 4 · Chunk boundary | Answer exists; no single chunk is a good match | answer-in-one-chunk rate | Chunk boundaries |
| 5 · Top-k cutoff | Absent at production k; appears when k is raised | recall@50 − recall@k | Top-k / reranker |
The document is on disk but not in the index
The cheapest miss to confirm, and the one smoke tests never catch: the file exists in your CMS or filesystem, but the vector store has no searchable row for it. Retrieval cannot return what was never a candidate.
Symptom
Operators open the policy PDF and point at the paragraph. Every RAG query about that paragraph returns unrelated chunks — or nothing useful — and the document ID never shows in retrieval logs.
Detection
Look up the document ID or source path directly in the index (the check unrag names as “manually search for the document ID”). If index.contains(doc_id) is false, stop tuning retrieval.
Cause
Ingest skipped the file, the parser failed silently, the doc landed in the wrong namespace, or a delete path removed vectors without a re-add. Barnett et al. (2024) call the broader case Missing Content when the answer is not in the available documents at all.
Fix
Re-run ingest for that source and verify the ID is searchable with a direct lookup before you touch embeddings or prompts. If the document was indexed and later drifted or was orphaned, that is stale index, not a first-time miss.
A filter is silently excluding the right document
A metadata filter that is correct on paper and wrong in practice removes the gold document from the candidate set before ranking runs. The index looks healthy; every filtered query still misses.
Symptom
The document is indexed and well-embedded. It never appears for any phrasing while production filters stay on.
Detection
Re-run the identical query with every metadata filter removed. If the gold document appears, a filter was excluding it. Measure recall(no-filter) − recall(with-filter).
Cause
Tenant, department, language, date, or permission tags on the document do not match the filters applied at query time — or the tags themselves are wrong.
Fix
Audit query-time filters against the document’s metadata. The mechanism page is metadata filtering. This is the inverse of tenant leakage, where a missing filter admits the wrong documents.
The query’s words don’t match the document’s
Dense retrieval encodes meaning, but when the signal is the surface form — an acronym, a synonym the docs never use, a product ID, an exact policy title — the embedding blurs it and the right document never enters the candidate set.
Symptom
The user asks “PTO carryover after 90 days”; the handbook says “unused leave accrual past probation.” Same rule, few shared tokens — dense retrieval returns adjacent policies, never the gold file.
Detection
Re-run the identical query with BM25 only. If sparse finds the gold document and dense does not, vocabulary was the gap. Measure recall@k(sparse) − recall@k(dense); a positive gap is the tell.
Cause
Dense-only retrieval has no lexical channel, so exact-term and rare-token queries fall through even when the document is indexed.
Fix
Add the lexical signal: run BM25 alongside the embedding retriever and fuse the results — hybrid search. The lexical baseline is BM25; the dedicated failure write-up is vocabulary mismatch.
The answer is split across two chunks
Sometimes the document is indexed and the vocabulary matches, but fixed-length splits cut the answer so neither chunk alone is a good match for the query. From the retriever’s point of view the document might as well be missing.
Symptom
Grep finds the answer in the source file. No retrieved chunk contains the full fact; near-miss chunks from other sections fill top-k instead.
Detection
Locate the answer string in the source and compare it to your chunk boundaries. Measure the answer-in-one-chunk rate on a labelled set: how often the full answer lives inside a single chunk.
Cause
Token-count splitting ignores meaning, so the sentence that answers the question lands on the wrong side of a boundary — or is diluted across two undersized fragments.
Fix
This page only diagnoses the miss. Tuning lives at chunk size and overlap; the dedicated failure mode is chunk boundaries. If a near-wrong chunk was returned, switch to wrong chunk.
Raising top-k makes the document appear
Barnett et al. (2024) name this Missed Top-Ranked Documents: the correct document exists in the store, but the retriever never ranks it high enough to enter the top-k fed to the model. Raising k “fixes” the demo and proves the document was a low-ranked candidate, not absent.
Symptom
At production k the gold document is absent. At k=50 it appears at a rank greater than production k. Similarity thresholds that are too aggressive produce the same pattern.
Detection
Set top-k to 50 and re-query. If the gold document appears only then, you have a cutoff or ranking problem. Measure recall@50 − recall@k.
Cause
First-stage similarity is a coarse sort; a weakly related neighbour outranks the exact document, or production k is simply too narrow for hard queries.
Fix
Widen k as a diagnostic, then fix ranking — tune top-k and add a cross-encoder reranker. A permanently huge k is not the production fix. When a wrong neighbour is what actually reached the model, see wrong chunk.
How do you measure that retrieval missed the document?
The metric that names this failure is recall@k (and its RAGAS cousin, context recall) on queries paired with a known gold document: at a given k, did that document appear in the retrieved set? A zero means it was never a candidate. Worked definitions, formulae, and how recall relates to precision, MRR and nDCG live on retrieval metrics — this page only needs the pass/fail reading.
Barnett et al. (2024) takeaway for practitioners: validating a RAG system matters more than assembling one. Build a small labelled set of queries whose gold documents you can name, run recall@k before and after each change, and refuse to trust a smoke-test chat. When recall is low and you still do not know which pipeline stage failed, follow the decision procedure on how to find which stage broke.
Thresholds are local
Do not import someone else’s “good” recall number as a universal constant. Calibrate pass/fail on your own labelled queries and corpus; then track the same metric over time so a missing-document regression cannot hide behind a fluent answer.
Why does RAG miss a document I know exists?
Five causes keep a known document out of the candidate set: it was never indexed, a metadata filter excludes it, query vocabulary does not match the document, the answer is split across chunk boundaries, or production top-k cuts it off below the cutoff. Prove which one with ordered checks — index ID lookup, filters off, raise k, sparse versus dense, boundary grep — before changing prompts or models.
Is a missing document the same as retrieving the wrong chunk?
No. A missing-document failure means the gold document never appears among retrieved candidates. A wrong-chunk failure means a chunk was returned and it does not contain the answer. Different symptom, different measurements — see the wrong-chunk page when something came back and it is simply wrong.
Should I change chunking or top-k first when retrieval misses?
Neither first — prove absence. If the document ID is missing from the index, fix ingest. If it appears only with filters off, fix filters. If it appears only when you raise k, you have a cutoff or ranking problem. If sparse finds it and dense does not, add hybrid. Only when the answer straddles a chunk boundary should you retune chunking.
When is hybrid search better than vector-only for missing documents?
When the miss is vocabulary: exact names, codes, policy titles, SKUs, or synonyms the embedding blurs. If BM25 alone finds the gold document and dense retrieval does not, hybrid search restores the lexical channel. Hybrid does nothing for a document that was never indexed or that a filter excludes.
What metric proves a recall miss?
Recall@k on queries paired with a known gold document — zero at your production k means the document was never a candidate. Context recall in RAGAS asks the same question for the retrieved context. Full formulae and related ranking metrics are on the retrieval-metrics page; calibrate thresholds on your own labelled set.
We updated the document — why does RAG still miss it?
The index may not contain the new version yet, or vectors for the old path were deleted without a re-add. Confirm the document ID is present and searchable after re-ingest. Ongoing freshness and orphaned-vector failures are covered on the stale-index page.