Why Your RAG Returns the Wrong Chunk

A wrong chunk is a retrieval miss in plain sight: the retriever returned a passage, and that passage does not contain the answer. The model still answers confidently, because it only sees what retrieval handed it. That is different from a document that never entered the candidate set at all, which is when RAG misses the document you know is there, and different from retrieval that succeeded while generation still failed, which is owned by when the right chunk is ignored, hallucination, or conflicting sources. The five causes below share one symptom and five different fixes, so the first job is to read the retrieved chunks and their scores before changing prompts or models. The failures taxonomy places this leaf among retrieval faults sorted by the stage that caused it.

Is a wrong RAG answer a retrieval problem or a generation problem?

A wrong RAG answer is a retrieval problem whenever the gold passage never reached the prompt, and a generation problem only after known-good context still produces a bad answer. AWS’s Dev.to walkthrough on RAG retrieval failures (captured 18 September 2026) states the instinct plainly: people blame the model, yet the model often did exactly what it was told with the wrong evidence. Medium’s Deno Data guide on wrong RAG answers (12 March 2026) splits the same field into retrieval failure versus generation failure and tells practitioners to fix retrieval first.

Latitude’s debug guide (22 July 2026) makes the split operational in two moves. First, for the failing query, pull the exact chunks the retriever returned and their similarity scores, and ask whether the answer string is present, ranked high enough to enter production top-k, and complete inside one chunk. Second, manually hand the model known-good context and regenerate. A good answer under gold context localizes the fault to retrieval. A still-wrong answer under gold context localizes the fault to generation or context assembly, which is no longer this leaf.

Barnett et al. (2024) catalogue seven failure points across three real RAG deployments (arXiv 2401.05856) and land on a blunt operational claim Latitude quotes: robustness evolves in production, so validation has to inspect intermediate states, not only final answers. The widely repeated claim that “70 percent of RAG failures are retrieval” has no primary behind it; Latitude rejects that figure as a heuristic without a source, and this site does not cite it as a measurement. When generation is the side that failed after gold context, follow how to find which stage of your RAG pipeline broke.

Once the fault is on the retrieval side and a chunk did come back, the next question is which of the five retrieval causes produced that wrong chunk.

How do you tell which of the five causes returned the wrong chunk?

Five distinct causes produce the same wrong-chunk symptom, and each has a different fix, so a stage-by-stage walk that always starts at chunking wastes time when the real fault is a filter or a vocabulary gap. Differential diagnosis starts from the symptom and runs the single measurement that rules each cause in or out before any code change. Latitude, AWS, and Zudyog all teach useful checks; none of them publishes one table where each row is a discriminating delta that names the cause.

Read the table as a triage card, not as a ranking of how often each cause appears on your corpus. The deltas are pass/fail on a labelled failing query: you already know which chunk should have won. Aggregate Precision@k and Context Precision still matter for fleet health, and Zudyog’s production example of Context Precision at 0.74 means 26 percent of retrieved chunks were irrelevant noise, but those fleet meters do not name which of the five causes fired on one query. Worked formulas for precision, recall, MRR, and nDCG live on pages about whether the right chunk was ever fetched.

Each row below expands one measurement into symptom, cause, and fix. The first cause to check on identifier-heavy corpora is vocabulary mismatch, because a dense-only index has no lexical channel for the tokens the user actually typed.

Why does a vocabulary mismatch return the wrong chunk?

A vocabulary mismatch returns the wrong chunk when the user’s tokens and the document’s tokens diverge while the meaning stays the same, so dense retrieval ranks near-miss neighbours above the line that actually answers the question. Zudyog’s production post on wrong chunks (captured 18 September 2026) names the pattern as a semantic miss on SKUs, brand fabric terms, and spacing variants such as “mirrorwork” versus “mirror work.” AWS’s “What’s the 90-day rule?” failure is the same geometry under ambiguity: several policies share the phrase, embeddings cannot separate them, and top-k fills with mixed contexts.

Weaviate’s hybrid search documentation (as of September 2026) defines hybrid as combining vector search with keyword BM25F search, then fusing the two result sets, with alpha of 1 as pure vector and alpha of 0 as pure keyword. The same fusion job appears across Weaviate, Qdrant, Elasticsearch, OpenSearch, and Azure AI Search under their own hybrid or RRF APIs. When the gold chunk shares tokens with the query but still never wins as a whole unit, the failure has usually moved from vocabulary into chunk granularity.

How does the wrong chunk size return the wrong answer?

Wrong chunk size returns the wrong answer when the fact that answers the question never sits wholly inside any single retrieved unit, or when an oversized unit dilutes that fact under unrelated sentences. AWS’s home-office stipend failure is the too-small case: retrieval found the right section headings and cross-references, scores around 0.47 looked healthy, and the dollar amounts that answered the question never entered top-k because paragraph splits cut the section short. Zudyog’s blob problem is the too-large case: one product chunk mixes fabric, sizing, care, and price, so the vector averages across concerns and a care query retrieves a diluted neighbour.

A pure boundary-split recall miss, where no returned chunk is even a near neighbour because the answer straddles a seam, is owned by when the answer is split across two chunks. When the gold unit is inside the candidate set and still loses to weaker neighbours, the next cause is ranking, not size.

Why was the right chunk found but ranked too low?

The right chunk was found but ranked too low when first-stage similarity ordered a weakly relevant neighbour above the exact answer, so production top-k never passed the gold passage to the model. Latitude’s debug checklist treats a chunk at position 15 under a top-5 cutoff as functionally not retrieved. Zudyog’s ranking mode tracks the same pattern: the correct chunk appears in the top five but not the top three often enough that the model generates from the wrong evidence.

When similarity scores for the winners look excellent and the content is still unrelated to the question, the geometry of the embedding space itself is the fault, not the cutoff.

Why do high-similarity chunks still come back irrelevant?

High-similarity chunks still come back irrelevant when the embedding places the query near passages that share distributional neighbours but not the answer, so cosine or dot-product confidence and human relevance diverge. Medium’s Deno Data guide lists “high similarity scores but wrong answers” as a named failure mode. Zudyog’s threshold mode shows the production shape: chunks scoring 0.55 to 0.65 still enter the prompt because the score floor was set at 0.50, and the model then answers from low-relevance noise.

Zudyog’s first production week reported Context Precision of 0.74 on its RAGAS meter, which means roughly one in four retrieved chunks was irrelevant. That fleet reading is a reason to open the differential table; it is not itself a diagnosis of which cause dominates. When the gold document never appears for any k or phrasing while the index lookup succeeds, suspect a filter before you retrain embeddings.

How does a metadata filter silently exclude the right document?

A metadata filter silently excludes the right document when tenant, category, date, language, or permission predicates remove the gold row from the candidate set before ranking runs, so every query looks like a retrieval miss even though the document is indexed and well embedded. Latitude’s root-cause table lists missing or too-loose metadata filtering as the fault behind pulls from the wrong document set. AWS’s retrieval toolkit treats metadata filters as the way to shrink the candidate set before scoring, which is exactly where an over-tight filter becomes a silent exclude.

Running the five measurements by hand on every failing query works for a single incident, yet packaging them into one function is what makes the differential repeatable across a golden set and comparable after each deploy.

How do you run all five wrong-chunk checks at once?

You run all five wrong-chunk checks at once by calling one function that takes the failing query, the gold chunk id, and your dense and sparse retrievers, then prints which discriminating delta fired. Medium’s Hit Rate@k evaluator and Zudyog’s audit_retrieval logger both assume a labelled expectation; Welcome Developer’s guide on knowing whether RAG changes helped (captured 18 September 2026) puts a golden dataset at the centre of that loop. The script below is the differential table in code: each block is one row, and ruled-out causes can be deleted.

Build the labelled pairs first. Without a gold chunk id, the function cannot compute recall gaps or a similarity percentile, so it cannot name a cause. How to assemble and keep that file honest is building a golden test set; how to block regressions after a fix is catching quality loss before it ships.

import numpy as np

def diagnose_wrong_chunk(query, gold_id, *, dense, sparse, embed, k=5):
    """Print which of the five wrong-chunk causes fired for one failing query.

    dense(query, k)  -> list[chunk_id] ranked by embedding similarity
    sparse(query, k) -> list[chunk_id] ranked by BM25
    embed(text)      -> np.ndarray  (the same model dense() uses)
    gold_id          -> id of the chunk you KNOW contains the answer
    """
    def recall_at(ids):
        return 1.0 if gold_id in ids else 0.0

    dense_k = dense(query, k)
    sparse_k = sparse(query, k)

    # Cause 1: vocabulary mismatch (sparse finds gold, dense misses)
    if recall_at(sparse_k) - recall_at(dense_k) > 0:
        print("CAUSE 1 vocabulary mismatch -> add hybrid (BM25 + dense)")

    # Cause 3: ranking miss (widening k surfaces the gold chunk)
    dense_50 = dense(query, 50)
    if recall_at(dense_50) - recall_at(dense_k) > 0:
        rank = dense_50.index(gold_id) + 1
        print(f"CAUSE 3 ranking miss (gold at rank {rank} > k={k}) -> add a reranker")

    # Cause 4: embedding mismatch (gold sits low in the similarity distribution)
    q = embed(query)
    sims = {
        cid: float(q @ embed(cid) / (np.linalg.norm(q) * np.linalg.norm(embed(cid))))
        for cid in set(dense_50) | {gold_id}
    }
    gold_sim = sims[gold_id]
    pct = np.mean([gold_sim >= s for s in sims.values()])
    if pct < 0.5:
        print(f"CAUSE 4 embedding mismatch (gold in {pct:.0%} percentile) -> domain embeddings")

    # Cause 5: filter error (gold appears only once filters are dropped)
    dense_nofilter = dense(query, k, filters=None)
    if recall_at(dense_nofilter) - recall_at(dense_k) > 0:
        print("CAUSE 5 filter error -> a metadata filter is excluding the gold chunk")

    # Cause 2: granularity (nothing above fired; inspect the chunk boundary)
    if not any([
        recall_at(sparse_k) - recall_at(dense_k) > 0,
        recall_at(dense_50) - recall_at(dense_k) > 0,
        pct < 0.5,
        recall_at(dense_nofilter) - recall_at(dense_k) > 0,
    ]):
        print("CAUSE 2 likely granularity -> grep the source; check the answer's chunk boundary")

Wire dense and sparse to the same Weaviate collection you serve in production, or to the pair of retrievers your stack already runs, so the printed cause matches live behaviour rather than a toy index. Each print line names the cause; it does not apply the fix. After a cause is confirmed, change only that lever, then re-run the golden pair so the same failure cannot return silently.

A wrong chunk is not a mysterious model failure. It is one of five retrieval faults: vocabulary mismatch, wrong granularity, ranking miss, embedding mismatch, or a silent filter, each isolated by one measurement and fixed by one matching change. Inspect the retrieved set first, run the discriminating deltas on a gold chunk id, and keep the failing query in a golden set so the next deploy cannot resurrect it.