Why Your RAG Returns the Wrong Chunk

Your retriever returned a chunk. It is the wrong one — the answer is plainly in your documents, but the chunk that came back does not contain it. This page is about that exact failure: a chunk was returned and it is wrong. It is not the case where nothing relevant was retrieved at all (that is a recall failure — see missing document), and it is not the case where retrieval was fine but the model still answered badly (see lost in the middle).

Before you change anything, look at what actually came back. Log the top-k chunks your retriever returned for the failing query, together with their similarity scores, and read them. The retriever has no way to tell you it fetched the wrong thing — it hands the chunks over and the model answers confidently from bad source. Reading the retrieval, not the answer, is where every diagnosis starts.

First, which of the five causes is it?

A wrong chunk has five distinct causes, and each has a different fix. Applying the wrong fix — adding a reranker when the problem is vocabulary, say — costs time and changes nothing. Most debugging guides walk the pipeline stage by stage: check chunking, then embeddings, then add hybrid, then a reranker, tuning each in turn. That works, eventually. The faster path is differential diagnosis: start from the one symptom, and run the single measurement that rules each cause in or out. Each row below is a measurement you run before you change code, not a fix you try and hope.

The query’s words don’t match the document’s

The most common cause. Your dense retriever encodes meaning, but when the signal is the surface form — an acronym, a synonym the docs never use, a product ID, an exact error code — the embedding blurs it and the right chunk never surfaces.

The chunk is the wrong size for the question

Even with perfect retrieval, a badly sized chunk cannot carry the answer: too large and the relevant sentence is diluted by unrelated text; too small and it lacks the context that makes it an answer; or the answer straddles a boundary and no single chunk holds it whole.

The right chunk was found but ranked too low

Sometimes the correct chunk is in the candidate set — it just sits below your top-k cutoff, because vector similarity is not the same as relevance. A stale document at 0.92 cosine similarity will happily outrank the correct, current one at 0.87; the score is high, the ranking is wrong.

High similarity, still irrelevant

The confusing one: the retriever reports a high similarity score and returns a chunk that is simply not about the query. High confidence, wrong content.

A filter is silently excluding the right document

The cheapest cause to confirm, and the easiest to miss: a metadata filter that unintentionally removes the correct document from the candidate set before ranking ever happens.

How do you run all five checks at once?

Rather than reason about the five causes one at a time, run them all at once. Given a failing query, the id of the chunk you know is correct, and your retriever, this prints which cause fired. It depends only on a sparse and a dense retriever you already have, plus your embedding model for the similarity check.

import numpy as np

def diagnose_wrong_chunk(query, gold_id, *, dense, sparse, embed, k=5):
    """Print which of the five wrong-chunk causes fired for one failing query.

    dense(query, k)  -> list[chunk_id] ranked by embedding similarity
    sparse(query, k) -> list[chunk_id] ranked by BM25
    embed(text)      -> np.ndarray  (the same model dense() uses)
    gold_id          -> id of the chunk you KNOW contains the answer
    """
    def recall_at(ids):                       # 1 if gold is in the list, else 0
        return 1.0 if gold_id in ids else 0.0

    dense_k  = dense(query, k)
    sparse_k = sparse(query, k)

    # Cause 1 — vocabulary mismatch: sparse finds it, dense doesn't
    if recall_at(sparse_k) - recall_at(dense_k) > 0:
        print("CAUSE 1 vocabulary mismatch -> add hybrid (BM25 + dense)")

    # Cause 3 — ranking miss: widening k surfaces the gold chunk
    dense_50 = dense(query, 50)
    if recall_at(dense_50) - recall_at(dense_k) > 0:
        rank = dense_50.index(gold_id) + 1
        print(f"CAUSE 3 ranking miss (gold at rank {rank} > k={k}) -> add a reranker")

    # Cause 4 — embedding mismatch: gold sits low in the similarity distribution
    q = embed(query)
    sims = {cid: float(q @ embed(cid) / (np.linalg.norm(q) * np.linalg.norm(embed(cid))))
            for cid in set(dense_50) | {gold_id}}
    gold_sim = sims[gold_id]
    pct = np.mean([gold_sim >= s for s in sims.values()])
    if pct < 0.5:
        print(f"CAUSE 4 embedding mismatch (gold in {pct:.0%} percentile) -> domain embeddings")

    # Cause 5 — filter error: gold appears only once filters are dropped
    dense_nofilter = dense(query, k, filters=None)
    if recall_at(dense_nofilter) - recall_at(dense_k) > 0:
        print("CAUSE 5 filter error -> a metadata filter is excluding the gold chunk")

    # Cause 2 — granularity: nothing above fired, so inspect the chunk boundary
    if not any([recall_at(sparse_k) - recall_at(dense_k) > 0,
                recall_at(dense_50) - recall_at(dense_k) > 0,
                pct < 0.5,
                recall_at(dense_nofilter) - recall_at(dense_k) > 0]):
        print("CAUSE 2 likely granularity -> grep the source; check the answer's chunk boundary")

The script is deliberately transparent: each block is one row of the detection table above, so you can delete the causes you have already ruled out. It reports the cause, never guesses the fix for you.