How Many Chunks Should You Retrieve?

Top-k in RAG retrieval is the number of highest-ranked chunks the retriever returns for one query: the cut-off that becomes the model’s retrieved context. There is no published universal best k. Too small and the gold chunk never enters the prompt; too large and noise, token cost and mid-context burial rise faster than answer quality. Relari’s Nuvi evaluation guide, Liu et al. (2023; TACL 2024) on lost-in-the-middle, and Amazon Bedrock’s knowledge-base retrieve docs all treat that cut-off as a measured trade-off, not a blog default. Generation APIs also expose a parameter called top_k that truncates the next-token distribution during decoding; that is unrelated to how many chunks you retrieve. Parent hub: the step the whole architecture is named after.

What happens when top-k is too small or too large?

When top-k is too small, a relevant chunk that ranks just below the cut-off never reaches the model. When it is too large, recall rises while precision falls and the prompt fills with distractors. Both failure modes look like “the model answered wrong,” which is why the cut-off has to be measured against labelled gold chunks rather than guessed from a tutorial.

Too small is a miss at the cut-off. In Open WebUI discussion #6210 (qiulang, 2024), a gold chunk sat at retrieval rank 7: top_k = 6 failed the question and top_k = 8 succeeded; after the knowledge base grew past about 20 documents the same chunk slipped to rank 9 and needed a higher cut-off again. Raising k is a diagnostic for top-k cutoff misses, not proof that a huge permanent k is the production fix.

Too large is noise and burial. Banerjee (LinkedIn, enterprise RAG note) describes the common anti-pattern of cranking k from 5 to 10 to 20 when answers are weak: recall improves on the question “is the answer somewhere in the set?” while precision collapses, tokens and latency rise, and relevant passages drift into a bloated middle the model under-uses. PE Collective’s 2026 architecture guide suggests starting around K = 5 for simple question-answering and trying K = 10 or 15 for multi-source questions. Those integers are first dial settings only. If retrieved chunks are consistently irrelevant, the same guide says the usual fault is chunking or the embedding model, not K alone; chunk length is a separate knob on chunk size.

The live search results for “how many chunks to retrieve” are dominated by chunk-size articles. Guides on ideal chunk size, embedding-first chunking, and chunk size as an experimental variable all answer a different question: how large each offline split should be. Chunk size and top-k multiply into prompt fill, but they are different decisions: size is how documents are split offline; k is how many scored neighbours enter the prompt online. Confusing them is how teams re-chunk when the real miss is a cut-off one rank below the gold chunk.

Weaviate’s hybrid documentation (as of September 2026) still leaves limit and alpha as separate request parameters for that reason: fusion changes who ranks, while the limit decides how many fused neighbours are returned. Changing chunk size without re-measuring Recall@k is the same category of mistake as changing k without re-measuring Precision@k. Both knobs move the prompt; only labelled curves tell you which knob fixed the miss.

How do you choose top-k with recall and precision?

You choose top-k by plotting Precision@k and Recall@k (or Hit@k) over increasing k on a labelled query set and stopping near the recall plateau you need, not by picking a blog’s favourite integer. The curve is the decision tool; the formulas behind the points live on the metrics leaf.

Precision@k is the fraction of the top-k results that are relevant; Recall@k is the fraction of all known relevant passages that appear anywhere in that window. Relari’s Nuvi evaluation guide (Yi Zhang; Action 2) states the operational rule: read precision and recall at the top K retrieved chunks to decide how many to retrieve. In their illustrative distribution, recall asymptotes near 85% after K@5 while precision keeps dropping, so a system aiming for recall above 80% would set K at least 5 on that curve, then stop adding chunks that only add noise. If the recall floor is only crossed at an impractically large K, the same guide recommends a second-stage reranker or filter rather than stuffing the generator.

Worked Precision@k and Recall@k arithmetic lives on retrieval metrics; the decision rule here is to read your own plateau. A curve that never plateaus usually points at a scorer or chunking problem, not at a missing integer between 20 and 50. Hit@k is useful when you only care whether any gold chunk entered the window; MRR and NDCG add rank sensitivity when the generator reads the list in order. Once the plateau is visible, the next question is why answer quality still stalls even when recall is still climbing a little.

Teams that skip the labelled set often substitute vibes: raise k until a demo question works, then ship that integer. Open WebUI #6210 shows why that fails as the corpus grows: the gold rank drifts, and yesterday’s k becomes today’s miss. Relari’s rule still holds after growth: re-plot the curve on a fixed query set when the index changes, then move k only if the plateau moved.

Why does bigger top-k stop helping answer quality?

Bigger top-k stops helping once extra chunks no longer raise usable recall but still add distractors the model mis-weights, especially in the middle of a long prompt. Past the recall plateau, each added neighbour is mostly noise with a token bill attached.

Liu et al. (2023; TACL 2024) measured a U-shaped accuracy curve on multi-document question answering: models use information best at the start or end of the context and worse in the middle. If your pipeline concatenates top-k chunks in rank order, chunk 1 sits at the front and chunks 2 through k-1 drift into that weak middle, so raising k can bury the very evidence you just retrieved. On an open-domain NaturalQuestions-Open case study in the same paper (section 5), using more than 20 retrieved documents improved GPT-3.5-Turbo by only about 1.5 percentage points and Claude-1.3 by about 1 percentage point while context length and cost kept rising. Those are diminishing returns with published numbers, not a slogan.

Banerjee’s bottom line matches the failure pattern: if you keep increasing k, you are often compensating for a retrieval problem (wrong neighbours, weak chunking, missing hybrid coverage) rather than fixing generation. Cut mid-context burial on lost in the middle; reorder edges on context ordering; do not treat “add more chunks” as the default repair. Hybrid coverage belongs on the hybrid leaf when identifiers are the miss; vocabulary mismatch belongs on query rewriting when the phrasing is the miss. The cost side of the same trade-off is the next lever.

The U-curve also explains why a reranker can look like magic after a wide retrieve: it shortens the list the generator sees, which reduces how much evidence sits in the weak middle. That is a precision intervention after a recall intervention, not proof that static k=100 is healthy for generation. Keep the wide window for the reranker; keep the short window for the prompt.

How does top-k interact with the context window and cost?

Top-k multiplies with chunk size to fill the prompt: usable context is roughly chunk_size × top_k plus system and question tokens. That product is already stated on the chunk-size leaf, and it is why two teams can share the same k and still blow different budgets when their chunkers disagree.

Window pressure appears first. If that product exceeds what the model can use cleanly, you truncate, drop tail chunks, or pay for a longer window that still suffers mid-context under-use (Liu et al.). Token and latency tax appear next. Banerjee lists the hidden cost of higher k: more tokens (spend), longer prompts (latency), and weaker instruction adherence as the prompt dilutes. Oversized top-k is also a named stage on RAG latency. Joshi’s dynamic top-k write-up (Medium) treats total LLM token count as an explicit evaluation metric beside answer quality for exactly this reason.

Operator control exists in managed stacks. Amazon Bedrock’s knowledge-base retrieve docs advise reducing the maximum number of retrieved results when you need to shorten what fills the $search_results$ placeholder in prompt templates. No published universal dollar or millisecond delta applies across models: measure on your pricing and p95 as of your deployment date. As of September 2026, treat any blog’s dollars-per-query claim as stale until re-verified. When recall@small-k is the real problem, the next pattern is retrieve wide and rerank down, not an unbounded context window.

The arithmetic is simple enough to keep beside the curve. If mean chunk size is 400 tokens and k is 20, the retrieved block alone is about 8,000 tokens before system instructions and the user question. Doubling k to 40 doubles that block; it does not double answer quality once you are past the recall plateau. Joshi’s use of token count as an evaluation metric is the discipline that makes that visible: plot quality against tokens, not only quality against k.

Should you retrieve a large top-k and then rerank?

Yes when Recall@small-k is low but Recall@large-k is high: retrieve a wide candidate set for recall, then let a cross-encoder (or other reranker) keep a short high-precision list for generation. Dual cut-offs are the design, not a single integer pretending to do both jobs.

Nuvi’s illustrative path is cosine retrieve of about 100 chunks then rerank down to about 5 before the LLM. Amazon Bedrock Knowledge Bases expose the dual cut-offs explicitly: numberOfResults for the vector search window and numberOfRerankedResults for how many survive reranking. Bedrock notes that the reranked count cannot usefully exceed the retrieve count, aside from query-decomposition edge cases. Open WebUI discussion #6210 shows why practitioners ask for two knobs: a gold chunk at retrieval rank 7 can rise to rank 2 after a reranker, so a single small top_k both starves the reranker and fails generation.

The quality and latency trade of that second stage is owned by reranking and cross-encoders. Retrieval k and post-rerank k are different numbers. Hybrid stacks still need both: Weaviate’s hybrid search (as of September 2026) can fuse BM25F and vector results with alpha and relativeScoreFusion, then hand a wider shortlist to a reranker. The fusion does not remove the need to choose how many fused neighbours enter the rerank stage.

A practical starting shape on many corpora is retrieve 20 to 50, rerank to 3 to 8, then measure. Those ranges are heuristics for the first sweep, not laws. If Recall@20 is already at your floor, a retrieve of 100 only buys reranker latency. If Recall@5 is 40% and Recall@50 is 85%, dual-k is the intervention that matches the curve. Nuvi’s 100-then-5 illustration is one point on that family of designs, not a universal recipe.

How do you measure the right top-k for your corpus?

There is no invented universal top-k. The right value is the smallest k that meets your recall and answer-quality floor on a labelled set without wasting tokens. Measurement is mandatory; shipping a blog’s favourite integer is not.

The procedure is fixed enough to repeat. Label queries with gold chunk ids or passages, using the same discipline as retrieval metrics. Hold the chunker and embedder fixed; run k in {1, 3, 5, 8, 10, 15, 20, …} and extend if recall is still climbing. Plot Recall@k, Precision@k (or Hit@k), an end-to-end answer metric, and input token count or p95 latency. Pick the smallest k at your recall floor that still keeps generation quality, or keep a larger retrieve-k and a smaller post-rerank-k.

Adaptive alternatives exist when a single static k is a bad fit for every query. Adaptive-k (Megagon Labs; arXiv 2506.08479, accepted EMNLP 2025 Main) sorts similarity scores and cuts at the largest gap, then adds a buffer B (the paper sets B = 5) so borderline relevant chunks after the gap are not dropped. Joshi’s Medium project trains a BERT cross-encoder to predict a per-query top-k; on a limited Qasper subset (100 training / 25 test documents, his own resource constraint) he reports dynamic selection similarity about 0.836, correctness about 2.93, and average LLM tokens about 1857, competitive with static k = 10 while using fewer tokens than static k = 5 / k = 10 baselines in that notebook. Verify before you rely on those figures; they are preliminary and corpus-specific.

Deciding whether to retrieve at all is a different node: adaptive retrieval. Self-RAG and Adaptive-RAG style controllers answer “should this question call the index?”; Adaptive-k and learned top-k answer “how many neighbours once you retrieve?” Keep those questions separate in the experiment log. Dynamic k is optional; a labelled sweep against a static baseline is not. Ship the smallest static k that meets the floor first, then test adaptive methods only if the per-query variance on that baseline is large enough to justify the extra machinery.

Top-k is the retrieval cut-off that becomes the model’s context, not the decoding parameter that truncates next-token candidates. Too small misses gold below the line; too large buys recall with precision loss, token cost and mid-context burial, as Liu et al. measured and as Banerjee’s anti-crank pattern describes. Choose k from Precision@k and Recall@k on your labels, prefer dual cut-offs when you retrieve wide and rerank down (Bedrock’s numberOfResults versus numberOfRerankedResults), and treat Adaptive-k or learned dynamic k as optional once the static curve is known. Chunk size remains a separate offline knob; the online decision is how many scored neighbours the generator is allowed to see.