How Many Chunks Should You Retrieve?

Top-k in RAG retrieval is the number of highest-ranked chunks the retriever returns for one query — the cut-off that becomes the model’s retrieved context. There is no published universal best k: too small and the gold chunk never enters the prompt; too large and noise, token cost and mid-context burial rise faster than answer quality. This page covers the trade-off, how to pick k from Precision@k / Recall@k curves, why bigger stops helping, and when to retrieve wide then rerank down — under the parent retrieval hub.

What happens when top-k is too small or too large?

When top-k is too small, a relevant chunk that ranks just below the cut-off never reaches the model; when it is too large, recall rises while precision falls and the prompt fills with distractors.

• Too small — a miss at the cut-off. In Open WebUI discussion #6210 (qiulang, 2024), a gold chunk sat at retrieval rank 7: top_k = 6 failed the question and top_k = 8 succeeded; after the knowledge base grew past ~20 documents the same chunk slipped to rank 9 and needed a higher cut-off again. Raising k is a diagnostic for top-k cutoff misses, not proof that a huge permanent k is the production fix.
• Too large — noise and burial. Banerjee (LinkedIn, enterprise RAG note) describes the common anti-pattern of cranking k from 5 → 10 → 20 when answers are weak: recall improves (“is the answer somewhere in the set?”) while precision collapses, tokens and latency rise, and relevant passages drift into a bloated middle the model under-uses.
• Starting heuristics are not a law. PE Collective’s 2026 architecture guide suggests starting around K = 5 for simple question-answering and trying K = 10 or 15 for multi-source questions — useful only as a first dial setting. If retrieved chunks are consistently irrelevant, the same guide says the usual fault is chunking or the embedding model, not K alone; chunk length is a separate knob on chunk size.

How do you choose top-k with recall and precision?

You choose top-k by plotting Precision@k and Recall@k (or Hit@k) over increasing k on a labelled query set and stopping near the recall plateau you need — not by picking a blog’s favourite integer.

Precision@k is the fraction of the top-k results that are relevant; Recall@k is the fraction of all known relevant passages that appear anywhere in that window. Relari’s Nuvi evaluation guide (Yi Zhang; Action 2) states the operational rule: read precision and recall at the top K retrieved chunks to decide how many to retrieve. In their illustrative distribution, recall asymptotes near 85% after K@5 while precision keeps dropping — so a system aiming for recall above 80% would set K at least 5 on that curve, then stop adding chunks that only add noise. If the recall floor is only crossed at an impractically large K, the same guide recommends a second-stage reranker or filter rather than stuffing the generator. Worked Precision@k / Recall@k arithmetic lives on retrieval metrics; this page only uses the curve as the decision tool.

Why does bigger top-k stop helping answer quality?

Bigger top-k stops helping once extra chunks no longer raise recall but still add distractors the model mis-weights — especially in the middle of a long prompt.

Liu et al. (2023; TACL 2024) measured a U-shaped accuracy curve on multi-document question answering: models use information best at the start or end of the context and worse in the middle. If your pipeline concatenates top-k chunks in rank order, chunk 1 sits at the front and chunks 2…k−1 drift into that weak middle — so raising k can bury the very evidence you just retrieved. On an open-domain NaturalQuestions-Open case study in the same paper (§5), using more than 20 retrieved documents improved GPT-3.5-Turbo by only about 1.5 percentage points and Claude-1.3 by about 1 percentage point while context length (and cost) kept rising — diminishing returns with a published number, not a slogan. Banerjee’s bottom line matches the failure pattern: if you keep increasing k, you are often compensating for a retrieval problem (wrong neighbours, weak chunking, missing hybrid coverage) rather than fixing generation. Cut mid-context burial on lost in the middle; reorder edges on context ordering; do not treat “add more chunks” as the default repair.

How does top-k interact with the context window and cost?

Top-k multiplies with chunk size to fill the prompt: usable context is roughly chunk_size × top_k plus system and question tokens — the same product already stated on chunk size.

• Window pressure. If that product exceeds what the model can use cleanly, you truncate, drop tail chunks, or pay for a longer window that still suffers mid-context under-use (Liu et al.).
• Token and latency tax. Banerjee lists the hidden cost of higher k: more tokens (spend), longer prompts (latency), and weaker instruction adherence as the prompt dilutes — oversized top-k is also a named stage on RAG latency. Joshi’s dynamic top-k write-up (Medium) treats total LLM token count as an explicit evaluation metric beside answer quality for exactly this reason.
• Operator control. Amazon Bedrock’s knowledge-base retrieve docs advise reducing the maximum number of retrieved results when you need to shorten what fills the $search_results$ placeholder in prompt templates. No published universal dollar or millisecond delta applies across models — measure on your pricing and p95 as of your deployment date (as of July 2026, treat any blog’s $/query claim as stale until re-verified).

Should you retrieve a large top-k and then rerank?

Yes when Recall@small-k is low but Recall@large-k is high: retrieve a wide candidate set for recall, then let a cross-encoder (or other reranker) keep a short high-precision list for generation.

Nuvi’s illustrative path is cosine retrieve of ~100 chunks then rerank down to ~5 before the LLM. Amazon Bedrock Knowledge Bases expose the dual cut-offs explicitly — numberOfResults for the vector search window and numberOfRerankedResults for how many survive reranking (and notes that the reranked count cannot usefully exceed the retrieve count, aside from query-decomposition edge cases). Open WebUI discussion #6210 shows why practitioners ask for two knobs: a gold chunk at retrieval rank 7 can rise to rank 2 after a reranker — so a single small top_k both starves the reranker and fails generation. The quality/latency trade of that second stage is owned by reranking and cross-encoders; this page only fixes the rule that retrieval k and post-rerank k are different numbers.

How do you measure the right top-k for your corpus?

There is no invented universal top-k — the right value is the smallest k that meets your recall (and answer-quality) floor on a labelled set without wasting tokens.

1. Label. Build queries with gold chunk ids (or passages) — the same discipline as retrieval metrics.
2. Sweep. Hold the chunker and embedder fixed; run k ∈ {1, 3, 5, 8, 10, 15, 20, …} (extend if recall is still climbing).
3. Plot. Recall@k, Precision@k (or Hit@k), an end-to-end answer metric, and input token count / p95 latency.
4. Pick. Choose the smallest k at your recall floor that still keeps generation quality — or keep a larger retrieve-k and a smaller post-rerank-k.

Adaptive alternatives exist when a single static k is a bad fit for every query. Adaptive-k (Megagon Labs; arXiv 2506.08479, accepted EMNLP 2025 Main) sorts similarity scores and cuts at the largest gap, then adds a buffer B (the paper sets B = 5) so borderline relevant chunks after the gap are not dropped. Joshi’s Medium project trains a BERT cross-encoder to predict a per-query top-k; on a limited Qasper subset (100 training / 25 test documents — his own resource constraint) he reports dynamic selection similarity ≈0.836, correctness ≈2.93, and average LLM tokens ≈1857, competitive with static k = 10 while using fewer tokens than static k = 5 / k = 10 baselines in that notebook — verify before you rely on those figures; they are preliminary and corpus-specific. Deciding whether to retrieve at all is a different node: adaptive retrieval. Measuring on your labels is mandatory; shipping a blog’s favourite integer is not.