Skip to content
RAG Explained Better

Teaching a RAG System to Say I Don’t Know

Abstention thresholds and how to set them, plus the measurable cost of refusing too often.

Mrunmay Phanse Updated 2026-08-01

Refusal / abstention in RAG is the generation-time decision to not answer when retrieved evidence is insufficient — say I don’t know (or escalate) instead of guessing. It is a policy on a confidence signal, not a vibes prompt alone.

Why should a RAG system say I don’t know?

A RAG system should say I don’t know because a fluent wrong answer from weak or off-corpus context is worse than a short refusal for support and handbook bots: users cannot reliably tell fabricated guidance from policy (Yushi, 2026; Pamela Fox, 2024). When the system follows a “answer only from provided context” instruction and refuses on coverage gaps, it reveals knowledge-base omissions and builds trust over time instead of silently falling back to general knowledge (Yushi, 2026). This refusal path is the hard clause of grounded generation, and weak refusal is a common upstream contributor to hallucination despite context.

In production, two jobs matter:

  • It prevents ungrounded fallback. Yushi describes the failure mode as “fake answers”: the prompt needs to forbid guessing when the context does not cover the question.
  • It makes missing documents visible. A clear “I don’t know” response tells users that the corpus does not contain the needed material, which creates an actionable signal for humans to fix the docs.

How does a RAG system decide to refuse?

A RAG system decides to refuse in three places: (1) the prompt clause that instructs the model what to do on coverage gaps, (2) a retrieval rejection gate that can skip the LLM when confidence is too low, and (3) a post-generation self-check that can trigger an abstention when the generated answer is not supported by the sources (Yushi, 2026; Tolga/Simprasuite, 2026; Pamela Fox, 2024).

A practical refusal stack runs in this order:

  1. Prompt-level refuse clause. The prompt tells the model to answer only from the provided context and to say it does not know when the context doesn’t cover the question (Yushi, 2026).
  2. Retrieval rejection gate. The retrieval layer converts similarity signals into a confidence scale, filters low-quality chunks, and can reject directly (skipping LLM generation) when the average confidence is below a threshold (Tolga/Simprasuite, 2026). Tolga/Simprasuite use MIN_CHUNK_CONFIDENCE=25 to decide what can enter context, and MIN_CONFIDENCE_GATE=40 to decide when to skip the LLM.
  3. Post-generation verifier. After generating an answer, the system can run an additional check step that asks whether the answer is actually supported by the retrieved sources, then abstains if it is not (Pamela Fox, 2024).

Don’t rely on a refusal prompt alone

Tolga/Simprasuite argue that similarity-based RAG is a ranking signal, and the system needs deterministic confidence gating; otherwise it can still produce “wrong-but-confident” answers on weak context (Tolga/Simprasuite, 2026).

Refusal stack in RAG: prompt clause, retrieval rejection gate with two-tier thresholds, and a post-generation self-check decide between answering and abstaining.
A refusal stack chooses between “answer” and abstain using prompt instructions, retrieval confidence thresholds, and a post-generation support check.

How do you set an abstention threshold?

Set an abstention threshold by calibrating retrieval confidence on your own labeled traffic so you can trade missed correct answers (false rejects) against wrong-but-confident answers (hallucinations) (Tolga/Simprasuite, 2026).

Four ordered steps to set an abstention threshold. One, log avg_confidence for real queries in production. Two, hand-label outcomes as correct versus incorrect. Three, plot a confidence histogram for each group and choose a gate where the distributions intersect. Four, sweep candidate gates and record the trade-off; Tolga/Simprasuite show gate values in 30, 35, 40, 45, 50.
Tolga/Simprasuite’s abstention-threshold recipe is four measurements: log avg_confidence on real traffic, hand-label correct vs incorrect, plot the confidence histograms and pick the intersection gate, then sweep candidate gates (30–50 in their write-up) and record missed answers versus hallucinations.

Tolga/Simprasuite’s practical recipe:

  1. Log avg_confidence for real queries in production.
  2. Hand-label outcomes as correct vs incorrect.
  3. Plot a confidence histogram for each group and choose a gate where the distributions intersect.
  4. Sweep candidate gates and record the trade-off (Tolga/Simprasuite show gate values in (30, 35, 40, 45, 50) and output missed vs hallucinations).

Two-tier designs matter because chunk-entry and answer-entry are different decisions: Tolga/Simprasuite use MIN_CHUNK_CONFIDENCE=25 to decide what can enter the context and MIN_CONFIDENCE_GATE=40 to decide when to skip the LLM entirely (Tolga/Simprasuite, 2026). Tolga/Simprasuite also caution that similarity is primarily a ranking signal, so nearest-k results can still be irrelevant in absolute terms—so thresholding must be based on calibrated confidence, not on raw similarity alone. If the right evidence never enters candidates, fix the retrieval recall first; the failure mode is missing document, not too-aggressive abstention.

What does refusing too often cost?

Refusing too often costs coverage: the system declines benign questions it could have answered from retrieved evidence. In the RagRefuse over-refusal paper, Maskey et al. (2025) report baseline over-refusal of 53.4% for Llama-3.1-8B-Instruct, reduced to 4.3% by SafeRAG-Steering; for Qwen1.5-7B-Instruct, the baseline is 4.7% and steering eliminates over-refusal entirely (to no refusal cases).

Where this shows up depends on how your pipeline triggers abstention:

  • User-facing risk: missed correct answers. Over-refusal is literally refusing benign-intent prompts, so users experience “no answer” even when the system had enough evidence.
  • Compute risk: the latency/compute shape shifts lower. In Tolga/Simprasuite’s retrieval-gated design, rejected queries return in about 48 ms because the LLM is skipped, while accepted queries typically take 800-2000 ms for the first token.

This is why abstention thresholds must be tuned on your own labeled set: refuse too rarely and you emit unsupported answers, but refuse too often and you suppress correct ones (Pamela Fox, 2024; Tolga/Simprasuite, 2026).

How do you measure whether the system actually abstains?

Measure abstention using an unanswerable eval set and a dont-know metric that scores whether the output conveys “I don’t know,” then separately measure over-refusal rate on benign-intent prompts. Pamela Fox (2024) describes a dontknowness metric scored 1-5 stars where 5 means the answer says straightforwardly that it does not know and makes no attempt to answer, and 1 means the answer fully answers with no uncertainty. Pamela also defines dont-know test categories as uncitable (unrelated vs related), unknowable, and nonsensical questions, and reports an example where her RAG app responds with “I don’t know” about 68% of the time (passing at 4 or 5).

To quantify “refusing too often,” Maskey et al. (2025) define over-refusal as direct or indirect refusal on benign-intent queries and the over-refusal rate as the fraction of benign prompts whose outputs are judged as refusal by an LLM judge (direct_refusal or indirect_refusal).

That split matters because dontknowness is an abstention meter (is the system refusing?), while hallucination/faithfulness meters are about whether an emitted answer contradicts or is supported by context. Use RAG test sets to build the unanswerable probes, then use generation metrics for the already-emitted answers.