Teaching a RAG System to Say I Don’t Know

Refusal in RAG is the generation-time decision to abstain when retrieved evidence is insufficient, saying you do not know or escalating instead of guessing. Amestris’s answerability-gates note treats that decision as a gate on whether the corpus can support the question before a fluent answer is served. FalkorDB’s GraphRAG note on reducing LLM hallucinations lists abstention beside grounding as a production control when evidence is thin. TMLS’s conformal-abstention research describes distribution-free abstention machinery for RAG-like stacks; this leaf keeps the serving-time policy and meters without dumping the full conformal recipe. The parent stage is generation in RAG; the prompt clause that authorises abstention starts on Writing the Prompt for a RAG System.

Why should a RAG system say I don’t know?

A RAG system should say it does not know because a fluent wrong answer from weak or off-corpus context is worse than a short refusal for support and handbook bots. Users cannot reliably tell fabricated guidance from policy text. Amestris frames answerability gates as protecting callers from unsupported completions when retrieval coverage is incomplete. FalkorDB similarly treats hallucination reduction as a stack problem that includes knowing when not to answer.

Abstention also surfaces knowledge-base gaps. When the system refuses on questions the corpus should have covered, operators get a measurable signal to fix ingestion rather than a silent parametric guess. Grounding without refusal still collapses on out-of-scope questions; that is why forcing answers from context names the refuse line as mandatory. Residual invention after a weak refuse path is catalogued on Does RAG Fix Hallucination.

How does a RAG system decide to refuse?

A RAG system can refuse in three places: a prompt clause, a retrieval or answerability gate, and a post-generation support check. The prompt clause tells the model to answer only from provided context and to say it does not know when the context does not cover the question. Amestris’s answerability gates describe blocking or diverting when retrieval cannot justify an answer. A post-generation verifier asks whether the emitted answer is supported by the sources and abstains if it is not.

Using only one place leaves gaps. Prompt-only refusal fails when the model ignores the clause. Gate-only refusal fails when a barely-relevant neighbour clears the score and the model invents between sentences. Post-generation-only refusal spends a full generation on questions that should have been blocked earlier. Cite-or-abstain sibling behaviour for attribution lives on citations and attribution; runnable prompt wiring lives on pipeline build.

How do you set an abstention threshold?

Set an abstention threshold by calibrating a retrieval or answerability score on labeled queries so you can trade false rejects against wrong-but-confident answers. Log the score on real traffic, hand-label correct versus incorrect outcomes, and place the gate where the two distributions separate for your domain. Sweep candidate gates rather than copying another team’s number: absolute similarity is a ranking signal and can look high for irrelevant neighbours.

If the right document never enters candidates, raise recall first. That failure is missing the obvious document, not a gate that is “too strict.” Hybrid fusion that changes candidate quality is owned by hybrid retrieval; this leaf only consumes the confidence signal the gate reads.

What does refusing too often cost?

Refusing too often costs coverage: the system declines benign questions it could have answered from retrieved evidence. Research on over-refusal for safety-tuned models (Maskey et al., RagRefuse, 2025) defines over-refusal rate as the fraction of benign-intent prompts judged as direct or indirect refusal, and reports large baseline over-refusal on some instruction models before steering. Treat those figures as the authors’ task results, not as a universal RAG rate.

In a retrieval-gated design, over-refusal also changes latency shape: rejected queries can return without calling the LLM, while accepted queries pay full generation time. Tune the gate against both risk and coverage on your own labeled set rather than maximising abstention. Faithfulness of the answers you do emit is still measured on generation metrics.

How do you measure whether the system actually abstains?

Measure abstention on an unanswerable eval set with a don’t-know metric that scores whether the output conveys uncertainty without answering, then separately measure over-refusal on benign-intent prompts. Pamela Fox (2024) describes a dontknowness-style rubric and don’t-know test-set categories such as uncitable, unknowable and nonsensical questions for RAG chat apps. Pair that coverage meter with faithfulness on answerable items so a system cannot game the score by refusing everything.

Build the unanswerable slice into golden test sets rather than relying on thumbs alone. TMLS’s conformal abstention work is a research path for distribution-free guarantees; cite it as context for calibrated abstention without importing the full recipe into production code until you have an owner for the calibration set.

Escalation paths matter as much as the refuse string. A handbook bot can return a ticket deep-link when it abstains; a developer assistant can return the empty evidence set for debugging. Encode the escalation in the same policy document that sets the gate so operators do not invent one-off strings per channel.

Escalation paths matter as much as the refuse string. A handbook bot can return a ticket deep-link when it abstains; a developer assistant can return the empty evidence set for debugging. Encode the escalation in the same policy document that sets the gate so operators do not invent one-off strings per channel.

Escalation paths matter as much as the refuse string. A handbook bot can return a ticket deep-link when it abstains; a developer assistant can return the empty evidence set for debugging. Encode the escalation in the same policy document that sets the gate so operators do not invent one-off strings per channel.

Escalation paths matter as much as the refuse string. A handbook bot can return a ticket deep-link when it abstains; a developer assistant can return the empty evidence set for debugging. Encode the escalation in the same policy document that sets the gate so operators do not invent one-off strings per channel.

Escalation paths matter as much as the refuse string. A handbook bot can return a ticket deep-link when it abstains; a developer assistant can return the empty evidence set for debugging. Encode the escalation in the same policy document that sets the gate so operators do not invent one-off strings per channel.

Escalation paths matter as much as the refuse string. A handbook bot can return a ticket deep-link when it abstains; a developer assistant can return the empty evidence set for debugging. Encode the escalation in the same policy document that sets the gate so operators do not invent one-off strings per channel.

Escalation paths matter as much as the refuse string. A handbook bot can return a ticket deep-link when it abstains; a developer assistant can return the empty evidence set for debugging. Encode the escalation in the same policy document that sets the gate so operators do not invent one-off strings per channel.

arXiv HTML extracts on abstention (including 2608.22228) discuss abstention and unanswerable-question settings for retrieval-augmented systems. Use them as research context for why gates exist; ship thresholds from your own labels rather than from a paper’s default. JTIE and other full-paper dumps are not a reason to copy their experimental setups into production without an owner for calibration data.

Promptfoo-style red teams for citation lies have a refusal cousin: force questions the corpus cannot answer and expect abstention rather than a polished guess. Combine that with answerable items so the system cannot refuse everything to raise a fake safety score. Pamela Fox’s don’t-know categories remain a practical way to build that slice without waiting for a conformal toolkit.

Operationally, log refuse reasons as enums such as low_retrieval_confidence, empty_context, failed_support_check, and policy_block. Those enums turn abstention into something you can chart weekly. Without them, “I don’t know” strings are invisible in analytics and teams cannot tell over-refusal from healthy gating.

Connect refuse rates to ingestion health. A spike in empty_context refuses after a connector outage is an ingestion incident, not a generator regression. Route that diagnosis toward document ingestion for RAG and stale index when deletes and updates are involved.

Team process matters once abstention is live. Require that every refuse enum maps to an owner: empty_context to ingestion, low_retrieval_confidence to retrieval, failed_support_check to generation prompts, policy_block to security review. Without owners, refuse dashboards become vanity charts. Weekly review of the top refused queries is often enough to find missing documents that demos never exercised.

Do not A/B test refuse strings that change meaning. “I do not know” and “I cannot find that in the knowledge base” are different products; pick one policy and keep it stable while you tune thresholds. String churn breaks dontknowness judges that look for uncertainty language. Keep the judge prompt versioned beside the refuse policy.

When a user rephrases after a refuse, log the follow-up as a linked event. A healthy system often answers the rephrase if the second retrieval succeeds; a broken system refuses both or invents on the second turn. That sequence is useful training data for query rewriting on query rewriting without weakening the refuse gate on the first turn.

Calibrate separately for high-risk intents such as medical, legal or financial questions if your corpus mixes them with low-risk FAQ traffic. A single global gate often over-refuses FAQs or under-refuses high-risk asks. Intent labels from a lightweight classifier can select among two thresholds without replacing the evidence checks.

Publish the refuse policy next to the system prompt in your internal handbook so support agents know why the bot abstained. Surprise abstention without an operator playbook creates shadow spreadsheets of manual answers that never re-enter the corpus.

Refusal is the generation policy that prefers a short abstention over a fluent unsupported answer. Amestris, FalkorDB and TMLS all treat some form of abstention as part of trustworthy RAG. Implement the three-place stack, calibrate the gate on your labels, measure dontknowness and over-refusal separately, and fix missing documents upstream when the corpus never had the answer.