Writing the Prompt for a RAG System

A RAG prompt template is the instruction contract that tells a model how to use retrieved context, how to produce citations tied to that context, and when to refuse. SurePrompts (2026) and Agentset.ai (2026) both treat this as a prompt-layer problem: production reliability improves when grounding, citations, and missing-information handling are enforced in the prompt rather than hoped for at the model level.

How do you structure a RAG prompt (system vs user)?

Structure a RAG prompt by putting non-negotiable control rules in the system prompt, and keeping the per-request payload (question + packed evidence units) in the user prompt. StackAI frames this as a dual prompt structure where the system prompt owns persistent behavior and the user prompt carries the dynamic query and context (StackAI, Nov 17, 2025). SurePrompts describes the same convergent idea as layered prompt surfaces for grounding, citations, and missing information (SurePrompts Editorial Team, April 12, 2026).

• System prompt: grounding (context-only), citation enforcement, missing-information refusal, and conflict policy.
• User prompt: the current question plus the packed context documents with stable IDs.
• Citation wiring: the citation markers you require must map back to the IDs present in the packed context.

If you are tempted to move query reformulation into the same prompt, split the concerns: query rewriting belongs earlier in the workflow. See query-layer prompt work →. For a deeper grounded-generation catalog, see forcing answers from retrieved context →.

What system instructions keep the model inside retrieved context?

The system prompt keeps the model inside retrieved context by explicitly banning outside knowledge and defining what to do when context is insufficient. SurePrompts recommends an “only from provided context” rule combined with an explicit abstention/refusal contract for unsupported questions (SurePrompts Editorial Team, April 12, 2026). AWS’s prescriptive RAG guidance similarly instructs the assistant to base answers on the provided documents/search results and to avoid fabricating when documents do not support the response (AWS Prescriptive Guidance, March 18, 2024).

When this fails in production, it often shows up as hallucination behavior. See hallucination failures → and grounding templates →.

How do you require citations inside a RAG prompt?

Require citations by specifying an exact citation marker format and by enforcing a rule that every factual sentence must carry a citation that points to a retrieved evidence unit. SurePrompts warns that vague language like “cite your sources” behaves like a suggestion; prompt contracts need an explicit, after-the-fact rule plus a fallback when the model cannot cite (SurePrompts Editorial Team, April 12, 2026). Agentset.ai’s prompt templates use mandatory chunk ID notation so the model’s attributions remain traceable to the packed context (Agentset.ai, 2026).

• Marker format: pick one stable form (for example [SRC-1]) and require that exact pattern.
• Placement: append the marker after the sentence that contains the factual claim.
• Enforcement fallback: if a claim cannot be cited to a context ID, omit the claim or trigger missing-information behavior.

For how to validate citation accuracy after the fact, see citing sources in a RAG answer → and generation metrics →.

How do you teach a RAG assistant to say I don’t know?

Teach a RAG assistant to say “I don’t know” by defining a refusal trigger: when the packed retrieved context does not support the answer (including partial-support cases), the assistant must not fill the gap from outside knowledge. SurePrompts calls out missing-information handling as a core prompt-contract clause: confirm what the context supports and explicitly flag what it does not (SurePrompts Editorial Team, April 12, 2026). AWS’s guidance similarly instructs the model to truthfully state it does not know when the documents do not contain sufficient information (AWS Prescriptive Guidance, March 18, 2024).

To tune where “missing evidence” starts in your system, start from the upstream failure symptom: missing document failures →. For the generation-stage abstention contract, see teaching refusal →.

How do you handle conflicts between retrieved chunks in a RAG prompt?

Handle conflicts by requiring the prompt to surface disagreement and cite both chunks, instead of silently choosing one version. SurePrompts recommends conflict-aware templates that present competing facts, attach citations to each version, and only apply precedence when the prompt has a stated reason to prefer one chunk over another (SurePrompts Editorial Team, April 12, 2026). Agentset.ai’s multi-source templates similarly steer the model toward comparison and dual-attribution when sources disagree (Agentset.ai, 2026).

• Surface disagreement: do not hide competing claims.
• Dual citations: every conflicting assertion must cite the context ID that produced it.
• Precedence only if grounded: if your documents include dates/authority labels that justify precedence, use those labels; otherwise present both.

This is the prompt-level control that prevents the failure class where citations “exist” but the answer is still unreliable. See conflicting sources → and then citation depth rules →.

How do you format context documents for reliable generation?

Format context documents as labeled, delimited evidence units with stable IDs so citation markers remain auditable and the model cannot blur boundaries between chunks. Agentset.ai emphasizes chunk ID style attribution as the substrate for citations (Agentset.ai, 2026). AWS’s templates wrap the evidence into a dedicated documents block and constrain the assistant to answer from that block (AWS Prescriptive Guidance, March 18, 2024). SurePrompts also recommends preserving chunk metadata such as source/section labels so the model can cite, compare, and refuse correctly across chunk boundaries (SurePrompts Editorial Team, April 12, 2026).

# Python 3.11+ (standard library only)
from typing import Iterable

def format_context_documents(context_docs: Iterable[dict]) -> str:
    """
    context_docs items must include:
      - id: stable evidence id (e.g., "SRC-1")
      - text: excerpt the model must ground on
      - optional source/title metadata for debugging
    """
    parts = []
    for d in context_docs:
        doc_id = d["id"]
        text = d["text"]
        source = d.get("source")
        if source:
            parts.append(f"[{doc_id}] {source}n{text}")
        else:
            parts.append(f"[{doc_id}]n{text}")
    return "nn".join(parts)

For the end-to-end wiring that passes retrieved chunks into this prompt, see building the pipeline →. For what the citations must look like downstream, see generation-stage citations →.

What runnable prompt contract can you start with?

Start with one runnable contract that combines role separation, grounding, citation enforcement, refusal, conflict handling, and stable chunk IDs. The brief gain is that you get a single, copy-ready system prompt plus user prompt builder (instead of scattered prose rules), which you can then debug with the rule→failure checks in the next section. SurePrompts frames many reliability wins as prompt-layer enforcement in exactly these categories (SurePrompts Editorial Team, April 12, 2026), while StackAI highlights why separating persistent system rules from per-request context reduces prompt drift (StackAI, Nov 17, 2025).

When you connect this to your RAG workflow, wire your retrieved evidence into the contract as a chunk-id labeled context pack in the pipeline you build next: pipeline build →. For the grounded-generation branch of the contract space, see grounding templates →.

# Python 3.11+ (standard library only)
from textwrap import dedent

def format_context_documents(context_docs: list[dict]) -> str:
    parts = []
    for d in context_docs:
        parts.append(f"[{d['id']}]n{d['text']}")
    return "nn".join(parts)

def build_prompts(question: str, context_docs: list[dict]) -> tuple[str, str]:
    """
    Returns:
      - system_prompt (persistent rules)
      - user_prompt (question + packed evidence)
    """
    system_prompt = dedent("""
    You are a RAG assistant.

    GROUNDED ANSWERING
    - Answer ONLY using the provided context documents.
    - Do NOT use outside knowledge or guess.

    CITATIONS (must be enforceable)
    - After every factual sentence, append one or more citation markers in brackets.
    - Citation markers must match the evidence ids in the context, e.g. [SRC-1].
    - If you cannot cite a claim to a context id, omit the claim.

    MISSING INFORMATION (refusal trigger)
    - If the context does not support the answer (or key parts of it), say "I don't know"
      and briefly state what is missing.

    CONFLICTS
    - If the context contains conflicting statements, present both versions and cite each one.
    """).strip()

    context_block = format_context_documents(context_docs)
    user_prompt = dedent(f"""
    QUESTION
    {question}

    CONTEXT DOCUMENTS
    {context_block}
    """).strip()

    return system_prompt, user_prompt

if __name__ == "__main__":
    retrieved = [
        {"id": "SRC-1", "text": "Refunds are processed within 30 days of purchase."},
        {"id": "SRC-2", "text": "Refunds require a receipt for verification."},
    ]
    q = "What is the refund processing window?"
    system, user = build_prompts(q, retrieved)
    print("SYSTEM_PROMPT:\n" + system)
    print("\nUSER_PROMPT:\n" + user)

How do you test a RAG prompt so it fails safely on out-of-context questions?

Test a RAG prompt for safe failure by running out-of-context (and partially answerable) questions and checking that the prompt contract behaves deterministically: unsupported questions trigger “I don’t know,” supported claims include citation markers that match evidence IDs, and conflicting facts are surfaced instead of silently selected. SurePrompts explicitly recommends grounding-and-refusal testing by asking questions you know the retrieved documents should not contain, then verifying the model abstains rather than improvises (SurePrompts Editorial Team, April 12, 2026).

To turn these checks into measurable scoring across datasets, pair the contract tests with generation metrics and then track failure classes like hallucination →.

# Python 3.11+ (standard library only)
import re

REFUSAL_RE = re.compile(r"(i\s*don't\s*know|can't\s*answer|cannot\s*answer)", re.IGNORECASE)
SRC_RE = re.compile(r"\[SRC-\d+\]")

def output_refused(output: str) -> bool:
    return bool(REFUSAL_RE.search(output))

def output_has_context_citations(output: str) -> bool:
    return bool(SRC_RE.search(output))

def diagnose_out_of_context(output: str, supporting_context_present: bool) -> list[str]:
    """
    supporting_context_present:
      - True when your ground-truth dataset says the answer is in the packed context
      - False for out-of-context questions
    """
    issues: list[str] = []

    if not supporting_context_present:
        if not output_refused(output):
            issues.append("missing_refusal_clause")
        return issues

    # For answerable cases, citations should appear at least once.
    if supporting_context_present and not output_has_context_citations(output):
        issues.append("missing_or_unenforced_citation_rule")

    return issues

if __name__ == "__main__":
    cases = [
        {"supporting_context_present": False, "output": "The refund window is 30 days [SRC-1]."},
        {"supporting_context_present": False, "output": "I don't know. The context does not include refund timing."},
        {"supporting_context_present": True, "output": "The refund window is 30 days [SRC-1]."},
        {"supporting_context_present": True, "output": "The refund window is 30 days."},
    ]

    for i, c in enumerate(cases, start=1):
        print(f"case {i} -> {diagnose_out_of_context(c['output'], c['supporting_context_present'])}")

What common RAG prompt mistakes break reliability?

Common RAG prompt mistakes break reliability when they weaken the prompt contract that keeps generation grounded, cited, and abstent on missing information. SurePrompts lists recurring production mistakes such as vague citation instructions, missing refusal clauses, unlabeled context chunks, missing conflict policy, and prompts that mix competing instructions (SurePrompts Editorial Team, April 12, 2026). StackAI also cautions that generic prompts and overloaded instructions can cause inconsistent behavior even when the retrieved context is correct (StackAI, Nov 17, 2025).

• Vague citation instructions: “cite your sources” without sentence-level enforcement (SurePrompts Editorial Team, April 12, 2026).
• No missing-information/refusal clause: the model fills gaps from training data (SurePrompts Editorial Team, April 12, 2026).
• Unlabeled or weakly delimited context: citations become brittle and hard to debug (SurePrompts Editorial Team, April 12, 2026).
• No conflict policy: the model silently picks one version and hides disagreement (SurePrompts Editorial Team, April 12, 2026).
• Mixed-purpose prompts: instructions compete (e.g., “be concise” + “always list every source”), increasing prompt drift (StackAI, Nov 17, 2025).

When you debug, follow the control-plane trail: grounding rules, citation enforcement, and refusal behavior.

When do prompt changes beat retrieval changes?

Prompt changes beat retrieval changes when retrieval already returns the right evidence units, but the model ignores them, misattributes them, or violates grounding/citation/refusal rules. StackAI notes that reliable RAG requires iterative prompt refinement against failure cases, because “good retrieval” does not automatically translate into “grounded outputs” (StackAI, Nov 17, 2025). SurePrompts also frames many production issues as prompt-layer enforcement gaps (grounding, citation, refusal, and conflicts) rather than retrieval alone (SurePrompts Editorial Team, April 12, 2026).

If the symptom is “the answer should be in context but it never appears,” fix retrieval and chunking first. Start from wrong chunk, then move up to retrieval mechanics and chunking strategy. If the evidence is present, then tune the prompt contract.