Prompt Injection in RAG: Attacks and Defences

Prompt injection is when text the model reads — a user message, or a retrieved document — carries instructions that override what you told it to do. RAG makes this sharper than in an ordinary LLM app: retrieval’s whole job is to fetch external content and feed it to the model as trusted context, so every retrieved chunk is untrusted input the model treats as trusted. There is no single fix. Defence is layered — and, honestly, prompt injection is not a solved problem.

How does prompt injection work — direct vs indirect?

Two forms, and the difference matters for RAG. Direct injection is the user typing the malicious instruction themselves — “ignore your previous instructions and…”. Indirect injection hides the instruction inside content the model reads later: a web page, an email, or, in RAG, a retrieved document. The root cause is the same for both — the model cannot separate data from instructions; both arrive as tokens in the same context. Indirect is the dangerous one for RAG: the attacker never touches your prompt. They plant text in a document your retriever will later fetch, and let your own pipeline deliver the payload.

Why does RAG create a distinct attack surface?

Because RAG is built to pull external content into the prompt as context — so an attacker who can get text into your corpus can get instructions into your model. That is a shift in the trust boundary most teams miss.

This is the non-malicious version’s evil twin: when retrieval fetches the wrong-but-honest passage you get a wrong-chunk failure; when it fetches an attacker’s planted passage you get injection. The deeper attack of getting malicious documents into the corpus in the first place is data poisoning.

What are the attack patterns in RAG?

The named patterns red-team suites actually fire, one line each:

• Retrieved-content injection — instructions hidden inside a fetched document that the model then obeys.
• Corpus / data poisoning — planting a malicious document crafted to rank well and get retrieved for target queries.
• Data / PII exfiltration — coaxing the model into leaking other users’ data or system content into its answer.
• Source-attribution fabrication — making the model cite a real-looking source that does not actually support the claim.
• Context-window overflow — flooding the context with content to push your real system instructions out of the model’s attention.

The exfiltration and permission side is covered at access control.

Why doesn’t input validation stop it?

Two reasons, and both are counter-intuitive. First, the payload usually isn’t in the user’s input at all — it is in a retrieved document, which never passes through your input filter. Second, even for direct attempts there is no reliable pattern for “malicious instruction”: the model understands paraphrase, encoding, translation and role-play, so any blocklist is trivially bypassed. The consequence sets the whole defensive posture: you cannot filter your way to safety. Design as if some injection will get through, and limit what it can do when it does.

How do you defend a RAG system against injection?

No single control is enough — you layer them so one failing doesn’t mean compromise. Four layers, worst to best return on effort:

• Privilege separation — scope the model’s tools and permissions so that even a fully hijacked model can’t do much. The most important layer: assume the prompt is compromised, and make that survivable.
• Treat retrieved content as untrusted — delimit it clearly as data, and never let a retrieved document trigger a tool call, an action, or a permission change on its own.
• Output filtering — scan the generated answer for leaked data or policy violations before it ships, via guardrails.
• Monitor and log every retrieval and generation, so an attack is at least detectable after the fact — see monitoring RAG.

Honestly: these reduce risk, they do not make injection impossible. Treat it as a standing threat to manage, not a bug you close once.

How do you test for prompt injection?

You red-team it — fire known injection payloads through your live pipeline and see what gets through, rather than assume your defences hold. Two levels: automated red-team suites that send retrieved-content injection, exfiltration and override payloads at the system end to end; and per-component tests (retriever, prompt assembly, generator) so you learn where a payload broke through, not just that it did. Make it a CI gate so a change can’t quietly weaken your injection posture — the same discipline as any other stage-level check.