Prompt Injection

Prompt injection is text that changes the model's behavior against the intent of the system's designers. Models read instructions and data in the same stream of text, and they cannot reliably tell them apart. Unlike SQL injection, there is no escaping function that fixes it.

That makes it the defining security problem of AI applications. The right goal is not "prevent all injection" but "make sure a successful injection cannot cause serious harm."

Direct and indirect injectionCopy Anchor Link

Direct injection is mostly a problem when the user is the attacker, which in an enterprise is less common. The user can only affect their own session, and permissions still apply.

Indirect injection is the serious threat. The attacker is not the user. The attacker plants text that some other user's assistant will read, and the assistant then acts with that user's permissions.

What attackers try to achieveCopy Anchor Link

Notice that most primary defenses are outside the model.

Defenses in layersCopy Anchor Link

1. Limit what the model can do (most important)Copy Anchor Link

• Write tools require human confirmation showing exact arguments. Injected text cannot click a button. See Agents and tools.
• Narrow tool scopes. A tool that can only create tickets in the user's own queue limits damage.
• No tools that send data to arbitrary destinations without confirmation: no free-form HTTP fetch, no email to any address without review.
• Separate high-risk contexts. When the assistant processes untrusted external content, such as inbound email or web pages, consider disabling write tools for that turn entirely.

2. Limit what output can doCopy Anchor Link

• No external images. The single most effective exfiltration defense. See Rendering output.
• Links: show the real destination, block javascript:, optionally require confirmation for unknown domains.
• Content Security Policy in the browser as a backstop.
• No HTML rendering of model output.

3. Separate instructions from data in the promptCopy Anchor Link

• Put retrieved content and tool results inside clear delimiters, such as <source> tags, and escape the delimiter text inside content so a document cannot close the tag early.
• Tell the model that content inside these tags is reference material, and that instructions within it must be ignored.
• Keep system instructions in the system role. Never put retrieved content in the system role.

This reduces success rates substantially but does not eliminate them. Treat it as one layer, not a solution.

4. Detect injection attemptsCopy Anchor Link

• Run a classifier on retrieved chunks and tool results to flag likely injection text, such as instructions addressed to an AI, hidden text, or unusual formatting.
• Flag at ingestion time as well as query time. A document with injected instructions can be quarantined and reported to its owner.
• Detect hidden text during parsing: white-on-white text, zero-size fonts, HTML comments, and invisible unicode characters are common hiding places. Strip or flag them.

Classifiers miss novel attacks and produce false positives. Use them to reduce risk and to alert, not as the only control.

5. Monitor and respondCopy Anchor Link

• Log when injection classifiers fire, when tool calls are rejected by users, and when output filters block links or images.
• Alert on patterns: many users hitting the same flagged document, or a spike in rejected tool calls after reading a particular source.
• Have a process to quarantine a document from the index quickly.

What not to rely onCopy Anchor Link

TestingCopy Anchor Link

Maintain injection cases in the safety dataset and run them on every change. See Red teaming. Cover at least:

• Injection in an indexed document, an uploaded file, a tool result, and an email or ticket body.
• Goals: exfiltration through images and links, tool calls, misinformation, system prompt extraction.
• Hiding techniques: HTML comments, invisible characters, white text, text in image alt attributes, instructions in other languages.
• Multi-turn: injection that activates on a later question.

Measure injection success rate, and make successful exfiltration or unconfirmed actions a zero-tolerance gate.

ChecklistCopy Anchor Link

• Write tools require explicit user confirmation with exact arguments.
• No tool can send data to arbitrary external destinations without confirmation.
• Model output cannot load external images, and links show their real destination.
• A Content Security Policy restricts image and script sources.
• Retrieved content is delimited, with delimiters escaped, and never placed in the system role.
• Ingestion strips or flags hidden text.
• An injection classifier flags suspicious chunks and tool results, with alerting.
• Documents can be quarantined from the index quickly.
• Injection test cases run on every change, with zero tolerance for exfiltration and unconfirmed actions.