What is a Prompt Injection?
Prompt Injection
Prompt Injection is a security vulnerability where a malicious user provides input that overrides the pre-configured system instructions or safety alignment filters of a Large Language Model, hijacking its control flow.
Why It Matters for AI Builders
Defines the safety alignment and security constraints of user-facing systems during vulnerability analysis, llm firewall design, and input sanitation; implementing Prompt Injection helps builders isolate instructions from injection exploits.
Detailed Deep Dive
Prompt injection is a security vulnerability where a user inserts malicious text instructions into a prompt to override the LLM's system guardrails and safety directives. This can cause the model to bypass safety alignment, generate restricted content, execute unauthorized function calls, or exfiltrate session data to external endpoints.
Frequently Asked Questions
Q:What is the difference between direct and indirect prompt injection?
Direct prompt injection occurs when a user prompts the model to ignore rules. Indirect prompt injection happens when a model reads untrusted external data (like a webpage) containing malicious instructions.
Q:How do you defend against prompt injection?
By separating user input from developer instructions, using strict system prompt isolation, and employing secondary filter models to inspect outputs.
Quick Facts
- CategoryModel Limitations
- Key ApplicationVulnerability analysis, LLM firewall design, and input sanitation.
Coverage Trend12 Weeks
Related AI Terms
Cite This Term
Prompt Injection Media Coverage & Intelligence
Prompt Injection Attacks Are Thwarting AI Hacking Agents
"Context bombing" tricks malicious AI agent into shutting down before they can do harm.
GPT-Red: Unlocking Self-Improvement for Robustness
Explore GPT-Red, OpenAI's automated red teaming system that uses self-play to improve AI safety, alignment, and prompt injection robustness.
Now, Defenders Are Embracing the Prompt Injection, Too
"Context bombing" tricks hacking agents into shutting down before they can do harm.