What is a Prompt Injection?
Prompt Injection
Prompt Injection is a security vulnerability where a malicious user provides input that overrides the pre-configured system instructions or safety alignment filters of a Large Language Model, hijacking its control flow.
Why It Matters for AI Builders
Defines the safety alignment and security constraints of user-facing systems during vulnerability analysis, llm firewall design, and input sanitation; implementing Prompt Injection helps builders isolate instructions from injection exploits.
Detailed Deep Dive
Prompt injection is a security vulnerability where a user inserts malicious text instructions into a prompt to override the LLM's system guardrails and safety directives. This can cause the model to bypass safety alignment, generate restricted content, execute unauthorized function calls, or exfiltrate session data to external endpoints.
Frequently Asked Questions
Q:What is the difference between direct and indirect prompt injection?
Direct prompt injection occurs when a user prompts the model to ignore rules. Indirect prompt injection happens when a model reads untrusted external data (like a webpage) containing malicious instructions.
Q:How do you defend against prompt injection?
By separating user input from developer instructions, using strict system prompt isolation, and employing secondary filter models to inspect outputs.
Quick Facts
- CategoryModel Limitations
- Key ApplicationVulnerability analysis, LLM firewall design, and input sanitation.
Coverage Trend12 Weeks
Related AI Terms
Cite This Term
Reference this definition in your articles, research, or documentation to credit this source:
Prompt Injection Media Coverage & Intelligence
Suspecting Court of Using AI, Man Injected Prompts in Filings to Try to Win Case
Judge warns pro se litigants are using chatbot wrong and getting desperate.
Prompt injection doesn't care what your agent does for a living
What 1,433 winning attacks looked like when we clustered them Most teams test agent security one domain at a time. Is the customer-support bot safe? The code assistant? The expense approver? Each gets its own red-team pass, its own scenario list, its own sense of "we checked."
OpenAI details GPT-Red, an AI that attacks its own models to find flaws
OpenAI Group PBC today detailed GPT-Red, an internal artificial intelligence system it built to attack its own models and surface prompt injection vulnerabilities before they reach users. Red teaming is the job of hammering software to find its weak points, work that normally falls to human...
Now, Defenders Are Embracing the Prompt Injection, Too
"Context bombing" tricks hacking agents into shutting down before they can do harm.
'GitLost' vulnerability let GitHub's AI workflows leak private repositories
Researchers at artificial intelligence security company Noma Security Inc. today disclosed a critical prompt injection vulnerability in GitHub Inc.'s new Agentic Workflows feature that allowed an unauthenticated attacker to siphon data from private code repositories by posting a single crafted...
OpenAI unveils Lockdown Mode to protect sensitive data from prompt injection attacks
Even with Lockdown Mode, ChatGPT could be still vulnerable to prompt injection, but the goal is to reduce the likelihood that sensitive data gets shared in...