Understanding Prompt Injection: A New Security Challenge

As AI tools perform complex tasks such as web browsing, research, and purchasing, a new security threat called Prompt Injection is emerging.

Prompt injection is a type of social engineering attack targeting conversational AI. It works by having a third party, not the user, hide malicious instructions in web pages, documents, or emails to induce the AI to take unintended actions.

Key attack examples include the following:

• Real estate search: Manipulated information leads the AI to recommend a specific listing, causing the user to be recommended a property they didn't want.
• Email management: Through instructions hidden within an email, an AI agent steals sensitive information such as the user's bank statements and sends it to the attacker.

The more data AI accesses and the more autonomous tasks it performs, the greater these risks become. OpenAI is building a multi-layered defense system, including Safety training, to defend against this, and is conducting research on Instruction Hierarchy, which distinguishes between trusted and untrusted instructions.