How to Design AI Agents to Counter Prompt Injection

As AI agents perform web browsing, information retrieval, and tasks on behalf of users, new attack vectors are emerging. In particular, Prompt Injection attacks, which manipulate models using instructions embedded in external content, have emerged as a major threat.

Recent attacks are evolving beyond simple command overrides toward leveraging Social Engineering techniques. For example, an attack may embed cleverly disguised instructions in the body of an email to induce an agent to extract sensitive information or submit it to a specific system.

Existing AI Firewalling approaches, which classify inputs as malicious or not, struggle to block these more sophisticated attacks. Detecting malicious input is as complex as discerning lies or false information, and grasping context is difficult.

Therefore, the focus of security must shift beyond simply identifying malicious input, toward designing systems that can constrain impact even when manipulation succeeds.