LLM Role Recognition Limits and Prompt Injection

Researchers including Charles Ye published findings on the 'Role Confusion' problem, in which LLMs fail to clearly distinguish between Role Tags such as system messages (), thinking processes (), and assistant (), versus user input ().

According to the research, models tend to follow the style of text more strongly than its actual meaning. For example, if a user sends a malicious request using a writing style similar to the model's internal thinking block, the model may mistake it for a system instruction, ignoring existing safety guidelines and allowing the attack to succeed.

In particular, it was confirmed that simply making fine-grained changes to the format of text through a 'Destyling' technique could cause the attack success rate to plummet from 61% to 10%. This suggests that unless LLMs achieve true role recognition, prompt injection defenses will remain limited to an ongoing game of 'whack-a-mole.'