ChatGPT Voluntarily Generates Sexual Violence and Hardcore Snuff Images

Mindgard's red-team research revealed that ChatGPT's image generation filters can be completely disabled under certain circumstances, allowing the generation of images depicting sexual violence and gruesome violence.

Key findings are as follows:

• Limitations of input filters: When users use vague prompts such as asking to 'restore an image' without using direct prohibited words, the system fails to detect inappropriate intent at the input stage.
• Output filter bypass: Certain prompt patterns can bypass safety mechanisms at the generation stage, producing snuff imagery that resembles real people or is extremely shocking.
• Dataset issues: It was pointed out that the generated images are connected to dark portions within the training dataset, such as images of actually murdered women.

This case presents a significant challenge regarding how AI models' safety guardrails should fundamentally control the generation of harmful content within the latent space, going beyond simple keyword blocking.