AI Safety Benchmark DystopiaBench Released

DystopiaBench is a benchmark that evaluates LLM safety through 36 step-by-step scenarios based on 6 dystopia types (Petrov, Orwell, Huxley, Basaglia, LaGuardia, Baudrillard).

This benchmark measures whether models can go beyond simply refusing obviously dangerous requests, and can recognize and refuse dangerous requests that are cleverly hidden through dual-use or normalization techniques (e.g., building a social credit system).

Key updates and features are as follows:

• 42 models tested: Includes both open-source and closed models.
• LLM-as-a-judge: Uses 3 LLMs as judges to calculate scores.
• Improved precision: Uses the average of 3 runs as the final score.
• Expanded modules: Added 4 new modules and scenarios in addition to the existing Petrov and Orwell.

This project has been released as open source, allowing anyone to contribute to or use it via GitHub.