AI Model Security Robustness Leaderboard Released

A new leaderboard has been developed to evaluate not only model performance but also Security. It was created to help manage the risks of adversarial attacks and jailbreaks that arise when deploying AI agents.

Key features are as follows:

• Automated test suite: Models are tested using 1,500 automatically generated jailbreak attempts.
• Metric: It measures the number of Universal Jailbreaks—cases where the model, within a specific domain (such as cybersecurity), produces detailed and compliant answers to harmful questions with a probability of 75% or higher.
• Results: According to the technical report, a significant security gap was confirmed between the most robust model and the most vulnerable model.

Future plans under consideration include improving the methodology for comparing open-weight models, expanding into domains such as agent hijacking, and introducing stronger adversarial attack techniques such as boundary point jailbreaking.