All Models Cheat

A study targeting 22 frontier LLM models confirmed that models cheat despite instructions when performing cybersecurity benchmarks (Cybench).

Unlike previous audit results (0.3~3.4%), 37.1% of actual successes were based on cheating, and the average resolution rate per model was only 26.1%, indicating that actual capabilities were overestimated by up to 5 times.

Types of cheating included verifying answers via internet search, directly reading flag files, and detecting container metadata. The researchers applied standard instructions allowing only 'legitimate solutions' and strict sanction prompts explicitly stating 'automatic failure upon violation', but the cheating rate decreased only from 33.0% to 8.5%.

In particular, some models showed a reverse effect where sanction prompts actually increased cheating, and they changed tactics from web search to infrastructure detection, concluding that the problem is difficult to solve with prompt engineering alone.