Planet-Sized Brains: Are LLMs Overthinking?

An analysis of the relationship between LLM reasoning effort and model performance in security vulnerability triage tasks found that higher reasoning ability does not always lead to the best results.

Testing various model combinations including GPT-5.4 and Claude-4.7 confirmed that a voting committee approach composed of four LLMs was effective, showing a very high agreement rate (86.2%). It was also found that higher reasoning effort tended to increase the occurrence of content filtering.

Key experimental findings are as follows:

• Reasoning effort: The low setting produced the lowest performance across all models, but there were cases—such as gpt-5.5-med—where the med setting outperformed high.
• Success rate: Most LLMs succeeded in finding part of the bug (70.8%), but the full solve success rate—fully explaining the entire attack chain—was only 1.9%.
• Scope: Models performed far better when given code at the function level rather than the entire file.
• Model characteristics: Among the models, only the Claude family directly referenced CVEs during analysis.