Criticism of Senior SWE-Bench's LOC Thresholds, LLM Grading Variance, and napkin-math's SSD Performance Figures

A critical analysis was presented regarding the data accuracy and evaluation methods of the AI benchmark Senior SWE-Bench and the hardware performance calculation tool napkin-math. The author points out that Senior SWE-Bench's LOC (Lines of Code) based evaluation distorts continuous code quality into binary classification due to arbitrary thresholds. For example, there is a discontinuous cutoff where 121 LOC passes but 122 LOC fails when the reference code is 61 LOC. Additionally, the author highlights the high variance in LLM-based grading, revealing that the probability of obtaining a score different from the official result when repeating grading under identical conditions reaches approximately 20%. This could lead to inconsistent results in the evaluation of major models such as Claude Fable 5, Opus 4.8, and GPT-5.6 Sol.

Regarding the napkin-math tool, it is criticized for presenting random SSD read performance as 100 us / 70 MB/s, which is deemed inappropriate. The author argues that simplified figures are not useful, citing that high-speed SSDs like the Kioxia CD9P-R exhibit a latency of approximately 30 us and show significant variance depending on queue depth and the number of jobs. In particular, the 8 GiB/s measurement for Google Cloud's c4-standard-48-lssd is excessively high compared to the official documentation's maximum of 5000 MiB/s, suggesting the possibility of errors or cache reads during the measurement process. These numerical issues and limitations in evaluation methods undermine the reliability of benchmark headline scores.