Can LLMs Perform Deep Technical Understanding of Computer Architecture Papers

This study investigated whether LLMs can achieve deep technical understanding that goes beyond simple summarization—grasping a paper's core mechanisms and identifying hidden assumptions. To this end, the researchers proposed Gauntlet, an open-source pipeline that employs 5 independent expert personas and goes through an adversarial synthesis stage.

As a result, in an evaluation of 20 papers from ISCA 2025 and HPCA 2026, Gauntlet was preferred over human analysts in 15 cases. In particular, it outperformed humans in Critical Rigor, the ability to pinpoint logical gaps in a paper.

The key findings are as follows:

• Effectiveness of the multi-agent structure: Performance was substantially higher with the multi-agent structure than with a single-agent model, and the Synthesis stage in particular played a critical role in improving performance.
• Areas where humans excelled: Humans scored higher than LLMs in Trust and Usefulness, but this was less a matter of depth than issues such as confidently wrong answers or insufficient explanations.
• Limitations: LLMs were somewhat weaker than humans in terms of Calibration.