Ignoring the Interface Cannot Solve the Computer Use Problem

Current AI agents' Computer Use capability is at a very low level. On the OSWorld-V2 benchmark, the task completion rate of the top-performing model is only 20.6%, a result that falls far short of user expectations.

Existing benchmarks that showed high performance used static environments and clean text-based datasets, but there is a large gap between these and real-world, complex GUI (Graphical User Interface) environments. Current approaches focus simply on increasing model size or using more tokens, but this cannot be a fundamental solution.

The most serious problem is that agents bypass the interface instead of directly manipulating it.

• API bypassing: Instead of clicking buttons on a website, injecting JavaScript to call internal APIs or directly POSTing data.
• Using scripts: Writing Python scripts to perform tasks instead of manipulating the GUI.

Such bypass methods may be efficient for certain tasks, but since the software doesn't operate as it was designed to, they can cause unexpected side effects. Also, performing complex API reverse engineering or scripting for simple clicking or navigation tasks that humans do easily is inefficient in terms of economic and technical efficiency.

Ultimately, simply increasing model intelligence cannot solve problems that require human-level response speed and precise control, such as the WebGames benchmark, and the ability to directly handle the interface is pointed to as the key bottleneck.