Development of Computer Use Technology

Claude 3.5 Sonnet has introduced a 'Computer Use' feature that moves the cursor, clicks, and enters information via a virtual keyboard according to user commands. Currently offered in public beta, this technology is regarded as a significant breakthrough in that it allows AI to directly operate general software beyond existing dedicated tools.

This feature was implemented by combining multimodality, the ability to interpret screen images, with reasoning ability, the ability to determine what action to perform in a given situation. In particular, training to calculate pixel-level movement distances so that the model can click on the exact location played a key role.

Research results showed that Claude exhibited high generalization ability even when trained only on simple software such as calculators or text editors, and it was also observed to correct its own errors and retry when mistakes occurred during a task.

In terms of performance, it scored 14.9% on the OSWorld benchmark, far surpassing the previous next-best model (7.7%). However, it has not yet reached human-level performance (70-75%).

From a safety standpoint, this update maintains AI Safety Level 2. Anthropic determined that since the Computer Use feature utilizes existing capabilities rather than fundamentally enhancing cognitive ability, current safety measures are sufficient to manage it.