AI Briefingpi-computer-use, GUI Automation Agent Extension Released
Key point
pi-computer-use, an MIT open-source extension that lets AI agents directly operate desktop apps without APIs, has been released.
Details
pi-computer-use is an extension for the AI agent runtime Pi that enables agents to automate GUI-only desktop apps on macOS and Windows. It explicitly states automating legacy software without APIs or MCP servers as its main use case.
The action loop consists of 5 steps:
find_roots— discover open windows and browser pagesobserve_ui— obtain UI outline and immutablestateIdsearch_ui/expand_ui/inspect_ui— explore stateact_ui— perform actions such as click, type, and scroll
The core design is reference-based on immutable state (stateId), which prevents errors from clicking on stale coordinates after the screen has changed. Actions are handled as transactions, and post-verification (verified / preexisting / failed) is possible via expect conditions.
On macOS, it works via a helper app (macOS 14 or later) that uses accessibility and screen recording permissions; on Windows, it operates through the platform's accessibility API. Browsers are controlled the same way via CDP-based tools (launch_browser, navigate_browser, evaluate_browser).
Installation: pi install npm:@injaneity/pi-computer-use / MIT License.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Whiteboard Launches Open-Source VS Code Fork for Human-AI Collaborative Software Design
OpenComputer Hosts 'Ship Video' Agent Using Claude Opus 5.5 to Generate Explainer Videos from Code