Vercel Labs releases agent-desktop CLI for AI agents to control desktop apps via accessibility trees

• Vercel Labs released agent-desktop, a CLI and C library that enables AI agents to manipulate desktop applications on macOS, Windows, and Linux by reading OS accessibility trees rather than using screenshots or pixel matching.
• The system assigns unique refs to UI elements within a JSON snapshot; agents interact via these refs in a loop, with the tool rejecting stale refs or ambiguous targets when the UI changes.
• It operates headlessly by default using accessibility APIs without moving the pointer, but supports a --headed flag for physical input simulation.
• The project includes 60 commands (56 operational), MCP support via stdio, and a C library (libagent_desktop_ffi) for in-process integration, licensed under Apache-2.0.

Platform Requirements

• macOS: Requires macOS 13+ (Intel/Apple Silicon) using the AX API.
• Windows: Requires Windows 10 1809+ (x64/ARM64) using UI Automation.
• Linux: Requires Ubuntu 24.04+ (x86_64/aarch64) using AT-SPI on X11 or Wayland.

Integration and Security

• Agents can retrieve usage guides via agent-desktop skills get desktop and integrate through MCP tools like desktop_snapshot and desktop_click.
• Security measures include piping text via --stdin for sensitive commands to prevent process argument exposure and warnings against logging snapshots or clipboard data due to potential PII.