agent-desktop: OS accessibility tree-based desktop automation instead of pixel guessing

A tool that allows AI agents to manipulate desktop apps by reading the OS accessibility tree directly instead of relying on pixel matching. Written in Rust, this CLI and C library runs on macOS, Windows, and Linux, returning the actual UI structure of the screen in JSON format. Each element is assigned a unique ref, enabling the agent to act precisely on specific targets.

The agent uses the ref obtained from a snapshot to perform clicks or text input, then takes another snapshot to verify the result. If the UI changes, it returns a STALE_REF error to prompt a retry, ensuring more stable and retryable behavior than pixel-based guessing. On macOS, it retains native accessibility objects to prevent deleted rows from being misinterpreted as other rows with the same identifier.

It provides 56 operational commands, supporting a wide range of functions including snapshots, clicks, keyboard input, clipboard management, and app and window control. By default, it operates in headless mode via the accessibility API without moving the pointer or changing focus, but physical input can be allowed with the --headed flag if needed. All commands output a JSON envelope to clearly convey error codes and retryability.

You can install the skill into an agent harness or add instructions to AGENTS.md to use it immediately. It can also be called directly in-process via the FFI library to reduce CLI execution overhead. It provides a stable foundation for UI manipulation for developers building desktop automation or computer use agents.