agent-desktop
Computer use CLI that reads a desktop app's accessibility tree as JSON
About agent-desktop
agent-desktop reads the accessibility tree of a desktop application and returns it as JSON, giving every element an agent can act on a stable ref. The calling agent maps an app with a shallow skeleton snapshot, drills into the one region it needs, acts on a ref, then reads the UI again to confirm something actually changed. That last step is the point: a successful click does not prove the interface moved, so the loop ends with a read rather than an assumption. It uses the AX API on macOS, UI Automation on Windows and AT-SPI on Linux, behind the same 60 command names on all three platforms, and ships both as a CLI installed from npm and as a C library you can link in process.
The documentation is unusually honest, and the entry page opens with a section called "What agent-desktop is and is not". It states plainly that this is a tool agents call rather than an agent itself, that it never embeds a model and has no GUI or interactive prompt, that it does not automate web pages, and that it is limited to what applications expose - it reads accessibility trees and does not match pixels, so it cannot see a UI that publishes no tree. That last limit is the real one in practice and rules out canvas-heavy software, games and poorly built Electron apps. It is also a Vercel Labs project, which is an explicitly experimental shelf, and on macOS it needs Accessibility permission, which is a genuine security decision for a tool an LLM drives.
You install the CLI globally from npm and run the status command, which reports the platform, which permissions are granted and which surfaces are supported. The agent then loads the bundled skill, which is the guide it should read before acting. From there the loop is four commands. A skeleton snapshot maps an application three levels deep and marks which branches have children, which keeps the first read small. A second snapshot rooted at one ref expands just that region. A click, type or scroll command acts on a ref. A wait command with a predicate verifies the result, for example that a display element now reads the expected value. Every command prints exactly one JSON envelope and exits, so it composes into scripts as easily as into an agent loop. Refs carry the id of the snapshot that produced them, so a stale ref is detectable rather than silently wrong. For a Chromium or Electron app you launched, agent-desktop opens a DevTools port that the companion agent-browser connects to, covering the web contents it deliberately does not handle itself.
- •Accessibility tree as JSON - Every actionable element gets a ref, so an agent addresses real UI elements rather than guessing at screen coordinates
- •Skeleton and drill-down - A shallow map first, then expand one region, so a complex application does not have to be read in full
- •Three platforms, one command surface - The AX API on macOS, UI Automation on Windows and AT-SPI on Linux behind the same 60 command names
- •CLI and C library - Use it from a shell or a script, or link agent-desktop-core in process; the core never imports a platform crate
- •Observe, act, verify - A wait command with predicates closes the loop, because a successful action is not proof the UI changed
- •Honest documented limits - No model inside, desktop apps only, and nothing it cannot see through an accessibility tree
Anyone building an agent that has to drive software with no API: an internal desktop application, a legacy tool, an installer, a native client that never got a web version. It fills a gap rather than competing for one, since browser automation already has good options and desktop automation has had very few. Developers of agent frameworks and automation scripts will get more from it than end users, because there is no product surface here at all. Be clear about what you are adopting. It is infrastructure rather than an AI tool in the usual sense, it is on Vercel's experimental shelf, and its reach stops exactly where an application's accessibility support stops. If your target is canvas-heavy, a game, or an Electron app with poor accessibility, this will not see it, and the documentation says so before you install.
Pricing
No pricing page found
Free and Apache-2.0, installed with npm install -g agent-desktop. There is no pricing page and no paid tier; the licence is read from the project's own GitHub repository.
Checked 2026-11-06
