agent-desktop: Vercel opens desktop computer use via the accessibility tree

The usual way AI agents drive desktop apps — screenshots plus pixel matching — is slow, brittle and blind to text. agent-desktop, recommended by a Chinese WeChat roundup on November 4, 2026 (“多次霸榜 GitHub”), is Vercel Labs’ alternative: it reads the OS accessibility tree and returns the app’s structure as JSON, with a stable ref on every interactive element; the agent clicks or types on a ref, reads the UI again to verify, and repeats. 13,752 stars as of November 4 (Apache-2.0), macOS, Windows and Linux, one-command install via npm or Homebrew, shipped as both a CLI and a C library — and no model inside: the loop lives in your agent. Our agent-browser-control research piece covered the browser side of this problem; agent-desktop extends it to the whole desktop.

Core features

• Accessibility-tree snapshots: snapshot returns the whole interactive tree, every element carrying a stable ref like @s8f3k2p9:e8 — cheaper than screenshots and far steadier than coordinates.
• Ref-level actions: click and typing commands act directly on refs, then a fresh snapshot verifies the result — the read-act-verify loop belongs to your agent.
• No model inside: agent-desktop only sees and acts; Claude Code, Codex or any agent supplies the judgment.
• Permission guidance: permissions --guide walks through the macOS Accessibility grant step by step; status shows platform coverage at a glance.
• CLI plus C library: scripts use the CLI (one JSON envelope per command), integrations link the C library; npm, Bun, Yarn, pnpm and Homebrew all work.

Typical use cases

• Desktop app automation: batch file cleanup, data entry and repetitive tasks inside Finder, Excel or your own clients.
• End-to-end GUI testing: the accessibility tree doubles as an assertion source — far more stable than pixel diffs.
• Accessibility audits: what the agent reads is what a screen reader reads; the tree is your audit surface.

Quick start

npm install -g agent-desktop
agent-desktop status                # platform, permissions, supported surfaces
agent-desktop snapshot --app Finder -i   # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8         # act on a ref
agent-desktop snapshot --app Finder -i   # observe again to verify

macOS needs the Accessibility permission (permissions --guide walks you through it); Linux and Windows need no grant.

Summary

agent-desktop fits developers who need agents operating desktop GUIs reliably over long sessions; game surfaces and custom-drawn UI that expose no accessibility tree remain screenshot territory — it complements visual computer use rather than replacing it, covering the large majority of apps that do have a tree. Facts and limits: Apache-2.0, Rust, three platforms; the README warns that snapshots, clipboard reads and screenshots can contain private desktop content, so keep them out of public logs; element refs change when an app redraws, so agents should re-read rather than memorize. Together with browser-side agent-browser-control, the “agent drives a graphical interface” picture is complete: the web goes through browser automation, the desktop through the accessibility tree.