typesafe-computer-use: macOS automation for $0.0002 per step

Frontier-model computer use sends a screenshot and waits seconds for a plan at every step, which makes a twelve-step task cost dollars. typesafe-computer-use, open-sourced in mid-September 2026 (MIT, Python, 1.2k stars as of 2026-10-12), takes “which action” away from the expensive model: you type a plain-English goal, it reads the screen, and a small classifier picks the next step — the writing model is called only when a text field genuinely needs free text.

Core features

• Five stages per step: OCR a crop of the frontmost window, then add the accessibility tree’s labels for the icons OCR cannot see; code fills in the facts — dates and their distances, rows with duplicate labels, the focused field, app and URL, actions already tried on this screen; one TypeSafe request then answers three Choice questions: which kind of action, which item, which site; the action executes deterministically; when the classifier stops, the writer reads the screen and answers or hands control back — it never picks an action.
• What a Choice can hold: up to 255 options with a full probability distribution and a calibrated confidence, answered in a few hundred milliseconds with free output tokens — the precondition for handing “which one” to a small model.
• Self-measured cost and latency: same screenshot, one decision each — $0.0002 per decision against $0.032 for Claude Opus 5 reading a bare screenshot; model latency 0.13–0.38 s against 5.2 s; end-to-end step of about 1.5 s against about 5.5 s; a 12-step task for $0.003 against $0.40–$0.90.
• Reasoning moved into code: dates, whether a field is focused, whether a URL is clean — all computed in code and handed over as state. The README is equally clear about the price: every piece of reasoning a frontier model does for free has to be rebuilt here as deterministic state.
• Dry run is a first-class citizen: uv run clicker "open the Playground" runs one step and prints what it would do; add --act to actually drive the machine, up to 100 steps; it stops on done, low confidence, a stall, or the step limit, and Ctrl-C or slamming the mouse into the top-left corner both halt it.
• A browser backend: switch to DOM perception over Chrome DevTools — no OCR, no screen permission needed.
• Every step is inspectable: runs land in runs/<timestamp>/; uv run clicker-inspect "any goal" opens the annotated screenshot and full payload, and failed runs can be replayed offline.

Typical use cases

• Individuals who want cheap desktop automation: dry-run first to see what it plans to click, then --act; the README’s own example is “go to techcrunch and take me to the checkout page for the cheapest tickets to their next upcoming event”.
• Date-driven tasks like ticket drops: date distances are computed in code and fed to the classifier instead of being read off pixels.
• Agent researchers: scripts/osworld runs single tasks on OSWorld beside its own GPT agent on the same task (results in docs/osworld.md; the README lists no numbers).

Quick start

git clone https://github.com/awlevin/typesafe-computer-use
cd typesafe-computer-use
uv sync
cp .env.example .env     # fill in the keys
uv run clicker "open the Playground"                 # dry run: one step, prints what it would do

Prerequisites: macOS 14 or newer, Python 3.12 or newer, and uv; grant the terminal Screen Recording and Accessibility in System Settings — without the first, captures are wallpaper, and without the second, synthetic clicks are silently dropped and --act refuses to start. TYPESAFE_API_KEY is required for every decision; ANTHROPIC_API_KEY is optional, for the writer’s free text.

Summary

typesafe-computer-use fits developers who want the “which action” step to become cheap, and anyone researching the cost structure of computer use. Know the boundaries: the main path is macOS 14+ only, with Windows 10/11 experimental; the README labels the project Beta and warns that settings and behavior may change between 0.x releases; it drives your real mouse and keyboard, so start with a dry run; and the fuller known limits live in docs/known-limits.md. Take the official comparison with a discount, too — the Opus 5 arm read the pixels unaided while this side has code doing the groundwork, so it is not a matched fight. MIT-licensed, written in Python. Related: five routes to agent browser control and the ten Jev use cases roundup.