claude-workman — computer use for Linux, as an MCP server

License: MIT Platform: Linux/X11

claude-workman is an open-source MCP server that gives an AIassistant human-mode control of a Linux desktop — it takes a screenshot, reads theaccessibility tree, then moves the mouse, clicks, types, and switches windows.It works with Claude, Claude Code, and any other MCP client.

Named after the Walkman: a small, portable thing that just plays. Point it at a displayand it works.

pip install claude-workman
claude mcp add --scope user workman -- python -m workman.server

Why accessibility-tree clicking beats pixel clicking

Most desktop automation guesses at pixel coordinates and breaks the moment a layout shifts,a font renders differently, or a window moves. claude-workman fuses two signals on everyturn — a screenshot and the AT-SPI2accessibility tree — so an action can be addressed semantically:

"click the push button named Continue"

…and land on that button's center, wherever it moved to. This is the same principle behindmacOS AXUIElement-based computer use, ported to Linux via AT-SPI2 (gi.repository.Atspi).

Approach	Addressing	Survives layout change	Used here
Pixel-only automation (raw xdotool, image matching)	guessed `x, y`	❌ brittle	fallback only
Accessibility-tree automation	role + name → element	✅ robust	default

Tools

Tool	What it does
`screenshot` / `screenshot_region`	Capture the display (or a sub-rectangle) as PNG; `max_dim` downscales for faster round-trips
`accessibility_tree`	Actionable elements as `{app, role, name, x, y, w, h}` in screen coordinates
`click_element`	Find an element by role + name and click its center — no pixel guessing
`enable_accessibility`	Turn on AT-SPI tree export (and silence the screen reader — see safety)
`list_windows`	Windows with id, name, pid, geometry
`focus_window`	Raise a window by id or name; minimizes the frontmost blocker first
`click` / `move` / `drag` / `scroll`	Pointer actions at pixel coordinates
`type_text` / `press_key`	Keyboard input — `press_key` uses xdotool syntax (`Return`, `ctrl+c`, `super+l`, `KP_0`)
`show_cursor`	Visual click cursor overlay: a ring + click ripple showing exactly where the agent is acting

Watch what it clicks

show_cursor starts a transparent, click-through overlay that draws a cyan ring at the actionpoint and an expanding ripple on every click, with a label. It exists for human oversight —you can watch an agent work and interrupt it, instead of guessing what it just did.

Requirements

Linux with X11, plus:

sudo apt install xdotool ffmpeg gir1.2-atspi-2.0 python3-gi
# at-spi2-core is normally already running on GNOME/KDE

ffmpeg — screen capture (x11grab)
xdotool — mouse and keyboard input
python3-gi + gir1.2-atspi-2.0 — the accessibility tree (import is gi.repository.Atspi, not pyatspi)
Pillow (optional) — only for max_dim screenshot downscaling
GTK 3 (optional) — only for the show_cursor overlay

Install

pip install claude-workman          # from PyPI
# or from source:
git clone https://github.com/AdvisorAGI/claude-workman.git
cd claude-workman && pip install -e .

Usage

Run the stdio MCP server:

python -m workman.server

claude mcp add --scope user workman -- python -m workman.server

Target a specific X display with WORKMAN_DISPLAY (falls back to DISPLAY, then :0):

WORKMAN_DISPLAY=:1 python -m workman.server

The accuracy loop

screenshot — and enable_accessibility + accessibility_tree when precision matters
Locate — prefer an element by role + name; fall back to pixels
Act — click_element when possible, else click / type_text / press_key
screenshot again to verify before anything irreversible

Speech hazard (handled)

Exporting the accessibility tree from Chromium and Electron apps requires the desktopscreen-reader flag — and that same flag makes Orca read the screen aloud throughspeech-dispatcher → espeak-ng. enable_accessibility therefore stops Orca immediately afterenabling, using an exact process-name match (pkill -x orca, never pkill -f orca, whichself-matches its own shell command and aborts).

GTK applications export their trees with toolkit-accessibility alone, so the web flag isopt-in.

FAQ

What is claude-workman?

An open-source MCP server that lets an AI assistant control a Linux desktop the way a persondoes — screenshot, locate, click, type, verify — with accessibility-tree targeting so clickshit real UI elements instead of guessed coordinates.

How is this different from browser automation?

Browser tools (Playwright, Puppeteer, CDP) drive a web page. claude-workman drives the wholedesktop: native apps, terminals, file managers, settings dialogs, and browsers alike.

Does it work with Claude Code?

Yes. Register it with claude mcp add --scope user workman -- python -m workman.server. Itworks with any MCP client, not only Claude.

Does it support Wayland or macOS?

Not yet — X11 today. A Wayland backend and a macOS AXUIElement backend are the natural nextsteps, and contributions are welcome.

Is it safe to let an agent control my desktop?

Treat it like handing over mouse and keyboard. Use show_cursor so you can see every action,prefer a dedicated display or VM for unattended runs, and keep the verify-by-screenshot stepbefore anything irreversible.

Why is it called Workman?

After the Walkman — a small portable thing that just plays. Point it at a display and it works.

Does it solve CAPTCHAs?

No. It is a general desktop-control server, not an anti-bot bypass tool.

Contributing

Issues and pull requests are welcome — see CONTRIBUTING.md. Good firstareas: a Wayland backend, a macOS AXUIElement backend, richer AT-SPI role coverage, and tests.

License

_{Keywords: MCP server, Model Context Protocol, computer use, desktop automation, Linuxautomation, X11 automation, AT-SPI2 accessibility, GUI agent, AI agent desktop control,xdotool alternative, screenshot automation, Claude MCP connector, agentic computer use.}

claude-workman — computer use for Linux, as an MCP server