🖐️ Grasp

What "eyes and hands" means. Eyes — Grasp captures the screen, resizes it to a vision-friendly resolution and draws a coordinate grid on top, so Claude can read exact positions instead of guessing. Hands — Grasp injects real, DPI-correct mouse and keyboard input at the OS level (a Rust backend), and runs shell commands directly. Together they close the loop: Claude looks, decides, acts, looks again.

Why Grasp

Most "computer use" setups are either a cloud VM you don't control, or a browser-only automation that can't touch the rest of your machine. Grasp runs locally, drives the real desktop, and plugs into the tools you already use through a single mcp add command.

• Pixel-perfect on scaled displays. The Rust input backend respects Windows display scaling (125% / 150% / 200%) — clicks land exactly where Claude looked, not two centimetres off.
• Vision tuned for accuracy. Screenshots are cropped/resized to ~1280 px with a 100-px coordinate grid overlay, the single biggest lever on click precision.
• Shell without the screenshot tax. run_command returns real text output — no opening a terminal, typing into it, and screenshotting the result.
• Safe by construction. Input is released on startup/shutdown so a stuck modifier can never freeze your keyboard; failing and elevated commands are reported honestly, never as silent successes.

Tools

How the coordinate model works

1. Claude calls screenshot. Grasp captures the screen, resizes the longest edge to 1280 px, and overlays a grid labelled every 100 px.
2. Claude reads a coordinate straight off the grid (e.g. "the button is near x≈540, y≈300").
3. Claude calls click with those numbers. Grasp maps them back through the resize + crop + DPI transform to the exact physical pixel and clicks there.

You never do the math — pass coordinates in the same space you see them. (Advanced: pass coord_space: "screen" to use raw physical pixels instead.)

Requirements

• Windows 10/11, x64. The mouse/keyboard backend ships as a prebuilt native binary for win32-x64. Screen capture, window enumeration and image processing are cross-platform, but Grasp is Windows-first today.
• Node.js ≥ 18.
• An MCP host — Claude Code, Claude Desktop, or anything that speaks MCP over stdio.

Install

git clone https://github.com/Jamshed7470/grasp-mcp.git
cd grasp-mcp
npm install
npm run doctor   # verify Grasp can see and control this machine

npm run doctor should end with ✅ Grasp is ready — Claude has eyes and hands.

Add to Claude Code

From the repo directory:

claude mcp add grasp -- node "%CD%\src\index.js"

…or with an absolute path from anywhere:

claude mcp add grasp -- node "C:\path\to\grasp-mcp\src\index.js"

Then just ask Claude, e.g. "take a screenshot and open Settings for me."

Add to Claude Desktop

Edit claude_desktop_config.json(%APPDATA%\Claude\claude_desktop_config.json) and add:

{
  "mcpServers": {
    "grasp": {
      "command": "node",
      "args": ["C:\\path\\to\\grasp-mcp\\src\\index.js"]
    }
  }
}

Restart Claude Desktop. Grasp's tools appear under the 🔨 menu.

Usage examples

Once installed, drive it in plain language — Claude picks the tools:

• "Open Notepad, type today's date, and save it to the Desktop."
• "Find the Wi-Fi icon in the tray and tell me which network I'm on." (screenshot → read)
• "Update all my winget packages." (run_command with elevated: true)
• "Scroll the page down and click the first search result."

Configuration

Per-tool options (image size, grid on/off, JPEG quality, command timeout, working directory, elevation) are passed as tool arguments — see each tool's description in the MCP schema.

⚠️ Security

Grasp gives an AI model real control of your computer — the same reach you have. Treat it accordingly.

• You are always in the loop. MCP hosts ask before each tool call by default. Keep it that way for anything destructive.
• Close sensitive windows (password managers, private messages) before running an agent — whatever is on screen is what Claude sees, and screenshots may briefly hold that content.
• Elevated commands need your hand. elevated: true triggers a Windows UAC prompt you must approve; UAC runs on a secure desktop that cannot be automated. If you dismiss it, Grasp reports the command as not run — never a silent success.
• Never commit screenshots. This repo's .gitignore blocks *.png/*.jpg for exactly this reason.
• Run Grasp only on machines and tasks you're comfortable handing to an assistant.

Troubleshooting

Run npm run doctor any time to re-check every backend.

How it's built

src/
  index.js            MCP server — registers the 13 tools over stdio
  native.js           screen capture + mouse/keyboard (node-enigo, node-screenshots, get-windows)
  image-processor.js  sharp pipeline: crop → resize → coordinate-grid overlay
  shell.js            PowerShell runner with UTF-8 output + UAC elevation
  doctor.js           standalone backend health check
native/
  node-enigo-win32-x64.node   prebuilt Rust input backend
test/
  mcp-test.js         end-to-end test over real MCP stdio (read-only + safe tools)
  live-notepad.js     opt-in live test that types into a throwaway file and reads it back

Backends: node-enigo (Rust, mouse/keyboard) · node-screenshots (Rust, capture) · get-windows · sharp · @modelcontextprotocol/sdk.

Development

npm run doctor          # health check, no server
node test/mcp-test.js   # 11 end-to-end assertions over MCP stdio
node test/live-notepad.js   # opt-in: drives the real desktop, cleans up after itself

Contributions welcome — especially a macOS/Linux input backend to make Grasp truly cross-platform.

License

MIT. Grasp bundles a prebuilt node-enigo binary and depends on node-screenshots, both built on the Rust enigo / xcap ecosystem.