windows-control

windows-control

An MCP server that lets an AI actually operate Windows apps — select a window, readwhat's on screen, click things by name, type, scroll, drag — plus a WhatsApp layerthat understands the app's UI (unread chats, contacts, conversations) instead ofreturning a wall of OCR text.

Three rules shape the design:

• No injection. Nothing is loaded into the target process and no automation oraccessibility provider is queried. The app is never touched from the inside.
• Windows OCR for sight. Windows.Media.Ocr via WinRT — offline, built intoWindows, no third-party service.
• Background by default (WhatsApp). The app is captured with PrintWindow andclicked with posted messages, so it can sit fully covered behind whatever you'redoing. Your cursor never moves and your focus is never stolen.

Setup

py -m pip install -r requirements.txt
py test_control.py             # generic: opens Notepad, types, OCRs, clicks a menu
py test_whatsapp.py            # WhatsApp: pixel heuristics + a live read-only run
py test_whatsapp.py --offline  # no WhatsApp needed

Copy .mcp.json.example to .mcp.json and set cwd to wherever you clonedthis, and Claude Code picks the server up in that folder. For Claude Desktop,add the same block to claude_desktop_config.json:

{
  "mcpServers": {
    "windows-control": {
      "command": "py",
      "args": ["-m", "windows_control.server"],
      "cwd": "C:\\path\\to\\windows-control-mcp"
    }
  }
}

Foreground vs background

Two ways to drive an app. Both are real OS-level input; neither injects.

Everything works in the background, including sending. Reading, openingchats, scrolling, searching and sending a message all leave your focus, cursorand window order untouched — verified by watching the foreground and z-ordercontinuously through a real send.

Typing needs one specific detail: post WM_CHAR alone. Wrapping it in theWM_KEYDOWN/WM_KEYUP pair looks more faithful to a real keyboard, butChromium derives its own character from the key event and processes yours, soevery letter arrives two or three times (abcd → aabbccdd). A bareWM_KEYDOWN alone does nothing.

If posted typing is ever refused, the field is cleared and the typing is redoneby briefly borrowing the foreground, then handing it straight back — so itdegrades instead of half-typing. Set WHATSAPP_NEVER_FOCUS=1 to forbid eventhat, making it impossible for anything to bring WhatsApp forward.

Sending safely

whatsapp_send_message reads the text back out of the composer before pressingEnter, so a half-typed or mistyped message is never sent, and it refusesoutright if the open chat isn't the one you asked for. Afterwards it confirmsfrom the screen: sent (the box returned to its placeholder) andconfirmed_in_chat (the message is visible in the conversation).

A covered Chromium window throttles repainting, so what you type can take severalseconds to appear in a capture — verification polls for up to 20s rather thanjudging on the first look. Expect a send to take ~25s.

The window must not be minimised for background capture — it can be behindanything, but a minimised window has nothing to render.

WhatsApp tools

These return structured data, not raw OCR dumps.

Detecting unread messages

Unread is decided by a pixel test — the green badge disc on the row — not byOCR, so it works even when the number inside the badge can't be read. Threescan modes trade speed for completeness, and all three stay in the background:

"filter" changes UI state, so it remembers which pill was selected (detected byits solid fill, since OCR drops a word as short as "All"), clicks it back, andverifies the restore actually took, retrying if not. If it still can't, theresult says so rather than leaving your list stuck on a filter you didn't pick.

whatsapp_watch_unread(timeout_seconds, poll_seconds) takes a baseline first, sochats that are already unread don't trigger it — only new arrivals do. It pollsin the background, so you can keep working while it waits.

whatsapp_unread()                          # fast look: who's waiting
whatsapp_unread(scan="filter")             # definitive: every unread chat
whatsapp_watch_unread(timeout_seconds=600) # tell me when someone messages

Reading chat content

whatsapp_read_chat returns the conversation as structured messages — one perbubble, so a long multi-line message stays whole instead of shattering into aline-per-entry:

{ "text": "Running late\nStarting in 20 minutes\nSee you there",
  "sender": "them", "sender_name": "Alex Rivera",
  "sender_phone": "+1 555 0100", "time": "1:42am", "kind": "message" }

• sender — you / them from the bubble colour, or system
• sender_name — who wrote it in a group chat (WhatsApp draws it in colour)
• kind — message, or system for join notices and privacy changes
• transcript — the whole thing rendered ready to quote

whatsapp_read_chat()                          # the open chat
whatsapp_read_chat(chat="Mom")                # open it first, then read
whatsapp_read_chat(scroll_back=3)             # include older history

chat= finds the conversation by scrolling the list, so it stays background.scroll_back=N walks up N screens and stitches them into one chronologicaltranscript, then returns the view to the newest messages.

Bubbles are found by colour continuity, not by gaps between lines — thebubble's background runs unbroken through the blank lines inside a long message,whereas the wallpaper shows through between two separate messages. That onedistinction is what keeps a multi-line message intact and stops two consecutivemessages merging.

Downloading attachments

whatsapp_list_media()            # what's attached in this chat
whatsapp_download_media()        # save the newest one           [background]
whatsapp_download_media(index=0, folder="C:\\keep")

Photos and videos contain no text, so OCR cannot see them at all — they'refound as blocks of high colour variance, which is what separates a photographfrom a flat message bubble and from the faint wallpaper. Documents are found bytheir filename, which is text. A video is told apart from a photo by theduration burnt into its thumbnail — matched as a bare 0:42, since a bubbletimestamp looks nearly identical but carries am/pm.

Downloading right-clicks the attachment, picks WhatsApp's own Save as, andfills in the Windows save dialog — all without focusing or raising WhatsApp.Files go to ~/Downloads/whatsapp unless you pass folder or setWHATSAPP_DOWNLOAD_DIR.

Two traps the implementation handles, both of which silently produce no file:

• WM_SETTEXT on the filename box does nothing useful. It changes the textyou can see but not the model the picker reads, so it commits the originalname. The field has to be edited the way an edit control expects(EM_SETSEL + EM_REPLACESEL).
• The name you ask for is not the name you get. The picker enforces thesource file's extension, so photo.png on a JPEG is written asphoto.png.jpeg. The saved file is located by diffing the directory, and thereturned path is authoritative — never assume the name you requested.

Only attachments on screen can be downloaded; scroll to older ones first.

How it understands the UI

No hardcoded coordinates — every cue is derived from the pixels, so it surviveswindow resizing and theme changes:

• Panels — the nav rail, chat list and conversation pane have differentbackground colours, so the dividers are found by scanning for column colourtransitions (with a proportional fallback if the read looks implausible).
• Chat name vs preview — the name is the highest-contrast text in a row,measured against that row's own background so the highlighted (open) rowstill parses correctly. Works in light and dark themes.
• Unread — the green badge is found as a solid, round, right-aligned colourblob (#21C063/#25D366), which is why a green timestamp or a green group avatardoesn't fool it.
• Not-a-chat rows — the end-to-end-encryption notice and similar banners aredropped, and icons that OCR reads as a stray one-character word (the mute bellbecoming "Family O") are stripped using the gap between word boxes.
• Sender names — drawn in colour, so they're found by saturation rather thanby position. Links are coloured too, so a name only counts when plain textfollows it — otherwise a green URL would be read as the sender.
• Avatars — a group avatar's initial OCRs as a one-letter line in the gutterleft of the bubbles; it's dropped rather than reported as a message.
• You vs them — sent bubbles are green; the bubble background colour besideeach message decides, falling back to alignment.

Example

whatsapp_overview()                       # who's messaged, what's unread   [background]
whatsapp_open_chat("Mom")                 # scrolls the list, clicks it     [background]
whatsapp_read_chat()                      # transcript, tagged you/them     [background]
whatsapp_send_message("On my way 🙂")     # verified, then sent    [background]

Generic tools

Choosing an app: list_windows, focus_window, active_window,window_action (maximize/minimize/restore/close), launch_app

Seeing: read_screen, find_text, wait_for_text, screenshot —each takes background=True with a window

Acting: click_text, click, type_text, press_keys, move_mouse,drag, scroll, cursor_position — click, click_text and scroll takebackground=True

Coordinates are absolute physical screen pixels across all monitors (the processis per-monitor DPI-aware), so a center_x/center_y from find_text goesstraight into click.

Known limits

• OCR sees text, not icons. Unlabelled icon buttons have nothing to match —screenshot and click coordinates for those.
• Per-chat unread counts are best-effort. Windows OCR is a line recogniserand usually refuses a lone digit, so a chat may come back has_unread: truewith unread: null. It never guesses a number. Which chats are unread isreliable (a pixel test), and how many chats are unread comes from the"Unread N" pill, which is ordinary text and reads reliably.
• Media has no text to read. Images, voice notes and stickers come back astheir caption or not at all — OCR reads pixels of text, not pictures. Usewhatsapp_download_media to get the file itself.
• OCR costs ~1–4s per call, dominated by recognition, not by startingPowerShell: a long-lived OCR host process is reused across calls (starting oneper call added seconds every time). Each read parses a single screenshot, soprefer whatsapp_overview / whatsapp_read_chat over repeated find_text.
• Opening a chat marks it read, exactly as it would if you clicked it.
• Background input is app-dependent. It works on Chromium/WebView2 apps;native Win32 apps may ignore posted clicks. Verify with a backgroundscreenshot, or fall back to background=False.
• Foreground mode shares your mouse. Don't type while it works.
• WHATSAPP_NEVER_FOCUS=1 forbids ever raising WhatsApp (typing then errors).WHATSAPP_READONLY=1 disables sending.WINDOWS_CONTROL_BLOCKLIST=banking,keepass refuses control of matching windows.WINDOWS_CONTROL_OCR_LANG=en-US pins the OCR language.

Layout

windows_control/
  server.py     MCP tool definitions (25 tools)
  whatsapp.py   WhatsApp UI understanding: panels, rows, badges, messages
  bginput.py    background capture (PrintWindow) + posted clicks/scroll/typing
  filedialog.py drives the Windows save picker without focusing it
  wininput.py   foreground SendInput mouse + Unicode keyboard
  winapi.py     window enumeration, verified focus, focus borrowing, DPI
  screen.py     capture, OCR (persistent host), text -> coordinate matching
  ocr.ps1       Windows.Media.Ocr via WinRT (one-shot or -Server mode)
test_control.py    generic end-to-end test
test_whatsapp.py   WhatsApp tests (offline heuristics + live background run)