OpenEyes（开源点睛）

OpenEyes（开源点睛）

AI-friendly computer-use primitives for Windows / macOS / Linux.Capture. Detect. Click. Let any LLM agent drive any desktop GUI.

GitHub: https://github.com/yangpeng366/openeyes · 中文 · English · Quick start · Architecture · Roadmap · License

中文

OpenEyes（点睛） 是一个开源的 AI 友好电脑使用平台。基于「see → resolve → act」三段式原语，让任何 LLM agent 都能可靠地操控桌面 GUI 应用；提供 Vimium 风格字母 hint overlay 作为人类可调试的回退通道，并通过 MCP server 暴露给所有 agent。

为什么做这个

OpenEyes 把这些能力合成一个 MIT 开源、跨平台、AI 友好的统一平台。

特性

• AI 友好 element schema：每个 UI 元素输出结构化 JSON（bbox / center / control_type / name / automation_id / class / state）
• 易点击：UIA 命中 → 自动点中心；不命中 → vision bbox → 点中心；都不命中 → 字母 hint overlay
• 安全：默认 dry-run，所有状态改变操作可审计
• MCP 优先：所有原语通过 MCP server 暴露给任何 agent
• 可插拔 vision backend：OmniParser / Florence-2 / 自训练
• 跨平台：Windows（首发）/ macOS / Linux，统一抽象

设计文档

完整设计书：https://my.feishu.cn/docx/Ul6gdMULGo5VfzxDVAYcVoX6n9e （中文，含架构图 / 模块清单 / 路线图）

English

OpenEyes is an open-source, AI-friendly computer-use platform. Built on a "see → resolve → act" pipeline, it lets any LLM agent reliably drive any desktop GUI. Includes a Vimium-style letter-hint overlay as a human-debuggable fallback, and exposes every primitive via MCP server for any agent.

Why

OpenEyes fuses these into one MIT-licensed, cross-platform, AI-friendly platform.

Features

• AI-friendly element schema — every interactive element exposes structured JSON
• Easy click — UIA hit → bbox center; fallback vision bbox → center; final fallback letter hint
• Safe by default — dry-run; full audit trail for state-changing actions
• MCP-first — every primitive exposed as an MCP tool to any agent
• Pluggable vision backend — OmniParser / Florence-2 / custom
• Cross-platform — Windows (first), macOS, Linux, with one unified abstraction

Quick start

Windows

git clone https://github.com/yangpeng366/openeyes.git
cd openeyes
pip install -e ".[windows,mcp]"

# 1. list visible top-level windows
eyes windows list

# 2. capture a window or the full screen
eyes capture --window 123456 --out shot.png

# 3. enumerate interactive elements
eyes detect --window 123456 --pretty

# 4. click by text (resolves to bbox center)
eyes click --window 123456 --name-contains "Submit" --dry-run
eyes click --window 123456 --name-contains "Submit" --go

# 5. start MCP server (for Codex / Claude / Cursor)
eyes-mcp

First showcase — 飞书 client

See examples/feishu_first_test.py.

python examples\feishu_first_test.py --dry-run

Architecture

┌────────────────────────────────────────────────────────────────┐
│  L5  Orchestration    Codex / Claude / Cursor / 自定义 Agent     │
├────────────────────────────────────────────────────────────────┤
│  L4  AI Layer         Intent → Element (LLM 驱动 + 缓存)        │
├────────────────────────────────────────────────────────────────┤
│  L3  Resolver         selector → coord   (UIA / Vision / Hint)  │
├────────────────────────────────────────────────────────────────┤
│  L2  Perceive         capture + detect    (UIA / AX / AT-SPI)    │
├────────────────────────────────────────────────────────────────┤
│  L1  Actuate          mouse / key / drag  (Win32 / CG / XTest)  │
├────────────────────────────────────────────────────────────────┤
│  L0  Platform         Windows / macOS / Linux                   │
└────────────────────────────────────────────────────────────────┘

See docs/architecture.md for the full design.

Roadmap

• v0.1.0 — MVP: UIA capture/detect/click + CLI + MCP + 飞书 first showcase
• v0.2.0 — Vision backend (OmniParser v2 / Florence-2)
• v0.3.0 — macOS (AXUIElement) + Linux (AT-SPI)
• v0.4.0 — Vimium-style letter hint overlay
• v0.5.0 — LLM intent resolver
• v0.6.0 — Audit log + replay
• v1.0.0 — Rust hot path + cross-platform binaries

Repository layout

openeyes/
├── openeyes/              # main package
│   ├── core/              # windows + capture + detect + click primitives
│   │   ├── windows.py     # EnumWindows wrapper
│   │   ├── capture.py     # PIL.ImageGrab wrapper
│   │   ├── schema.py      # Element / WindowInfo dataclasses
│   │   ├── selector.py    # element find / filter
│   │   └── _platform.py   # cross-platform backend selection
│   ├── backends/
│   │   └── uia.py         # Windows UIA backend (pywinauto)
│   ├── actuators/
│   │   └── win32.py       # Windows mouse/keyboard input
│   ├── cli/               # `eyes` command
│   │   └── main.py
│   └── mcp/               # MCP server
│       └── server.py
├── examples/
│   ├── feishu_first_test.py
│   └── anyvpn_keepalive.py
├── tests/
│   └── test_smoke.py
├── .codex-plugin/
│   └── plugin.json
├── skills/openeyes/
│   └── SKILL.md
├── pyproject.toml
├── LICENSE                 # MIT
└── README.md

License

MIT — see LICENSE.