Terminal Control MCP

A Model Context Protocol (MCP) server that enables AI agents to interact with terminal-based TUI applications through a virtual X11 display approach.

Overview

This project provides a comprehensive solution for controlling terminal applications programmatically using:

Xvfb for headless virtual X11 display
xterm for terminal emulation
xdotool for input simulation and window management
ImageMagick for PNG screenshot capture

The system captures actual visual terminal output as PNG screenshots, making it ideal for AI agents that need to see and interact with terminal applications.

Features

Virtual Display Management: Headless X11 display using Xvfb
Input Simulation: Send keyboard input and text to terminal applications
Screenshot Capture: Take PNG screenshots of terminal output
Window Management: Reliable window detection and focus handling
Resource Cleanup: Proper process management with timeout handling

System Requirements

The following system packages must be installed:

# Ubuntu/Debian
sudo apt-get install xvfb xterm xdotool imagemagick

# CentOS/RHEL/Fedora
sudo yum install xorg-x11-server-Xvfb xterm xdotool ImageMagick

Installation

This project uses the uv package manager:

# Clone the repository
git clone <repository-url>
cd terminal-control-mcp

# Install dependencies
uv sync

# Activate virtual environment
source .venv/bin/activate

Quick Start

Running the Example

Try the included htop example to see the system in action:

python examples/example_htop.py

This will:

Launch htop in a virtual xterm session
Press F3 to open the search dialog
Type "python" as a search term
Capture PNG screenshots at each step
Clean up all processes

Basic Usage

from examples.example_htop import XTermSession

# Create a session
session = XTermSession(width=1920, height=1080)

try:
    # Start virtual display and terminal
    session.start_virtual_display()
    session.start_xterm("your-command-here")
    
    # Take a screenshot
    session.take_screenshot("output.png")
    
    # Send input
    session.send_key("F1")
    session.send_text("hello world")
    
finally:
    session.cleanup()

Architecture

Core Components

XTermSession Class: The main interface for terminal control
- Manages Xvfb virtual display lifecycle
- Spawns and controls xterm processes
- Handles input simulation via xdotool
- Captures screenshots using ImageMagick
Virtual Display Approach: Unlike direct TTY manipulation, this system:
- Creates a real X11 environment with Xvfb
- Launches actual xterm instances
- Captures genuine visual output as PNG files
- Provides reliable input simulation

Key Methods

start_virtual_display(): Initialize Xvfb virtual display
start_xterm(command): Launch xterm with specified command
send_key(key): Send special keys (F1, Escape, etc.)
send_text(text): Send alphanumeric text input
take_screenshot(filename): Capture PNG screenshot
cleanup(): Properly terminate all processes

Development

Project Structure

terminal-control-mcp/
├── src/terminal_control_mcp/    # Main MCP server implementation (planned)
├── examples/
│   ├── example_htop.py         # Reference implementation
│   └── README.md
├── tests/                      # Test suite
├── pyproject.toml             # Project configuration
└── CLAUDE.md                  # Development guidelines

Development Commands

# Run the main application
python main.py

# Run the htop example
python examples/example_htop.py

# Activate virtual environment
source .venv/bin/activate

MCP Server Implementation (Planned)

The full MCP server will provide these tools:

terminal_launch: Start a new terminal session
terminal_input: Send keyboard/text input
terminal_capture: Take PNG screenshot
terminal_close: Clean up terminal session

Technical Details

Window Detection

The system uses multiple fallback strategies for reliable window ID detection:

search_methods = [
    ['xdotool', 'search', '--class', 'XTerm'],
    ['xdotool', 'search', '--name', 'xterm'],
    ['xdotool', 'search', '--class', 'xterm'],
    ['xdotool', 'getactivewindow']
]

Screenshot Capture

Uses ImageMagick's import command for reliable PNG capture:

subprocess.run(['import', '-window', 'root', filename], env=env)

Resource Management

Implements proper cleanup with timeout handling:

def cleanup(self):
    if self.xterm_proc:
        self.xterm_proc.terminate()
        try:
            self.xterm_proc.wait(timeout=5)
        except subprocess.TimeoutExpired:
            self.xterm_proc.kill()

Contributing

[TBD]

Terminal Control MCP