MCP WebScout
A Model Context Protocol (MCP) server providing web search (DuckDuckGo) and intelligent content extraction with LLM-powered analysis.
Features
- search: Search the web using DuckDuckGo
- fetch: Advanced web fetching with Crawl4AI and LLM extraction
System Requirements
| Requirement | Version | Notes |
|---|---|---|
| Python | >= 3.10 | Required runtime environment |
| pip | latest | Package manager (included with Python) |
| Playwright | latest | Required by Crawl4AI for browser automation |
| DeepSeek API Key | - | Required for LLM extraction mode |
| Proxy (optional) | - | Required for users in mainland China |
Python Dependencies (14 packages)
| Package | Version | Purpose |
|---|---|---|
| mcp | >=1.0.0 | MCP protocol implementation |
| duckduckgo-search | >=3.0.0 | DuckDuckGo search API |
| requests | >=2.32.0 | HTTP requests |
| beautifulsoup4 | >=4.12.0 | HTML parsing |
| openai | >=1.30.0 | OpenAI API client for DeepSeek |
| crawl4ai | >=0.5.0 | Advanced web scraping |
Quick Start
Get started in 5 steps:
1. Clone and Setup Environment
git clone <repository>
cd mcp-webscout
python -m venv .venv
On Windows:
.venv\Scripts\activate
On macOS/Linux:
source .venv/bin/activate
2. Install Dependencies
pip install -e ".[dev]"
3. Install Playwright Browsers
playwright install chromium
4. Configure Environment Variables
cp .env.example .env
Edit .env and add your configuration:
# Required for LLM extraction
DEEPSEEK_API_KEY=sk-your-actual-key-here
# Required for mainland China users
PROXY_URL=http://127.0.0.1:7890
USE_PROXY=true
5. Verify Installation
# Run tests
pytest tests/ -v
# Test the server
python -m mcp_webscout --help
Detailed Configuration
For detailed environment setup instructions, see ENV_SETUP.md.
Usage
As a Command
mcp-webscout
As a Python Module
python -m mcp_webscout
With Claude Desktop
Add to your claude_desktop_config.json:
Basic Configuration
{
"mcpServers": {
"webscout": {
"command": "mcp-webscout"
}
}
}
With Environment Variables (Recommended)
{
"mcpServers": {
"webscout": {
"command": "mcp-webscout",
"env": {
"DEEPSEEK_API_KEY": "sk-your-key-here",
"PROXY_URL": "http://127.0.0.1:7890",
"USE_PROXY": "true",
"DEFAULT_MAX_LENGTH": "5000",
"PYTHONUTF8": "1"
}
}
}
}
Windows Configuration
{
"mcpServers": {
"webscout": {
"command": "python",
"args": ["-m", "mcp_webscout"],
"env": {
"DEEPSEEK_API_KEY": "sk-your-key-here",
"PROXY_URL": "http://127.0.0.1:7890",
"USE_PROXY": "true",
"PYTHONUTF8": "1"
}
}
}
}
Tools
search
Search the web using DuckDuckGo.
Parameters:
| Name | Type | Required | Description |
|---|---|---|---|
| query | string | Yes | Search query |
| max_results | integer | No | Maximum results (1-10, default: 5) |
Returns:
Formatted search results with titles, URLs, and snippets.
Example:
{
"query": "Python programming",
"max_results": 3
}
fetch
Advanced web fetching with Crawl4AI and LLM extraction.
Parameters:
| Name | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | URL to fetch |
| mode | string | No | Extraction mode: simple, llm (default: simple) |
| prompt | string | No | Custom extraction prompt for LLM mode |
| max_length | integer | No | Maximum characters (default: 5000) |
| use_proxy | boolean | No | Use proxy (default: true) |
Returns:
Fetched and optionally extracted content.
