Web Crawler MCP

Web Crawler MCP

A powerful web crawling tool that integrates with AI assistants via the MCP (Model Context Protocol). This project allows AI assistants to crawl websites, extract dynamic content, navigate through links, and save structured Markdown files directly.

📋 Features

• Native integration with AI assistants via MCP
• Return scraped Markdown content directly to the AI
• Extracts and surfaces internal/external links for AI navigation
• Website crawling with configurable depth
• Detailed crawl result statistics
• Error and not found page handling
• Advanced Scraping Capabilities:
  • Magic Mode: Bypass anti-bots (like Cloudflare) and simulate real browser behavior
  • Targeted Extraction: Fetch only what you need using CSS selectors
  • Custom JavaScript: Execute code before extraction (clicks, scrolls, form fills)
  • Persistent Sessions: Keep cookies and state across requests for authenticated sites
  • SPA Support: Wait for dynamic CSS selectors or set explicit pre-extraction delays

🚀 MCP Configuration

The simplest and recommended way to use this tool is via uvx, which automatically fetches and runs the latest published version from PyPI.

Prerequisites

• uv installed on your system.

Setup for AI Assistants (e.g., Claude Desktop, Cline)

Add the following to your AI Assistant's MCP configuration file (e.g., cline_mcp_settings.json or claude_desktop_config.json):

Note for Windows Users: It is highly recommended to specify --python 3.13 to avoid compilation issues with certain dependencies.

From PyPI (recommended):

{
  "mcpServers": {
    "crawl": {
      "command": "uvx",
      "args": [
        "--python",
        "3.13",
        "crawl4ai-mcp-llm"
      ],
      "disabled": false,
      "autoApprove": [],
      "timeout": 600
    }
  }
}

From GitHub (latest unreleased):

{
  "mcpServers": {
    "crawl": {
      "command": "uvx",
      "args": [
        "--python",
        "3.13",
        "--from",
        "git+https://github.com/laurentvv/crawl4ai-mcp-llm",
        "crawl4ai-mcp-llm"
      ],
      "disabled": false,
      "autoApprove": [],
      "timeout": 600
    }
  }
}

Important: Browser Installation

The crawler uses Playwright to handle dynamic content. You must install the required browsers after setting up the tool:

uv run playwright install chromium

🖥️ Usage

Once configured, you can use the crawler by asking your AI assistant to perform a crawl.

Usage Examples with Claude/Cline

• Simple Crawl: "Can you crawl the site example.com and give me a summary?"
• Crawl with Options: "Can you crawl https://example.com with a depth of 3 and include external links?"
• Dynamic Content: "Crawl this React app and wait for the .main-content selector to load."
• Bypass Protections: "Crawl example.com but use 'magic mode' to bypass the anti-bot protection."
• Targeted Extraction: "Crawl the docs site but only extract content matching the h1, p.lead CSS selector."

🛠️ Available Parameters (MCP Tool)

The crawl tool accepts the following parameters:

👨‍💻 Development

If you want to modify the crawler or run it locally:

1. Clone this repository:

git clone https://github.com/laurentvv/crawl4ai-mcp-llm
cd crawl4ai-mcp-llm

2. Install dependencies using uv:

uv sync

3. Test the MCP server locally using the official MCP Inspector:

npx -y @modelcontextprotocol/inspector uv run crawl4ai-mcp-llm

4. Run the automated test suite:

uv run pytest tests/

5. Run the MCP server directly (for standard usage):

uv run crawl4ai-mcp-llm

🤝 Contribution

Contributions are welcome! Feel free to open an issue or submit a pull request.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.