web-researcher-mcp

web-researcher-mcp

A production-grade MCP server that gives AI assistants the power to search the web, extract content, and conduct multi-source research.

Why Web Researcher MCP?

AI assistants are only as good as the information they can access. web-researcher-mcp bridges the gap between LLMs and the live internet through the Model Context Protocol standard:

• Multiple specialized research tools in a single server (see Tools below)
• Pluggable search backends (Google, Brave, Serper, SearXNG)
• 4-tier content extraction -- markdown negotiation, stealth HTTP, HTML parsing, headless browser (go-rod + stealth)
• Search lenses for domain-focused research (programming, news, legal, medical, and more)
• Single static binary with optional Chromium for JS rendering (auto-downloaded on first use)
• Enterprise-ready with OAuth 2.1, multi-tenancy, rate limiting, and audit logging

Works with Claude Code, Claude Desktop, Cursor, and any MCP-compatible client.

Tools

Quick Start

Option 1: Install with Go (Recommended)

go install github.com/zoharbabin/web-researcher-mcp/cmd/web-researcher-mcp@latest

The binary is now in your $GOPATH/bin. Add to Claude Code:

claude mcp add --scope user --transport stdio web-researcher -- web-researcher-mcp

Option 2: Download Binary

Download the latest release for your platform from Releases. Archives are named web-researcher-mcp_<version>_<os>_<arch>.tar.gz.

# Example: macOS Apple Silicon (replace VERSION with actual version, e.g. 1.0.4)
curl -L https://github.com/zoharbabin/web-researcher-mcp/releases/download/v${VERSION}/web-researcher-mcp_${VERSION}_darwin_arm64.tar.gz | tar xz
chmod +x web-researcher-mcp

Option 3: Docker

docker run -e GOOGLE_CUSTOM_SEARCH_API_KEY=YOUR_KEY \
           -e GOOGLE_CUSTOM_SEARCH_ID=YOUR_CX \
           docker.io/zoharbabin/web-researcher-mcp:latest

Also available from GHCR: ghcr.io/zoharbabin/web-researcher-mcp:latest

Option 4: Build from Source

git clone https://github.com/zoharbabin/web-researcher-mcp.git
cd web-researcher-mcp
go build -o web-researcher-mcp ./cmd/web-researcher-mcp

Connect to Your AI Assistant

Add this to your MCP client configuration (example for Claude Code ~/.claude/settings.json):

{
  "mcpServers": {
    "web-researcher": {
      "command": "/path/to/web-researcher-mcp",
      "env": {
        "GOOGLE_CUSTOM_SEARCH_API_KEY": "YOUR_GOOGLE_API_KEY",
        "GOOGLE_CUSTOM_SEARCH_ID": "YOUR_SEARCH_ENGINE_ID"
      }
    }
  }
}

Done. Your AI assistant now has access to all research tools.

Configuration

Required

Search Provider

HTTP Transport (Optional)

See docs/DEPLOYMENT.md for the complete reference of all environment variables (cache, rate limiting, scraping, observability, etc.).

Architecture

web-researcher-mcp/
├── cmd/web-researcher-mcp/     # Entry point (wiring only)
├── internal/
│   ├── config/                 # Env-based strongly-typed configuration
│   ├── server/                 # MCP server lifecycle + signal handling
│   ├── tools/                  # Tool handlers (one file per tool)
│   ├── search/                 # Pluggable search providers + lens routing
│   ├── scraper/                # 4-tier scraping pipeline (markdown → stealth → HTML → browser)
│   ├── documents/              # PDF, DOCX, PPTX parsing
│   ├── cache/                  # Hybrid cache (memory + disk + optional Redis)
│   ├── auth/                   # OAuth 2.1 middleware + JWKS
│   ├── session/                # Per-tenant session management
│   ├── content/                # Sanitize, dedup, truncate, quality score
│   ├── metrics/                # Prometheus metrics + per-tool stats
│   ├── ratelimit/              # Three-tier rate limiting
│   ├── circuit/                # Circuit breaker for external APIs
│   └── resources/              # MCP Resources + Prompts
├── lenses/                     # Search lens JSON files
└── docs/                       # Extended documentation

┌─────────────────────────────────────────────────────────────────┐
│                         MCP Protocol Layer                        │
│  ┌──────────────────┐              ┌─────────────────────────┐  │
│  │  STDIO Transport │              │  HTTP/SSE Transport     │  │
│  │  (zero-config)   │              │  (OAuth 2.1 + CORS)     │  │
│  └────────┬─────────┘              └──────────┬──────────────┘  │
│           └────────────────┬───────────────────┘                 │
│                    ┌───────▼───────┐                             │
│                    │  MCP Server   │                             │
│                    │  (go-sdk)     │                             │
│                    └───────┬───────┘                             │
└────────────────────────────┼─────────────────────────────────────┘
                             │
┌────────────────────────────┼─────────────────────────────────────┐
│                    Tool Dispatch Layer                             │
│  ┌─────────┐ ┌────────┐ ┌┴───────┐ ┌────────┐ ┌─────────────┐  │
│  │ Search  │ │ Scrape │ │Combined│ │Academic│ │ Sequential  │  │
│  │ Tools   │ │ Tool   │ │  Tool  │ │& Patent│ │  Research   │  │
│  └────┬────┘ └───┬────┘ └───┬────┘ └───┬────┘ └──────┬──────┘  │
└───────┼──────────┼───────────┼──────────┼─────────────┼──────────┘
        │          │           │          │             │
┌───────┼──────────┼───────────┼──────────┼─────────────┼──────────┐
│       │     Service Layer    │          │             │           │
│  ┌────▼────┐ ┌───▼────┐ ┌───▼───┐ ┌───▼────┐ ┌─────▼─────┐   │
│  │ Search  │ │Scraper │ │Quality│ │Citation│ │  Session   │   │
│  │Provider │ │Pipeline│ │Scorer │ │Extract │ │  Manager   │   │
│  └────┬────┘ └───┬────┘ └───────┘ └────────┘ └────────────┘   │
│       │          │                                               │
│  ┌────▼─────┐ ┌─▼──────────────────────────────────┐           │
│  │ Brave    │ │  Scraper Tiers (4-tier pipeline)     │           │
│  │ Google   │ │  markdown > stealth > HTML > browser│           │
│  │ Serper   │ │  + YouTube (3-strategy) + documents │           │
│  │ SearXNG  │ └─────────────────────────────────────┘           │
│  └──────────┘                                                    │
└──────────────────────────────────────────────────────────────────┘
        │          │
┌───────┼──────────┼──────────────────────────────────────────────┐
│       │   Infrastructure Layer                                    │
│  ┌────▼────┐ ┌───▼────┐ ┌─────────┐ ┌────────┐ ┌───────────┐  │
│  │  Cache  │ │  SSRF  │ │  Rate   │ │Metrics │ │   Audit   │  │
│  │(hybrid) │ │Protect │ │ Limiter │ │(Prom.) │ │   Logger  │  │
│  └─────────┘ └────────┘ └─────────┘ └────────┘ └───────────┘  │
│  ┌──────────────────┐  ┌──────────────────────────────────────┐ │
│  │  Circuit Breaker  │  │  Content Pipeline (sanitize, dedup,  │ │
│  │                   │  │  truncate, quality score)             │ │
│  └───────────────────┘  └──────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────┘

Design Principles

1. Zero global state -- all dependencies injected via constructors
2. Interface-driven -- every external dependency behind an interface for testing and swapping
3. Bounded concurrency -- explicit semaphores for external API calls
4. Defense in depth -- SSRF protection, rate limiting, content sanitization at every layer
5. Fail loud -- errors returned, never swallowed; validation at boundaries

Search Providers

The server supports four search backends. Google PSE is always used for lens-restricted and site-restricted queries (free, works indefinitely). The configured provider handles unrestricted whole-web searches.

Routing Logic

Request arrives
  |-- lens specified?     --> Google PSE (site-restricted, free forever)
  |-- site: param set?    --> Google PSE (site-restricted)
  `-- unrestricted?       --> Configured SEARCH_PROVIDER

Brave Search (recommended for whole-web):

export SEARCH_PROVIDER=brave
export BRAVE_API_KEY=BSAxxxxxxxxxx
export GOOGLE_CUSTOM_SEARCH_API_KEY=AIza...  # still needed for lenses
export GOOGLE_CUSTOM_SEARCH_ID=017...

SearXNG (self-hosted, privacy-first):

export SEARCH_PROVIDER=searxng
export SEARXNG_URL=http://localhost:8080
export GOOGLE_CUSTOM_SEARCH_API_KEY=AIza...
export GOOGLE_CUSTOM_SEARCH_ID=017...

Google PSE only (simplest setup):

export GOOGLE_CUSTOM_SEARCH_API_KEY=AIza...
export GOOGLE_CUSTOM_SEARCH_ID=017...
# SEARCH_PROVIDER defaults to "google"

Search Lenses

Search lenses are curated domain lists that focus search results on high-quality sources for specific topics. They route through Google PSE in site-restricted mode -- free and works indefinitely.

Built-in Lenses

Each lens is a JSON file in lenses/ containing the curated domain list. See Creating Custom Lenses below for the format.

Usage Example

{
  "tool": "web_search",
  "arguments": {
    "query": "golang context best practices",
    "lens": "programming"
  }
}

This searches only stackoverflow.com, github.com, go.dev, developer.mozilla.org, and other curated programming sites.

Add a JSON file to the lenses/ directory:

{
  "name": "my-custom-lens",
  "description": "Description of what this lens covers",
  "domains": [
    "example.com",
    "docs.example.org",
    "*.trusted-source.io"
  ],
  "cx": ""
}

Fields:

• domains -- Up to 5,000 URL patterns per lens (Google PSE limit)
• cx -- Optional dedicated PSE engine ID. If empty, site: operators are injected at query time (limited to ~10 domains per query)

Security

The server implements a custom DialContext that validates all resolved IPs before connecting:

• Blocks all private/reserved IP ranges (127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, fc00::/7)
• Blocks cloud metadata endpoints (169.254.169.254)
• Validates against DNS rebinding by connecting only to the first resolved IP
• Re-validates redirect targets at each hop

In HTTP mode, the server supports OAuth 2.1 with:

• JWKS-based token validation with automatic key rotation
• Per-tenant session isolation
• Audience and issuer validation
• Configurable claim extraction for multi-tenancy

Three-tier rate limiting protects both the server and upstream APIs:

1. Per-client -- token bucket per authenticated session
2. Per-provider -- prevents exceeding upstream API quotas
3. Global -- server-wide backpressure valve

• HTML sanitization via whitelist-based policy (bluemonday)
• Paragraph-level deduplication across scraped results
• Smart truncation at natural content breakpoints
• Quality scoring to filter low-value results before returning to the LLM

For the full threat model and security architecture, see docs/SECURITY.md.

MCP Client Integration

Claude Code

Add to ~/.claude/settings.json:

{
  "mcpServers": {
    "web-researcher": {
      "command": "/path/to/web-researcher-mcp",
      "env": {
        "GOOGLE_CUSTOM_SEARCH_API_KEY": "AIza...",
        "GOOGLE_CUSTOM_SEARCH_ID": "017...",
        "SEARCH_PROVIDER": "brave",
        "BRAVE_API_KEY": "BSA..."
      }
    }
  }
}

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "web-researcher": {
      "command": "/path/to/web-researcher-mcp",
      "env": {
        "GOOGLE_CUSTOM_SEARCH_API_KEY": "AIza...",
        "GOOGLE_CUSTOM_SEARCH_ID": "017..."
      }
    }
  }
}

Cursor

Add to .cursor/mcp.json in your project root:

{
  "mcpServers": {
    "web-researcher": {
      "command": "/path/to/web-researcher-mcp",
      "env": {
        "GOOGLE_CUSTOM_SEARCH_API_KEY": "AIza...",
        "GOOGLE_CUSTOM_SEARCH_ID": "017..."
      }
    }
  }
}

HTTP/SSE Mode (Multi-Client, Teams)

For shared deployments serving multiple clients or web applications:

PORT=3000 \
OAUTH_ISSUER_URL=https://auth.example.com \
OAUTH_AUDIENCE=https://api.example.com \
./web-researcher-mcp

Connect any MCP client to http://localhost:3000/sse.

version: "3.8"
services:
  web-researcher:
    image: zoharbabin/web-researcher-mcp
    ports:
      - "3000:3000"
    environment:
      PORT: "3000"
      GOOGLE_CUSTOM_SEARCH_API_KEY: ${GOOGLE_CUSTOM_SEARCH_API_KEY}
      GOOGLE_CUSTOM_SEARCH_ID: ${GOOGLE_CUSTOM_SEARCH_ID}
      SEARCH_PROVIDER: brave
      BRAVE_API_KEY: ${BRAVE_API_KEY}
      REDIS_URL: redis://redis:6379
    depends_on:
      - redis

  redis:
    image: redis:7-alpine
    volumes:
      - redis-data:/data

volumes:
  redis-data:

Performance

Search results are cached in-memory for sub-millisecond hits. The scraping pipeline tries the fastest tier first and falls back progressively — most pages resolve in under a second via stealth HTTP, with the headless browser reserved for JS-heavy sites. See ARCHITECTURE.md for detailed latency breakdowns.

Development

go build -o web-researcher-mcp ./cmd/web-researcher-mcp   # Build
go test -race ./...                                        # Test (with race detector)
golangci-lint run                                          # Lint
govulncheck ./...                                          # Security audit

See CONTRIBUTING.md for the full development workflow, code style guide, and PR process.

Troubleshooting

The server starts even with missing credentials (to allow MCP handshake). Set your API keys in the env block of your MCP client config, not in your shell profile.

The browser tier (go-rod) requires Chromium. On first use it auto-downloads ~200MB. Set CHROME_PATH to use an existing Chrome installation, or use the Docker image which includes headless Chrome.

The disk cache auto-invalidates on version change. If you're running from source without -ldflags, the version is always "dev" — delete the ./cache directory manually or set CACHE_DIR to a versioned path.

Google PSE free tier allows 100 queries/day. Either upgrade to paid ($5/1K queries), or switch to Brave Search (SEARCH_PROVIDER=brave) for unrestricted queries while keeping Google for lens-restricted searches.

Contributing

Contributions are welcome. Please see CONTRIBUTING.md for code style guidelines, development workflow, and how to submit pull requests.

Documentation

License

MIT

Built with Go and the Model Context Protocol If this project helps your workflow, consider giving it a star.