Fetch URL MCP Server

Fetch URL MCP Server

Fetch public web pages and convert them into clean, AI-readable Markdown.

Overview

Fetch URL is a Model Context Protocol (MCP) server that fetches public web pages, extracts meaningful content using Mozilla's Readability algorithm, and converts the result into clean Markdown optimized for LLM context windows. It handles noise removal, caching, SSRF protection, async task execution, and supports both stdio and Streamable HTTP transports.

[!NOTE]Content extraction quality varies depending on the HTML structure and complexity of the source page. Fetch URL works best with standard article and documentation layouts. Pages relying on client-side JavaScript rendering may yield incomplete results.

Key Features

• HTML to Markdown — Content extraction via Mozilla Readability + node-html-markdown
• Noise removal — Strips navigation, ads, cookie banners, and other non-content elements
• In-memory LRU cache — Faster repeat fetches with configurable TTL (24 h default)
• Raw URL rewriting — Auto-converts GitHub, GitLab, Bitbucket, and Gist URLs to raw content endpoints

Tech Stack

Architecture

URL → Validate → DNS Preflight → HTTP Fetch → Decompress
  → Truncate HTML → Readability Extract → Noise Removal
  → Markdown Convert → Cleanup Pipeline → Cache → Response

1. URL Validation — Normalize, block private hosts, transform raw-content URLs (GitHub, GitLab, Bitbucket)
2. Fetch — HTTP request with redirect following, DNS preflight SSRF checks, and size limits (10 MB)
3. Transform — Offloaded to worker threads: parse HTML with linkedom, extract with Readability, remove DOM noise, convert to Markdown
4. Cleanup — Multi-pass Markdown normalization (heading promotion, spacing, skip-link removal)
5. Cache + Respond — Store result in LRU cache, apply inline content limits, return structured content

Repository Structure

fetch-url-mcp/
├── assets/              # Server icon (logo.svg)
├── scripts/             # Build & test orchestration
├── src/
│   ├── workers/         # Worker-thread child for HTML transforms
│   ├── index.ts         # CLI entrypoint, transport wiring, shutdown
│   ├── server.ts        # McpServer lifecycle and registration
│   ├── tools.ts         # fetch-url tool definition and pipeline
│   ├── fetch.ts         # URL normalization, SSRF, HTTP fetch
│   ├── transform.ts     # HTML-to-Markdown pipeline, worker pool
│   ├── config.ts        # Env-driven configuration
│   ├── resources.ts     # MCP resource/template registration
│   ├── prompts.ts       # MCP prompt registration (get-help)
│   ├── mcp.ts           # Task execution management
│   ├── http-native.ts   # Streamable HTTP server, auth, sessions
│   └── instructions.md  # Server instructions embedded at runtime
├── tests/               # Unit/integration tests (Node.js test runner)
├── package.json
├── tsconfig.json
└── AGENTS.md

Requirements

• Node.js >= 24

Quickstart

npx -y @j0hanz/fetch-url-mcp@latest --stdio

Add to your MCP client configuration:

{
  "mcpServers": {
    "fetch-url-mcp": {
      "command": "npx",
      "args": ["-y", "@j0hanz/fetch-url-mcp@latest", "--stdio"]
    }
  }
}

Installation

NPX (Recommended)

No installation required — runs directly:

npx -y @j0hanz/fetch-url-mcp@latest --stdio

Global Install

npm install -g @j0hanz/fetch-url-mcp
fetch-url-mcp --stdio

From Source

git clone https://github.com/j0hanz/fetch-url-mcp.git
cd fetch-url-mcp
npm install
npm run build
node dist/index.js --stdio

Docker

docker compose up --build

Configuration

Runtime Modes

When no --stdio flag is passed, the server starts in HTTP mode (Streamable HTTP on port 3000 by default).

Environment Variables

Core Settings

Task Management

Authentication (HTTP Mode)

Transform & Workers

Content Tuning

Server Tuning

Hardcoded Defaults

Usage

Stdio Mode

fetch-url-mcp --stdio

The server communicates via JSON-RPC over stdin/stdout. All MCP clients that support stdio transport can connect directly.

HTTP Mode

fetch-url-mcp
# or
PORT=8080 HOST=0.0.0.0 ALLOW_REMOTE=true fetch-url-mcp

The server starts a Streamable HTTP endpoint at /mcp. Authenticate with bearer tokens via the ACCESS_TOKENS or API_KEY environment variables.

For POST /mcp, clients should send:

• Accept: application/json, text/event-stream
• MCP-Protocol-Version: 2025-11-25 (or 2025-03-26 for legacy clients)

MCP Surface

Tools

fetch-url

Fetches a webpage and converts it to clean Markdown format optimized for LLM context.

Useful for:

• Reading documentation, blog posts, or articles
• Extracting main content while removing navigation and ads
• Caching content to speed up repeated queries

Limitations:

• Does not execute complex client-side JavaScript interactions
• Inline output may be truncated when MAX_INLINE_CONTENT_CHARS is set

Parameters

Returns

{
  "url": "https://example.com",
  "inputUrl": "https://example.com",
  "resolvedUrl": "https://example.com",
  "finalUrl": "https://example.com",
  "title": "Example Domain",
  "metadata": {
    "title": "Example Domain",
    "description": "...",
    "author": "...",
    "image": "...",
    "favicon": "...",
    "publishedAt": "...",
    "modifiedAt": "..."
  },
  "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
  "fromCache": false,
  "fetchedAt": "2026-02-11T12:00:00.000Z",
  "contentSize": 1234,
  "truncated": false
}

Annotations

Async Task Execution

The fetch-url tool supports optional async task execution (execution.taskSupport: "optional"). Include a task field in the tool call to run the fetch in the background:

{
  "method": "tools/call",
  "params": {
    "name": "fetch-url",
    "arguments": { "url": "https://example.com" },
    "task": { "ttl": 30000 }
  }
}

Then poll tasks/get until the task status is completed or failed, and retrieve the result via tasks/result.

Prompts

Resources

Tasks

The server declares full MCP task support:

HTTP Mode Endpoints

* verbose=true can be read without auth only for local-only deployments (ALLOW_REMOTE=false).

Session Behavior

• Sessions are created on the first POST /mcp request with an initialize message
• Session ID is returned in the mcp-session-id response header
• Sessions expire after 30 minutes of inactivity (max 200 concurrent)

Authentication

• Static tokens: Set ACCESS_TOKENS or API_KEY environment variables; pass as Authorization: Bearer <token>
• OAuth: Configure OAUTH_* environment variables to enable OAuth 2.0 token introspection

Client Configuration Examples

Add to your VS Code settings (.vscode/mcp.json or User Settings):

{
  "servers": {
    "fetch-url-mcp": {
      "command": "npx",
      "args": ["-y", "@j0hanz/fetch-url-mcp@latest", "--stdio"]
    }
  }
}

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "fetch-url-mcp": {
      "command": "npx",
      "args": ["-y", "@j0hanz/fetch-url-mcp@latest", "--stdio"]
    }
  }
}

Or manually add to Cursor MCP settings:

{
  "mcpServers": {
    "fetch-url-mcp": {
      "command": "npx",
      "args": ["-y", "@j0hanz/fetch-url-mcp@latest", "--stdio"]
    }
  }
}

Add to your Windsurf MCP configuration:

{
  "mcpServers": {
    "fetch-url-mcp": {
      "command": "npx",
      "args": ["-y", "@j0hanz/fetch-url-mcp@latest", "--stdio"]
    }
  }
}

Use the published image from GitHub Container Registry:

{
  "mcpServers": {
    "fetch-url-mcp": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/j0hanz/fetch-url-mcp:latest",
        "--stdio"
      ]
    }
  }
}

Or build and run locally:

docker build -t fetch-url-mcp .
docker run -i --rm fetch-url-mcp --stdio

Security

SSRF Protection

Fetch URL blocks requests to private and internal network addresses:

• Blocked hosts: localhost, 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16, 100.64.0.0/10
• Blocked IPv6: ::1, fc00::/7, fe80::/10, IPv4-mapped private addresses
• Cloud metadata: 169.254.169.254 (AWS), metadata.google.internal, metadata.azure.com, 100.100.100.200 (Azure IMDS)

DNS preflight checks run on every redirect hop to prevent DNS rebinding attacks.

Stdio Transport Safety

The server never writes non-protocol data to stdout. All logs and diagnostics go to stderr.

Rate Limiting

HTTP mode enforces a rate limit of 100 requests per 60-second window per client.

Content Safety

• HTML downloads are capped at 10 MB
• Worker threads run in isolation with configurable resource limits
• Auth tokens are stored in-memory only and compared using timing-safe equality

Development Workflow

Install Dependencies

npm install

Scripts

Build and Release

npm run build           # Clean → Compile → Copy Assets → chmod
npm run prepublishOnly  # Lint → Type-Check → Build
npm publish             # Publish to npm

CI/CD is handled via a GitHub Actions workflow (release.yml) that runs lint, type-check, test, build, and publishes to npm with version bumping.

Troubleshooting

MCP Inspector

Use the built-in inspector to test the server interactively:

npm run inspector

Common Issues

Stdout / Stderr Guidance

In stdio mode, stdout is reserved exclusively for MCP JSON-RPC messages. Logs and diagnostics are written to stderr. Never pipe stdout to a log file when using stdio transport.

License

MIT