Website Content Crawler MCP Server
MCP wrapper that runs the apify/website-content-crawler actor for real via apify-clientand returns its pages as JSON. Every call either returns crawled content or a structurederror — it never reports a fake success.
What it costs
Runs bill your Apify account at the rate listed forapify/website-content-crawler — thisserver adds nothing on top. No token, no charge: calls return missing_token before anyrun starts.
Live output demo (same crawl logic, rendered): https://website-content-crawler.vercel.app
Setup
npm install
export APIFY_TOKEN=apify_api_... # https://console.apify.com/settings/integrations
npm start # stdio MCP server
Tool: crawl_website
Input:
| field | type | default | notes |
|---|---|---|---|
url |
string | required | http/https URL to crawl |
maxPages |
number | 10 | capped at 50 |
crawlerType |
string | cheerio |
or playwright:chrome for JS-rendered |
Output envelope (same shape every call):
{
"status": "ok",
"run": { "id": "<apify run id>", "status": "SUCCEEDED" },
"page_count": 3,
"total_in_dataset": 3,
"pages": [{ "url": "...", "title": "...", "text": "...(≤5000 chars)" }]
}
Error codes: invalid_url, missing_token, apify_auth_failed, actor_run_failed,run_not_succeeded, dataset_fetch_failed.
Smoke test
printf '%s\n' \
'{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"t","version":"0"}}}' \
'{"jsonrpc":"2.0","method":"notifications/initialized"}' \
'{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \
'{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"crawl_website","arguments":{"url":"https://example.com"}}}' \
| node index.js
With no APIFY_TOKEN set, request id 3 must return{"status":"error","error":{"code":"missing_token",...}} — proof the handler reaches thereal Apify boundary instead of inventing a result.