webpeel

Turn any web page into AI-ready markdown. Smart escalation. Stealth mode. Free to start.

Website · Docs · Playground · Dashboard · Discussions

Quick Start

# Zero install — just run it
npx webpeel https://news.ycombinator.com

# Stealth mode (bypass bot detection)
npx webpeel https://protected-site.com --stealth

# Crawl a website
npx webpeel crawl https://example.com --max-pages 20

# Search the web
npx webpeel search "best AI frameworks 2026"

# Autonomous agent (BYOK LLM)
npx webpeel agent "Find the founders of Stripe" --llm-key sk-...

First 25 fetches work instantly, no signup. After that, sign up free for 125/week.

Why WebPeel?

Install

# Node.js
npm install webpeel        # or: pnpm add webpeel

# Python
pip install webpeel

# Global CLI
npm install -g webpeel

Usage

Node.js

import { peel } from 'webpeel';

const result = await peel('https://example.com');
console.log(result.content);    // Clean markdown
console.log(result.metadata);   // { title, description, author, ... }
console.log(result.tokens);     // Estimated token count

// With options
const advanced = await peel('https://example.com', {
  render: true,           // Browser for JS-heavy sites
  stealth: true,          // Anti-bot stealth mode
  maxTokens: 4000,        // Limit output
  includeTags: ['main'],  // Filter HTML tags
});

Python

from webpeel import WebPeel

client = WebPeel()  # Free tier, no key needed

result = client.scrape("https://example.com")
print(result.content)  # Clean markdown

results = client.search("python web scraping")
job = client.crawl("https://docs.example.com", limit=100)

Zero dependencies. Pure Python 3.8+. Full SDK docs →

MCP Server

11 tools for Claude Desktop, Cursor, VS Code, and Windsurf:

webpeel_fetch · webpeel_search · webpeel_crawl · webpeel_map · webpeel_extract · webpeel_batch · webpeel_brand · webpeel_change_track · webpeel_summarize · webpeel_answer · webpeel_screenshot

{
  "mcpServers": {
    "webpeel": {
      "command": "npx",
      "args": ["-y", "webpeel", "mcp"]
    }
  }
}

Where to add this config: Claude Desktop → ~/Library/Application Support/Claude/claude_desktop_config.json · Cursor → Settings → MCP Servers · VS Code → ~/.vscode/mcp.json · Windsurf → ~/.codeium/windsurf/mcp_config.json

Docker (Self-Hosted)

git clone https://github.com/webpeel/webpeel.git
cd webpeel && docker compose up

Full API at http://localhost:3000. AGPL-3.0 licensed. Commercial licensing available.

Features

🎯 Smart Escalation

Automatically uses the fastest method, escalates only when needed:

HTTP Fetch (200ms)  →  Browser Rendering (2s)  →  Stealth Mode (5s)
     80% of sites          15% of sites             5% of sites

🎭 Stealth Mode

Bypass Cloudflare and bot detection. Masks browser fingerprints, navigator properties, WebGL vendor.

npx webpeel https://protected-site.com --stealth

🕷️ Crawl & Map

Crawl websites with link following, sitemap discovery, robots.txt compliance, and deduplication.

npx webpeel crawl https://docs.example.com --max-pages 100
npx webpeel map https://example.com --max-urls 5000

🤖 Autonomous Agent (BYOK)

Give it a prompt, it researches the web using your own LLM key.

npx webpeel agent "Compare pricing of Notion vs Coda" --llm-key sk-...

📊 More Features

Integrations

Works with LangChain, LlamaIndex, CrewAI, Dify, and n8n. Integration docs →

Hosted API

Live at api.webpeel.dev — Firecrawl-compatible endpoints.

# Fetch a page (free, no auth needed for first 25)
curl "https://api.webpeel.dev/v1/fetch?url=https://example.com"

# With API key
curl "https://api.webpeel.dev/v1/fetch?url=https://example.com" \
  -H "Authorization: Bearer wp_..."

Pricing

Extra credit costs: fetch $0.002, search $0.001, stealth $0.01. Resets every Monday. All features on all plans. Compare with Firecrawl →

Project Structure

webpeel/
├── src/
│   ├── core/           # Core library (fetcher, strategies, markdown, crawl, search)
│   ├── mcp/            # MCP server (11 tools for AI assistants)
│   ├── server/         # Express API server (hosted version)
│   │   ├── routes/     # API route handlers
│   │   ├── middleware/  # Auth, rate limiting, SSRF protection
│   │   └── premium/    # Server-only premium features
│   ├── tests/          # Vitest test suites
│   ├── cli.ts          # CLI entry point
│   ├── index.ts        # Library exports
│   └── types.ts        # TypeScript type definitions
├── python-sdk/         # Python SDK (PyPI: webpeel)
├── integrations/       # LangChain, LlamaIndex, CrewAI, Dify, n8n
├── site/               # Landing page (webpeel.dev)
├── dashboard/          # Next.js dashboard (app.webpeel.dev)
├── benchmarks/         # Performance comparison suite
└── skills/             # AI agent skills (Claude Code, etc.)

Development

git clone https://github.com/webpeel/webpeel.git
cd webpeel
npm install && npm run build
npm test

See CONTRIBUTING.md for guidelines.

Links

Documentation · Playground · API Reference · npm · PyPI · Migration Guide · Blog · Discussions

Star History

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0).

What this means:

• ✅ Free to use, modify, and distribute
• ✅ Free for personal and commercial use
• ⚠️ If you run a modified version as a network service, you must release your source code under AGPL-3.0

Need a commercial license? Contact us at [email protected] for proprietary/enterprise licensing.

Note: Versions 0.7.1 and earlier were released under MIT. Those releases remain MIT-licensed.

© WebPeel

Like WebPeel? ⭐ Star us on GitHub — it helps others discover the project!