Crawlberg

Crawlberg

Turn any website into clean, structured data. Point Crawlberg at a URL and get back Markdown, metadata, and links — from a single page or a whole site — in the language you already use.

What and Why?

You need data that lives on the web, and raw HTML is not it. Crawlberg does the crawling, scraping, and cleanup end-to-end: it fetches pages, follows links, converts each one to Markdown, and hands you structured metadata (titles, links, images, social-card and JSON-LD data) — so you skip the parsing and go straight to the content.

It runs from a single Rust core with identical results across 14 language bindings, and it handles the awkward parts for you: JavaScript-heavy pages fall back to a real headless browser, bot filters are detected and worked around, robots and sitemaps are respected, requests are throttled per domain, and requests to private or internal addresses are refused by default. Drive it from your code, an AI agent, a REST service, or the CLI.

Every part of the pipeline is a trait you can swap — the crawl frontier, rate limiter, storage, event stream, and content filters — so you can plug in your own behavior. Managed extras like proxy pools, tuned bot-evasion, authenticated sessions, scheduling, and billing live in xberg-enterprise.

Features

Supported Platforms

Precompiled binaries for Linux (x86_64/aarch64), macOS (ARM64), and Windows (x64) across every binding. See the platform support reference for the full matrix.

⭐ Star this repo to show your support — it helps others discover Crawlberg.

Quick Start

Language Packages

pip install crawlberg

See Python README for full documentation.

npm install @xberg-io/crawlberg

See Node.js README for full documentation.

cargo add crawlberg

See Rust README for full documentation.

go get github.com/xberg-io/crawlberg/packages/go

See Go README for full documentation.

Available on Maven Central as io.xberg.crawlberg:crawlberg. See Java README for the dependency snippet and current version.

dotnet add package Crawlberg

See C# README for full documentation.

gem install crawlberg

See Ruby README for full documentation.

composer require xberg-io/crawlberg

See PHP README for full documentation.

Add {:crawlberg, "~> 0.3"} to your mix.exs dependencies. See Elixir README for full documentation.

dart pub add crawlberg

See Dart README for full documentation.

Available on Maven Central as io.xberg.crawlberg.android:crawlberg-android. See Kotlin README for the dependency snippet and current version.

Add via Swift Package Manager. See Swift README for full documentation.

See Zig README for installation and usage.

npm install @xberg-io/crawlberg-wasm

See WebAssembly README for full documentation.

C header + shared library from GitHub Releases. See FFI crate for full documentation.

cargo install crawlberg-cli

brew install xberg-io/tap/crawlberg

See CLI README for full documentation.

AI Coding Assistants

Install the Crawlberg plugin from xberg-io/crawlberg. It ships the Crawlberg agent skills (site crawling, HTML→Markdown scraping, headless-Chrome fallback) plus the crawlberg MCP server, and works with every major coding agent — expand your harness below.

/plugin marketplace add xberg-io/crawlberg
/plugin install crawlberg@crawlberg

/plugins add https://github.com/xberg-io/crawlberg

Then search for crawlberg and select Install Plugin.

Settings → Plugins → Add from URL → https://github.com/xberg-io/crawlberg, then select crawlberg.

gemini extensions install https://github.com/xberg-io/crawlberg

droid plugin marketplace add https://github.com/xberg-io/crawlberg
droid plugin install crawlberg@crawlberg

copilot plugin marketplace add https://github.com/xberg-io/crawlberg
copilot plugin install crawlberg@crawlberg

Add the package to opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "plugin": ["@xberg-io/opencode-crawlberg"]
}

Documentation

Full guides, per-language API references, the substrate/operational model, antibot strategy, and observability live at docs.crawlberg.xberg.io.

Contributing

Contributions are welcome! See our Contributing Guide.

Part of Xberg.dev

• Xberg — document intelligence: text, tables, metadata from 98+ formats with optional OCR.
• Xberg Enterprise — managed extraction API with SDKs, dashboards, and observability.
• crawlberg — web crawling and scraping with HTML→Markdown and headless-Chrome fallback.
• html-to-markdown — fast, lossless HTML→Markdown engine.
• liter-llm — universal LLM API client with native bindings for 14 languages and 143 providers.
• tree-sitter-language-pack — tree-sitter grammars and code-intelligence primitives.
• alef — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos.

License

MIT License

Links

• Documentation
• GitHub Repository
• Issue Tracker
• Changelog
• Discord — community, roadmap, announcements.