pyscrappy

PyScrappy: robust, all-in-one Python web scraping toolkit

PyScrappy is a Python toolkit for web scraping that works out of the box. Point it at any URL and get structured data back — or use built-in scrapers for Wikipedia, IMDB, Yahoo Finance, news feeds, and more.

Key features

• Generic scraper — give it any URL, get back structured text, links, images, tables, and metadata
• Auto-pagination — automatically follows "next page" links
• JS rendering — optional Playwright backend for JavaScript-heavy sites
• Custom selectors — pass CSS selectors to extract exactly what you need
• 20+ built-in scrapers — Wikipedia, IMDB, stocks, news, GitHub, Hacker News, books, weather, Amazon/Newegg/IKEA, LinkedIn, YouTube, Uber Eats, and more
• MCP server — expose the scrapers as tools for AI agents (Claude, Cursor, local LLMs, …)
• Concurrent scraping — scrape_many / scrape_all run scrapes in parallel
• Proxy & scraping-API support — route through a proxy or ScraperAPI/ScrapeOps for blocked sites
• Clean API — every scraper returns a ScrapeResult with .to_dataframe() and .to_json()
• Retry & rate-limiting — built-in exponential backoff and per-domain rate limiting
• Type-safe — full type hints, py.typed marker

Installation

pip install pyscrappy

Optional extras:

# Browser support (for JS-rendered pages)
pip install 'pyscrappy[browser]'
playwright install chromium

# DataFrame support
pip install 'pyscrappy[dataframe]'

# MCP server (use PyScrappy's scrapers as AI-agent tools)
pip install 'pyscrappy[mcp]'

# Everything
pip install 'pyscrappy[all]'

Quick start

Scrape any URL (one-liner)

from pyscrappy import scrape

result = scrape("https://en.wikipedia.org/wiki/Web_scraping")
print(result.data[0]["metadata"]["title"])
print(result.data[0]["text"]["word_count"])

Custom CSS selectors

from pyscrappy import GenericScraper

with GenericScraper() as gs:
    result = gs.scrape(
        url="https://news.ycombinator.com",
        selectors={"title": ".titleline a", "score": ".score"},
    )
    for item in result.data:
        print(item["title"], item.get("score", ""))

Wikipedia

from pyscrappy import WikipediaScraper

with WikipediaScraper() as ws:
    result = ws.scrape(query="Python (programming language)", mode="summary")
    print(result.data[0]["text"])

Stock data

from pyscrappy import StockScraper

with StockScraper() as ss:
    result = ss.scrape(symbol="AAPL", mode="history", period="1mo")
    df = result.to_dataframe()
    print(df.head())

IMDB (via OMDb API)

IMDB's own pages are protected by an anti-bot challenge, so PyScrappy fetches IMDBdata through the free OMDb API. Set an OMDb API key inthe OMDB_API_KEY environment variable (or pass api_key=...).

from pyscrappy import IMDBScraper

with IMDBScraper() as scraper:      # reads OMDB_API_KEY from the environment
    # Search by title
    result = scraper.scrape(query="inception")
    # ...or look up a specific IMDB id
    result = scraper.scrape(query="tt1375666")
    df = result.to_dataframe()
    print(df[["title", "year", "rating", "genre"]])

News (RSS feeds)

from pyscrappy import NewsScraper

with NewsScraper() as ns:
    result = ns.scrape(feed_url="https://rss.nytimes.com/services/xml/rss/nyt/World.xml")
    for article in result.data[:5]:
        print(article["title"])

Image search

from pyscrappy import ImageSearchScraper

with ImageSearchScraper() as iss:
    result = iss.scrape(query="golden retriever", max_images=10, download_to="./dogs")

Configuration

from pyscrappy import ScraperConfig, GenericScraper

config = ScraperConfig(
    timeout=20.0,            # request timeout in seconds
    max_retries=3,           # retry failed requests
    rate_limit=2.0,          # seconds between requests per domain
    proxy="http://...",      # proxy URL, or a list to rotate through
    scraper_api=None,        # route via a scraping-API service (see below)
    headless=True,           # browser runs headless
    render_js="auto",        # auto-detect if JS rendering is needed
    cache_ttl=0,             # response cache TTL in seconds (0 = disabled)
)

with GenericScraper(config) as gs:
    result = gs.scrape(url="https://example.com")

Proxies and blocked sites

Some sites (e.g. eBay, Instagram, Twitter/X, Spotify) block direct automatedrequests. PyScrappy supports two ways to get through them.

A proxy (or a rotating list) — applies to both the HTTP and browser backends:

from pyscrappy import ScraperConfig, AmazonScraper

# Single proxy
config = ScraperConfig(proxy="http://user:pass@host:port")

# Rotating list (one picked per request)
config = ScraperConfig(proxy=["http://p1:8080", "http://p2:8080"])

A scraping-API service (ScraperAPI, ScrapeOps, ScrapingBee) — routes requeststhrough the service, which handles proxies and anti-bot challenges for you:

config = ScraperConfig(scraper_api={
    "provider": "scraperapi",   # or "scrapeops", "scrapingbee"
    "api_key": "YOUR_KEY",
    "render_js": True,           # optional
})

# Now any scraper works through the service, unchanged:
with AmazonScraper(config) as scraper:
    result = scraper.scrape(query="laptop")

This is the reliable way to use the scrapers marked "needs proxy" below.

Concurrent scraping

Scraping is I/O-bound, so running several scrapes at once parallelizes thenetwork waits. scrape_many runs one scraper over many inputs; scrape_allruns a mix of scrapers together. Both preserve input order.

from pyscrappy import scrape_many, scrape_all, AmazonScraper, WikipediaScraper, NewsScraper

# One scraper, many queries, concurrently:
results = scrape_many(AmazonScraper, [{"query": "laptop"}, {"query": "phone"}])

# Different scrapers at once:
results = scrape_all([
    lambda: WikipediaScraper().scrape(query="Python"),
    lambda: NewsScraper().scrape(feed_url="https://rss.nytimes.com/services/xml/rss/nyt/World.xml"),
])

Response caching

Set cache_ttl to a positive number of seconds to cache successful GETresponses. Repeated requests for the same URL (and query params) within the TTLare served from cache, skipping both the network and the rate limiter. Cachingis disabled by default (cache_ttl=0).

from pyscrappy import WikipediaScraper
from pyscrappy import ScraperConfig

config = ScraperConfig(cache_ttl=300)   # cache for 5 minutes

with WikipediaScraper(config) as ws:
    ws.scrape(query="Python")   # fetched over the network
    ws.scrape(query="Python")   # served from cache

The cache is in memory and shared across scraper instances in the same process(so it also speeds up repeated calls through the MCP server), and is clearedwhen the process exits. Call HttpClient.clear_cache() to empty it manually.

YouTube

from pyscrappy import YouTubeScraper

with YouTubeScraper() as scraper:
    result = scraper.scrape(query="python tutorial", max_results=10)
    for video in result.data:
        print(video["title"], video.get("views", ""))

SoundCloud

from pyscrappy import SoundCloudScraper

with SoundCloudScraper() as scraper:
    result = scraper.scrape(query="lo-fi beats", max_results=10)

E-Commerce (Amazon, Newegg, IKEA)

from pyscrappy import AmazonScraper, NeweggScraper, IKEAScraper

# Amazon — general marketplace
with AmazonScraper() as scraper:
    result = scraper.scrape(query="laptop", max_pages=2)

# Newegg — electronics / computer hardware
with NeweggScraper() as scraper:
    result = scraper.scrape(query="graphics card", max_pages=2)

# IKEA — furniture / home (uses IKEA's JSON search API).
# Prices and products are per-country: pass the store's country/lang.
with IKEAScraper(country="gb", lang="en") as scraper:   # or "us"/"en", "de"/"de", …
    result = scraper.scrape(query="desk", max_results=24)
    df = result.to_dataframe()

Food Delivery (Zomato, Uber Eats)

from pyscrappy import ZomatoScraper, UberEatsScraper

with ZomatoScraper() as scraper:
    result = scraper.scrape(city="bangalore", max_results=20)

# Uber Eats: restaurants by city (any country Uber Eats operates in),
# then the full menu of any restaurant. Pass the country's locale.
with UberEatsScraper(locale="gb") as scraper:
    result = scraper.scrape(city="London", max_results=30)
    store = result.data[0]
    menu = scraper.get_menu(store["url"])   # items with prices

Built-in scrapers

Every scraper that works without a proxy is also exposed as an MCP tool (last column).

MCP server (use PyScrappy from an AI agent)

PyScrappy ships an optional Model Context Protocolserver, so an AI agent (e.g. Claude) can call PyScrappy's scrapers as tools andget structured web data back.

pip install 'pyscrappy[mcp]'

This installs a pyscrappy-mcp command (a stdio MCP server). You can also run itwith python -m pyscrappy.mcp.

Register with Claude Code

claude mcp add pyscrappy pyscrappy-mcp

Register with Claude Desktop

Add to your claude_desktop_config.json and restart the app:

{
  "mcpServers": {
    "pyscrappy": {
      "command": "pyscrappy-mcp"
    }
  }
}

Tip: Claude Desktop does not inherit your shell PATH. If pyscrappy-mcpis not found, use the absolute path to the command (e.g. the one printed bywhich pyscrappy-mcp).

Available tools

The lookup_movie tool needs a free OMDb APIkey. Pass it to the server through your MCP client config, e.g. for Claude Desktop:

{
  "mcpServers": {
    "pyscrappy": {
      "command": "pyscrappy-mcp",
      "env": { "OMDB_API_KEY": "your-key" }
    }
  }
}

Once registered, just ask the agent naturally, e.g. "use pyscrappy to get thelatest headlines from bbc.co.uk and the AAPL stock quote."

Dependencies

Required: httpx, beautifulsoup4, lxml

Optional: playwright (JS rendering), pandas (DataFrames), mcp (MCP server)

License

MIT

Contributing

All contributions welcome. See Issues.

This package is for educational and research purposes.