free-search-mcp

free-search-mcp

A local-first, no-API-key Model Context Protocol server that gives anyLLM (Claude, GPT, local Ollama, …) the ability to search the web, fetch andclean up pages, and read documents — without you signing up for a singlesearch API.

It bundles together the best ideas from a handful of open-source MCPs intoone Python package, and adds the LLM-ergonomics and reliability work theywere each missing.

research("how does reciprocal rank fusion work", depth=3)
   ↓
# Research brief: how does reciprocal rank fusion work
_engines: duckduckgo, mojeek, googlenews · sources: 3 · ~3,400 tokens_

## Sources
- [1] Reciprocal rank fusion | Elasticsearch Reference — <https://…>
- [2] Hybrid Search Scoring (RRF) | Microsoft Learn — <https://…>
- [3] RRF explained in 4 mins — Medium — <https://…>

## Documents
…full Markdown bodies of each page, ready for the LLM to read…

One tool call. Three sources. No API key. No OPENAI_API_KEY-but-for-searchshakedown.

🚀 One-click deploy

Three commands — keyless engines work immediately, no signup, no key:

git clone https://github.com/sweetcornna/free-search-mcp.git
cd free-search-mcp
./scripts/install.sh        # installs uv + deps + Chromium, writes .env, smoke-tests

Then wire it into your client (pick one):

# Claude Code — register the server globally
claude mcp add search uv -- --directory "$PWD" run search-mcp
# …or just run `claude` inside the repo: the bundled .mcp.json auto-detects it.

Optional extras, any time (the defaults already work without them):

uv run search-mcp-admin        # browser UI to add API keys / a proxy  (127.0.0.1:8765)
uv run search-mcp-login zhihu  # one-time Zhihu login (persists cookies)

Prefer containers? docker compose run --rm search-mcp. Claude Desktop andother clients: see Install below.

Why this exists

Existing search MCPs each do one thing well, but you usually want all of it:

"LLM-tuned" here means: Markdown-first output, token estimates, smarttruncation at paragraph boundaries, "Best for / Not for / Returns / Commonmistakes" docstrings the model uses to pick the right tool, actionableerror hints, MCP prompts and resource templates, and a one-shotresearch() that collapses search→fetch→fetch→fetch into a single turn.

"Trafilatura" means we extract main content usingtrafilatura — winner of theBevendorff 2023 ROUGE benchmark (~0.85 vs ~0.55 for naive boilerplatestripping). Each fetched page also returns author, published_date, andsitename for free.

"Filters" means search/research accept freshness, include_domains,exclude_domains, category (news/pdf/github/paper/forum/blog),include_text, exclude_text.

Anti-detection & resilience

• HTTP fast path uses curl_cffiwith a real Chrome 131 JA3/JA4 + HTTP/2 fingerprint, fixing the DDG"anomaly 202" rate-limit response that vanilla httpx triggered.
• Playwright fallback uses launch_persistent_context (cookies surviverestarts on disk), prefers a real installed Chrome (channel="chrome"),drops the --no-sandbox fingerprint marker on macOS, and randomizes theviewport per session.
• Result dedup is title-fuzzy + host-canonical (rapidfuzztoken_set_ratio >= 92, host normalized for www./m./amp. andcountry-TLD collapse), catching bbc.co.uk vs bbc.com duplicates thatURL-only dedup misses.
• search includes an honest extractive lead_snippet — picks the top-3result whose snippet contains ≥2 query terms and is ≥80 chars; renderedas > **Lead:** According to {host}: …. No LLM call. Returns nothingif no snippet qualifies (no fake answer).

⚠️ We deliberately do not attempt to defeat proof-of-work captchason Bing or Brave — that crosses the ToS line. When those engineschallenge us, we fall back to other engines instead.

Tools (9)

Plus 4 MCP prompts (Research thoroughly, Fact-check claim,Compare sources, News brief) and 2 resource templates(cache://page/{url}, cache://search/{query_hash}).

Filters (search / research)

All tools default to format="markdown" — readable, ~40% fewer tokens thanJSON, with provenance and a token-budget header. Pass format="json" forstructured access.

Tool annotations

Every tool ships correct readOnlyHint, idempotentHint, andopenWorldHint annotations so MCP clients can label them and gateelevated actions.

Engines

Default set (all-HTTP, no browser, ~2x faster than the old defaultthat included Startpage):duckduckgo, mojeek, googlenews.

Opt-in:

• startpage — browser-rendered (~5-10s/query); good for hard-to-reachresults that the HTTP defaults miss.
• brave, bing, baidu — intermittent challenges to headless clients.
• searx — meta-search proxy via public SearXNG instances; included forcompleteness but most public instances are slow/unreliable in 2026.
• google — keyless Google web SERP scrape (HTTP first, Playwrightfallback when Google serves a JS/consent shell). serpsearch is a purealias of google (all dedicated "SERP APIs" require a key, so the onlykeyless SERP is a direct scrape).
• anysearch — AnySearchunified-search REST API, anonymous (keyless) tier; one HTTP call returnsfused, re-ranked results. IP rate-limited; 429/5xx degrade to empty.
• bilibili — keyless Bilibili (哔哩哔哩) video search via the publicweb-interface/search/all/v2 JSON API (synthetic buvid3 cookie, nologin). Returns video results only.
• zhihu — best-effort keyless Zhihu (知乎) search. Zhihu'sapi/v4/search_v3 needs login cookies + x-zse-96 signing, so the onlyno-key path is browser-rendering the public search page. Zhihu hard-gatesheadless clients, so a login wall / empty result is common and honest —treat it like baidu/brave.

All five added engines are keyless (no API key, no account) and stayopt-in — they're not in the fast default pool, so the ~2x latency winof the all-HTTP defaults is preserved. Enable per call withengines=["google","bilibili", ...], or globally viaSEARCH_MCP_DEFAULT_ENGINES.

API-key engines & the admin backend

Keyless is the default, but you can also plug in keyed providers for higherreliability/quality. These engines stay dormant (and return a clear "notconfigured" hint) until you add a key:

Simplest setup — the admin UI:

uv run search-mcp-admin          # opens a local config page on 127.0.0.1:8765

It serves one page (bound to localhost only) with, per provider: ahow-to-get-a-key guide + signup link, a masked key field, Save (applieslive — no server restart), a Test button, and Clear. Keys are written to~/.config/search-mcp/config.json (0600); they're never echoed back to thepage. Prefer env vars? Set SEARCH_MCP_<PROVIDER>_API_KEY instead (theseoverride the saved file). Full walkthrough for each provider:docs/API_KEYS.md.

search("…", engines=["brave_api"])        # once a key is saved
search("…", engines=["tavily", "serper"]) # mix keyed + keyless freely

When an engine is gated (proxy · fallback · login)

Some engines get blocked by the provider — Google/Bing serve a CAPTCHA todatacenter IPs, Zhihu needs a login. We don't defeat CAPTCHAs (ToS); instead:

• Proxy (the real fix for IP gating): set a proxy in the admin UI"Network / Proxy" card or SEARCH_MCP_PROXY (http/https/socks5, optionaluser:pass@). It routes the HTTP engines, the browser, and fetch through anon-blocked IP. Scope it with SEARCH_MCP_PROXY_ENGINES="google bing zhihu".
• SearXNG auto-fallback: when google/serpsearch/bing are CAPTCHA-gated,they transparently retry via the working searx meta-search — you still getresults, honestly attributed to searx.
• Gate diagnostics: the response includes gated_engines + gated_hinttelling you which engine was gated (captcha/consent/login) and how it washandled.
• Zhihu login: run uv run search-mcp-login zhihu (or the admin "Login"button) once — a browser opens, you log in, cookies persist, and zhihu searchthen works. Requires a desktop session.

Full guide: docs/PROXY_AND_GATES.md.

Brave/Bing/Baidu all gate headless browsers after a handful of calls (PoWCAPTCHAs, "something went wrong" pages, redirect wrappers). Passengines=["brave"] etc. only when the defaults can't find what you need.

Sparse-result diagnostics

When filters drop results so aggressively that ≤3 are returned, theresponse includes filter_diagnostics so the LLM knows which knob torelax. Example for category="forum" + exclude_text="beginner":

⚠️ **Filter diagnostics** (results were sparse)
Raw results: 20 across 3 engines → 0 after filters.
Top drops: category_forum (20).
Hint: Filters dropped 20 of 20 raw results. Most were excluded by
category=forum. Try widening or removing one filter.

Install

One-click setup

git clone https://github.com/sweetcornna/free-search-mcp.git
cd free-search-mcp
./scripts/install.sh        # installs uv (if needed) + deps + Chromium, writes .env, smoke-tests

The script prints the exact command to wire the server into Claude Code orClaude Desktop when it finishes. Prefer to do it by hand?

uv sync
uv run playwright install chromium
cp .env.example .env        # optional: customize engines/limits

Run as a stand-alone server (stdio transport):

uv run search-mcp

Docker (one-click, containerized)

docker compose build
docker compose run --rm search-mcp     # attaches stdio for MCP

Config & env vars

All settings are env vars prefixed with SEARCH_MCP_. Copy .env.example →.env and edit — it documents every knob, including how to enable the newengines via SEARCH_MCP_DEFAULT_ENGINES. See the full table underConfiguration and the usage guide.

Tests

uv run pytest -q                              # offline (default, no network)
SEARCH_MCP_TEST_NETWORK=1 uv run pytest -v    # live tests, hit the real web

Wire into Claude Code

This repo ships a project-scoped .mcp.json, so running claude inside theproject auto-detects the search server. To register it globally instead:

claude mcp add search uv -- --directory /absolute/path/to/free-search-mcp run search-mcp

Wire into Claude Desktop

Add this to ~/Library/Application Support/Claude/claude_desktop_config.json(macOS) or the equivalent on your platform:

{
  "mcpServers": {
    "search": {
      "command": "uv",
      "args": ["--directory", "/absolute/path/to/free-search-mcp", "run", "search-mcp"]
    }
  }
}

Restart Claude Desktop. The nine tools above will appear in the tooldrawer.

Wire into other clients

The server speaks plain MCP over stdio. Anything that supports MCP works:

• Claude Code (claude mcp add search uv --directory /…/free-search-mcp run search-mcp)
• Cursor / Continue / Cline (use the JSON snippet above)
• Custom Python / TypeScript clients via the official MCP SDK

Configuration

All settings can be overridden by environment variables prefixed withSEARCH_MCP_:

Architecture

   ┌─────────────────────────────────────────────────────┐
   │  FastMCP server (stdio)                             │
   │  tools: search / research / fetch / fetch_batch /   │
   │         read_doc / cache_search / engines           │
   └────────────┬────────────────────────────────────────┘
                │
   ┌────────────▼────────────┐  ┌────────────────────────┐
   │  aggregator             │  │  fetcher               │
   │  - parallel engines     │  │  - httpx fast path     │
   │  - reciprocal rank      │  │  - playwright fallback │
   │    fusion               │  │  - markdownify         │
   │  - search cache (FTS5)  │  │  - page cache (FTS5)   │
   └────┬────────────────────┘  └────────────┬───────────┘
        │                                    │
   ┌────▼─────────────────┐  ┌──────────────▼─────────────┐
   │  engines/            │  │  browser pool              │
   │   duckduckgo.py      │  │   - persistent context     │
   │   mojeek.py          │  │   - stealth init script    │
   │   searx.py           │  │   - shared cookies         │
   │   startpage.py (opt) │  │   - semaphore-bounded pages│
   │   brave.py     (opt) │  └────────────────────────────┘
   │   bing.py      (opt) │
   │   baidu.py     (opt) │
   │   google.py    (opt) │
   │   serpsearch.py(opt) │
   │   anysearch.py (opt) │
   │   bilibili.py  (opt) │
   │   zhihu.py     (opt) │
   └──────────────────────┘

   ┌────────────────────────────┐    ┌──────────────────┐
   │  documents/                │    │  ratelimit       │
   │   pypdf, python-docx,      │    │   token bucket   │
   │   markdownify              │    │   per engine     │
   └────────────────────────────┘    └──────────────────┘

   ┌────────────────────────────┐    ┌──────────────────┐
   │  formatting                │    │  research        │
   │   token estimate           │    │   composed       │
   │   smart truncation         │    │   workflow       │
   │   markdown renderers       │    │                  │
   └────────────────────────────┘    └──────────────────┘

Engine adapter pattern

Each engine in src/search_mcp/engines/ implements:

class Engine:
    name: str
    needs_browser: bool          # Force Playwright?
    wait_selector: str | None    # CSS to wait for in browser mode

    def build_url(self, query: str, max_results: int) -> str: ...
    def parse(self, html: str) -> list[SearchResult]: ...

The base class handles transport (httpx → Playwright fallback), ratelimiting, and the case where HTTP returns a captcha shell instead ofresults (auto-retries via the browser).

Credits

This project stands on the shoulders of:

• mrkrsl/web-search-mcp —smart httpx-then-Playwright fetch strategy, multi-engine fallback chain
• Aas-ee/open-webSearch —multi-engine breadth (Bing/DDG/Baidu/Brave/Startpage)
• VincentKaufmann/noapi-google-search-mcp —anti-detection patterns (navigator.webdriver, UA, cookies), SQLiteFTS5 cache idea, multi-format read_document
• nickclyde/duckduckgo-mcp-server —per-engine rate limiting, LLM-friendly content cleanup
• Mojeek — independent search index thatdoesn't gate on User-Agent
• Model Context Protocol and theofficial Python SDK

License

MIT — see LICENSE.