Context Gateway

Context Gateway

A tiny always-loaded gateway + searchable registry that lets any MCP-capableclient discover all of your context — tools, skills, plugins, MCP servers,folders, agents, data sources, reference docs — on demand, without payingthe token cost of loading everything into the first prompt.

It generalises the "code-mode" progressive-discovery trick (a small gatewaythat reveals capabilities as you ask for them) to every context type.

first prompt  →  map()      →  search()     →  describe()    →  load()
 ~1-2 KB        categories    name+one-liner   full card       actual content /
 (bootstrap)    + counts      matches only     (usage, cost)   activation steps

Nothing heavier than the layer you asked for ever enters the context window.

Why this exists

Clients only bind the tools/skills/context they can see at session start.If you expose everything up front you blow the context window; if you don't,capabilities are "not found" and never used. The gateway resolves the tension:the only thing a client needs at session start is the gateway server + a~500-token bootstrap block. Everything else is discoverable in one tool call.

The four discovery layers

Control plane vs data plane

Discoverability ("can this be found") and calling ("how does it run") are keptseparate on purpose.

Control plane — discoverability (persists in servers.yaml):

Disabling hides an asset from map/search without deleting it. search(..., include_disabled=True) still reveals hidden assets (marked [disabled]).

Health plane — availability (probed, stored, timestamped):

The gateway can actually test the MCPs it finds — connect, initialize, counttools, measure latency — and record a snapshot per server: available,timeout (slow to start, not dead), unavailable (with the error), orunconfigured (latent). Snapshots carry a UTC timestamp and go stale after a TTL.State refreshes lazily on use: a successful proxy call marks a server available,a failed one marks it unavailable, and suggest/search(available_only=True)re-probe only the stale servers among the ids they touch. You can then filterby availability: search(..., available_only=True), list_servers(available_only= True), or health_status(status="available"). Because probing spawns realdownstream processes, bulk probing is opt-in and scoped (pinned → enabled →all), concurrency-capped, and gives slow servers a generous timeout.

Data plane — calling (default: direct):

By default the gateway is a catalog, not a proxy — it hands the client thecoordinates and gets out of the data path, so your servers connect directly tothe host with no abstraction, no credential concentration, and native features(OAuth, resources, sampling) intact. For the occasional server where zero-configcalling is worth it, flip set_proxy(id, True) and the gateway will relay asingle call via call(id, tool, args) — reading the live config only at calltime, never storing secrets in a card.

Recommendation: keep direct as the default; use latent servers +enable/disable as your discoverability control plane; reach for proxy onlyon trusted, non-interactive servers where instant zero-config calling pays off.

Components

• registry/ — one flat card (.md with YAML front-matter) per asset.Git-tracked, human-editable, diffable. Source of truth for content.
• servers.yaml — the manifest: discoverability state (enabled/proxytoggles) that persists across reindexing, plus latent servers not yetwired into any client. Source of truth for state.
• indexer.py — crawls your MCP configs, skills, agents, commands andnominated folders; refreshes cards; applies the manifest; builds a SQLite FTS5index. Also does CLI toggles (--enable/--disable/--proxy/--list-servers).
• gateway_server.py — the small MCP server exposing the discovery, control,and data-plane tools. The one thing every client registers. --selftestruns an in-process client→server check.
• cg_common.py — shared library imported by every script so they never drift.
• cg_insights.py — background layer: usage logging, recipes, evidence-basedsuggestions. Nothing here alters a response until it has real data.
• cg_health.py — probes downstream MCPs (connect / list-tools / latency),stores timestamped health snapshots in state/health.json with staleness TTL.
• recipes.yaml — named bundles of cards used together (deterministic co-use).
• profiles/ — YAML scopes (e.g. dev, hydra-ops) with scope_types /allow / deny / pin, filtering discovery per agent / workspace.
• bootstrap/ — generated first-prompt blocks, one per profile.
• ops/ — nightly reindex (reindex.sh, launchd plist, install-launchd.sh).
• logs/ — local usage log (git-ignored).

Pins & the background insight loop

Pins are ≤5 hand-set anchors per profile — entry points in the bootstrap, nota toolkit, and never auto-written. Static keyword auto-pinning is deliberatelyavoided: it optimises for word overlap, not for what's actually used or usedtogether. Instead, value is measured, not guessed:

• Usage logging — every describe/load/call is appended to logs/usage.jsonlwith session + profile + timestamp. Zero context cost, fail-safe.
• Recipes — you name a bundle of cards used together (save_recipe) and pullit in one call (load_recipe). Deterministic capture of co-use — the reliableversion of what auto-pinning faked.
• suggest(profile) — ranks cards by real frequency + recency + sessionco-occurrence, excluding current pins. Returns a collecting status untilenough use has accrued, then honest evidence you can promote to pins.
• recommend_profiles() — clusters co-occurring cards into candidate profilesfor you to review and save.
• usage_stats() — shows what's been captured so far.

So anchors work today; everything else quietly gathers evidence so futuresurfacing is grounded in what happened, not keywords.

State model (what's authoritative)

• Content → registry/*.md cards. What exists, its description, how to activate.
• State → servers.yaml. What is discoverable, what may be proxied.
• Index → index.db. Derived from both, disposable, git-ignored, rebuilt onevery toggle so the manifest always wins.

Hand-written notes below the <!-- END GENERATED --> marker in any card arepreserved across reindexing.

Quick start

cd context-gateway
pip install -r requirements.txt

python3 indexer.py --root /Volumes/dev-4tb/AA-GitHub/MCP   # build registry + index
python3 indexer.py --list-servers                          # see servers + state
python3 indexer.py --disable mcp-digitalocean-all          # hide one from discovery
python3 indexer.py --proxy mcp-github-mcp                  # allow proxy calling
python3 gateway_server.py --selftest                       # verify it works
python3 gateway_server.py                                   # run the MCP server (stdio)

Then register the server with any client (see INSTALL.md) and paste therelevant bootstrap/<profile>.md into that client's CLAUDE.md / projectinstructions / system prompt.

Design principles (your stated preferences)

• Simple — flat files + one small server. No DB as source of truth.
• Durable — plain markdown in git; survives tooling churn.
• Easy to maintain — cards regenerate from source; hand notes are preserved.
• Scalable — add cards; search cost stays flat (FTS index).
• Profiles — per-agent / per-workspace scoping built in.
• Fast — keyword FTS5, no network, no embeddings required.
• Accurate — cards written from the real source (tool lists, frontmatter).

Security

The indexer never copies secrets into cards. All env, headers,Authorization, tokens and API keys in MCP configs are stripped. Cards storeonly names, descriptions, tags and how to activate — not credentials.